Accelerating Non-Conjugate Gaussian Processes By Trading Off Computation For Uncertainty
Authors: Lukas Tatzel
Jonathan Wenger
Frank Schneider
Philipp Hennig
Venue: Accepted by TMLR ()

Abstract: Non-conjugate Gaussian processes (NCGPs) define a flexible probabilistic framework to model categorical, ordinal and continuous data, and are widely used in practice. However, exact inference in NCGPs is prohibitively expensive for large datasets, thus requiring approximations in practice. The approximation error adversely impacts the reliability of the model and is not accounted for in the uncertainty of the prediction. We introduce a family of iterative methods that explicitly model this error. They are uniquely suited to parallel modern computing hardware, efficiently recycle computations, and compress information to reduce both the time and memory requirements for NCGPs. As we demonstrate on large-scale classification problems, our method significantly accelerates posterior inference compared to competitive baselines by trading off reduced computation for increased uncertainty.

Published in Transactions on Machine Learning Research (04/2025)
Accelerating Non-Conjugate Gaussian Processes By Trading Off Computation For Uncertainty
Lukas Tatzel
lukas. htwasstttteeeeee--uuennnnn....e
University of Tübingen, Tübingen AT Center
Jonathan Wenger
julk2466@cckumbica.e edu
Columbia University
Frank Schneider
fschredderrmmi--uueenggn..dd
University of Tübingen, Tübingen AT Center
Philipp Hennig
philipp. Salliipp.ennii@@uniitueeeeeee
University of Tübingen, Tübingen AT Center
Reviewed on OpenReview: https: // openreview. net/
metteeetuueeeee forum? id- UdcF3JeSKb
Abstract
Non-conjugale Gaussian processes (NCGPs) define flexible probabilistic framework to model categorical, ordinal and continuous data, and widely used in practice. However, exact inference in NCGPs is prohibitively expensive for large datasets, thus requiring approximations in practice. The approximation error adversely impacts the reliability of the model and is not accounted for the uncertainty of the prediction. We introduce a family of iterative mothods that explicitly model this error. They are uniquely suited to parallel modern computing hardware, efficiently recycle computations, and compress information to reduce both the time and memory requirements for NCGPs. As demonstrate on large-scale classification problems, method significantly accelerates posterior inference compared to competitive baselines by trading off reduced computation for increased uncertainty.
1 Introduction
Non-conjugate Gaussian processos' (NCGPs) form fundamental interpretable model class widely used throughout the natural and social sciences. For example, NCGPs are applied to count data in biomedicine, categorical data object classification tasks, and contimuous data in time series regression. An NCGP assumes the data generated from an exponential family likelihood with a Gaussian process (GP) prior over the latent function. Such a probabilistic approach is essential in domains where critical decisions must be made based on limited information, such in public policy, medicine robotics.
Unfortunately, even the conjugate Gaussian case, where fitting an NCGPs reduces to GP regression, naively has cubic time complexity O(N3) in the number of training data N and requires O(N²) memory, which is prohibitive for modern large-scale datasets. For non-Gaussian likelihoods, inference has to be done approximately, which generally exacerbates this problem. For example, inference via the Laplace approximation (LA) boils down finding the mode of the log postcrior via Newton's 's method, which equivalent to solving sequence of regression problems (Bishop, 2006; MacKay, 1992; Spiegelhalter & Lauritzen, 1990).
Such model also called Generalized Caussian Process Model (Chan & Dong, 2011) Generalized Lincar Model (Nelder & Wedderburn, 1972). The latter sometimes used only for latent Gaussian modcls, which can lead to confusion. The models studied this work are generally nonporametric nature. The resulting large linear problems are main reason why the algorithms propose are relevant the first place.

---

Published in Transactions on Machine Learning Research (04/2025)
Loss
IterN Chol
0.7
0.6
i=0,j=1
i=
1=1,j=0
1=3,j=5
0.5
IterNCGP-CC CG
(recycle & compress)
0.4
Train
Test
0.01
0.02
0.03
Runtime in
Figure 1: Binary Classification with IterNCGP. Comparison of two ITERNCGP variants: (Top) ITERNCGP variant corresponding to data subsampling and solving each regression problem exactly in each Newton step i. (Bottom) ITERNCGP variant with a more informative policy (details in Section 3.2), recycling of computations between Newton steps (details Section 3.3) and compression to reduce memory (details in Section 3.4). The panels show the marginal uncertainty
over the latent function at Newton step and solver ileration j. Using recycling, the current. belief is efficiently propagated between mode-finding steps @ without performance drops (Right), Details in Appendix C.1.
Due to limited computational resources, large-scale problems often require approximations. The resulting error affects model's predictive accuracy but also its uncertainty quantification. Hence, the question arises: Can NCGPs be efficiently trained on extensive data without compromising reliability?
Recently, iterative methods have emerged which in the conjugate Gaussian case allow an explicit, tunable tradc-off between reduced computation and increased uncertainty (Trippe et al., 2019; Wenger et al., 2022b). This computational uncertainty quantifies the inevitable approximation error in the sense of probabilistic numerics (Cockayne et al., 2019b; Hennig et al., 2015; 2022; Oates & Sullivan, 2019).
Contributions. In this work, we take a similar approach and extend Wenger et al. (2022b)'s ITERGP (that assumes conjugate Gaussian likelihood) to non-conjugate exponential family likelihoods. This is non-trivial extension, the posterior is no longer Gaussian and multiple related regression problems have to be solved. Specifically, we propose (i) TrERNCGP: family of efficient inference algorithms for NCGPs with tunable trade-off between computational savings and added uncertainty (Section 3.1) with (ii) mechanisms to tailor the inference algorithm to a specific downstream application (Section 3.2). In response to the specific computational challenges the non-conjugate setting, we develop (iii) novel strategies to optimally recycle costly computations (Section 3.3) and (iv) restrict the memory usage, with minimal impact on inference (Section 3.4).
Our algorithm ITERNCGP consists two nested loops: An outer loop (indexed by i) which iterates over Newton steps/ regression problems, each of which solved approximately via an inner loop (indexed by j) that implements probabilistic linear solver. Figure shows the marginal uncertainty over the latent function at different stages of that process and illustrates the effectiveness of (ii), (iii) and (iv). Specifically, by recycling computations between Newton steps, are able to traverse the two-loop structure "diagonally" which leads to steady progress without performance drops between Newton steps.
2 Background
Let (X,y) y) be a dataset of N input vectors {a,, X)N stacked into X (æI,... TN) e NND and corresponding outputs y = (y1.....yv) YN, where X RD and Y No (regression) or V = (1...., C) (classification).
2

---

Published in Transactions on Machine Learning Research (04/2025)
2.1 Non-conjugate Gaussian Processes (NCGPs)
We consider the probabilistic model ply, f x) ply f() X), where the vector f ; [(X) E RNC is given by latent function f: X - EC evaluated at the training data.
Prior. Assume a multi-output Gaussian process prior gP (m, K) over the latent function with mcan lunction m: X RC and kernel function K: x X RC Therefore the latent vector has densily p(f | = Wff:m, K) with mean m = m(X) E IN and covariance K , E RNCXNC defined by N blocks K( æ;) E RCxC, Each such block represents the covariance between the C latent functions evaluated at imputs z; and Z-.
Likelihood. Assume iid data, which depends on the latent function via an inverse link function : - RC s.t. pyy f) = II p(yn {(fn)), where p(t, | {(G.) is a log-concave likelihood, e.g. any exponential family distribution." 2 For example, for Poisson regression the inverse link function is given by A(fn) = exp(fn) and for multi-class classification by X(fn) softmax (fn).
For nonlinear inverse link functions, the posterior p( X, and predietive distribution plyo X, y, (b) Sply fo) X, dofof computationally intractable, requiring approximations.
2.2 Approximate Inference via Laplace
A popular way to perform approximate inference in NCGP to Laplace approximation (T.A) (Bishop, 2006; MacKay, 1992; Spiegelhalter & Lauritzen, 1990). The idea is to approximate the posterior
p(f x,y) = off x.3) = fMAP. E),
(1)
with a Gaussian with mean given by the mode ΠMAP of the log-posterior and covariance Σ == -(V log pl fMAP (x y)) given by the negative inverse Hessian (with respect to f) at the mode. Due to the assumed GP prior over the latent function, the oog-posterior given by
1 (f) = log X,y) y) - log f) + (fg X) & logp(y f) (f - -m) -m)]K" K- - m)
(2)
We - to denote equality up to an additive constant.
Mode-Finding via Newton's Method. To find the mode SMAP, one typically uses Newton steps, i.e.
MMPP R fi-1 = S: V (V(())
(3)
where V (f)) Vlogp(y fi) - K 181-mm m) and (f - -W() - --1, The negative Hessian W(f) 2 log fi) of the log likelihood at fi positive definite for all f, since we assumed a log-concave likelihood. Therefore is concave and the Newton updates are well-defined.
2.3
Predictions
Using a local quadratic Taylor approximation of the log-posterior v around the current iterate fú, we obtain the LA (f+::f++i--o V (f)) whose mean is given by the maximizer of the local quadratic, i.c. the subsequent Newton iterate f++1- Substituting this in place of the posterior, the predictive distribution for the latent function p(f(-) X, y) Sp(f() [)N(( f411, - df is Gaussian process GP (mi, Ki with mean and covariance functions
mi,4() (-) == m(-) + K(, ()K 1aKK-(faa (ft41 -m),
(4)
Kg (:.) K(5-) K(:, -f.)
(5)
where k(f)) = K - (ef. Eq. (3.24); Rasmussen & Williams, 2006). We obtain the predictive distribution for yo at test input x, by integrating this approximative posterior against the likelihood, i..
2The Hessian of exponential family likelihood is the negative Hessian of its log-partition function, which equals the positive definite covariance matrix of its sufficient. statistics.
3

---

Published in Transactions on Machine Learning Research (04/2025)
Latent function f
Prior @P K)
Posterior Ki..)
1=0
-2.
2.5
5.0
2.5
0.0
2.5
5.0
2.5
0.0
2.5
5.0
X
Figure 2: Approximate Inference in NCGPs Sequential GP Regression. Performing LA at Newlon ilerate Si results in posterior GP that coincides with the posterior to a GP regression problem with pseudo targets ğ(fi) observed with Gaussian noise (O, (ff)) The plot shows an illustration of this connection for binary classification on a toy problem with the latent function drawn from GP. Notice how similar the posteriors are between Newton steps. This motivates our proposed strategy for recycling computations between steps Section 3.3. Details in Appendix C.I.
ply. y, = Sp(v. pof X, v.) dfo. This C-dimensional integral can be approximated via quadrature, MC-sampling or specialized approaches (like the probit method (MacKay, 1992) for categorical likelihood and softmax inverse link function).
3 Aomputation-Aware Inference in NCGPs
While Newton's method typically converges in a few steps for a log-concave likelihood, each step in (3) requires linear system solves with symmetric positive (semi-)definite matrices of size NC x NC. Naively computing these solves via Cholesky decomposition problematic even for moderately sized datasets due its cubic time C(N and quadratic memory complexity C(N2C)). We will demonstrate in the following how to circumvent this issue by reducing the computations in exchange for increased uncertainty about the latent function.
3.1 Derivation of the IterNGGP Framework
Overview. As a first step, we reinterpret the posterior predictive mcan (Equation (4)) as the GP postcrior for specific regression problem (Equation (6)). The sequence of regression problems is connected to the sequence of Newton steps and forms the outer loop of our algorithm ITERNCGP, indexed by i (Algorithm 1). The formulation GP regression problem allows us to apply ITERGP (Wenger et al., 2022b) as an inner loop, indexed by j (Algorithm 2). By using ITERGP, we can solve each regression problem approximately and quantify the resulting error in the form of additional uncertainty (Fquation (9)).
Outer Loop: Newton's Method as Sequential GP Regression. Through the LA, we obtain a posterior predictive f ~ GP K,, over the latent function for each Newton step. As we show in Appendix A.1, the posterior predictive (Equations (4) and (5)) in step can be written
mi, (+) = m(.)) + K(., X) ()))) - m)
(6)
Kg (.,())))) K(.)- ():)))(()))((())))))
(7)
where ú(f:) : fo + (Vf.))VVoogyp ft). Equations (6) and (7) have the exact form of the posterior for
GP regression problem with fictitious pseudo targets ((f)) observed with Gaussian noise N NO W(f) Figure 2 shows illustration of this interpretation.
Equation (6) requires solving a linear system k(f.) v (f.)) m of size NC x NC. The posterior mean is then simply given by (.) = m(-) + K(. X)u. However, also the Newton update from Equation (3)
3If (f does not exist, e.g, in multi-class classification, substitute its pscudo-inverse W(f which for multi-class classification can evaluated efficiently (see Appendix A.6). Alternatively, place prior the sum the C latent. functions (MacKay, 1998, Eq. (10)).
4

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 1: IterNCGP Outer loop.
Input: GP prior GP (m, K), training data (X, Vp(ulf: X) and access to products with K and W(f)
Output: GP posterior GP (ma,, K5,5)
procedure (GE, K, $o m))
Time
Memory
m - (X),
m: Prior mean vector ((Tm)
((NC)
Provide access to ~ Kw
K: Prior covariance/ rrariance//ernel /kennel matrix
O(uk)
Initialize buffers S, S, E R NOXD
S: actions, T: products with K
5 for i 2... while not OUTESSTOPPINCAATENNNI
do
6 Provide access to - W(f))='
W(f:) Observation noise
Olew 1) O(NC)
7
ú(f.) - fi
ÿ(f.): Pseudo targets O(w--++NC)
GP(ms.. K..)) Uj - ImERGP(m, y.m.
m.m ( . Approximate Newton update O(TK+NC)
fi+1 < Kv + m
((NC)
10 return GP (ma.y K..))
The ITERGP algorithm given Algorithm 2. Instructions blue are needed for recycling (see Section 3.3). The matrices K and (f) are evaluated lazily. We thus report the runtime costs when the matrix-vector products are actually computed. For an in-depth discussion of the computational costs, see Appendix B.3.
follows directly from a since fi+1 = Ku + m. In that sense, computing the posterior predictive mcan and performing Newton updates are equivalent. The sequence of Newlon sleps/GP regression problems forms the outer loop of our algorithm ITERNCGP. The pseudo code given in Algorithm
Inner Loop: Computation-Aware GP Regression via IterGP. Reframing the Newton iteration sequential GP regression does not yel solve the need for linear solves with matrix size NC x NC. However, it allows us to leverage recent advances for GP regression, specifically the ITERGP algorithm introduced by Wenger et al. (2022b). ITERGP is matrix-free, i.. only relics on matrix-voctor products s - K(f)) s, reducing the required memory from quadratic to linear, and efficiently exploits modern parallel GPU hardware (Charlier al., 2021).
Internally, ITERGP uses a probabilistic linear solver (PLS) (Cockayne et al., 2019a; Hennig, 2015; Wenger & Hennig, 2020) to iteratively compute Gaussian belief over the so-called representer weights, i.e. over the solution of the linear system (fss) v ((f:) m. In each solver iteration, indexed by j, the belief N( Uy, S2,) is updated by conditioning the Gaussian on a one-dimensional projection aj 8 rー of the precoding residual ry-1 ((fi) m (f.)o,-1 The vector 8, - POLICY() is called an action and is generated by user-specified policy. The policy determines the behavior of the solver by "weighting" the residual r- for specific data points (we discuss the role of the policy in the context of NCGPs in Section 3.2.). Wenger et al. (2022b, Tab. lists policies that have classic counterparts, e.g. unit vectors 8j - ej correspond to partial Cholesky and residual actions 8j 75-1 to conjugate gradients (Hestenes & Stiefel, 1952).
The belief over the representer weights translates into an approximate GP posterior over the latent function,
m.,(.)) == m(-) + K(, X)u, K.g(,))= -(.)) X) X) C,K(X,)) K(.
where 0j Cy(i()) m). Crucially, by Wenger et al. (2022b, Thm. 2), the posterior covariance in Equation (9) exactly quantifies the error in each approximate Newton step introduced by only using limited computational resources, i.c. rumning the lincar solver for j & NC itcrations. This reduces the time complexity to OGNN2C2). The approximate precision matrix in Equations (8) and (9),
C = S,(S] - ((
(10)
with S₁ (s1,...,8,) R has rank and approaches (fa) j NC. Intuitively, it projects k().) onto the subspace spanned by the actions Sj, then inverts and projects the result back into the original space. The ITERGP algorithm is given in Algorithm 2.
The Marginal Uncertainty Decreases in the Inner Loop. As we perform more solver iterations, the marginal uncertainty captured by the posterior covariance (Fquation (9)) contracts, i.e. for each i it holds
5

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 2: IterNCGP Inner Loop: IterGP with a Virtual Solver Run.
Input: GP prior GP (m, K); training data (X, V), m, access to products with K and W pseudo targets y. buffers
S,T
Output:
GP posterior GP (mas, Kay))
procedure ImERGP(m, K, y, m. K, W S,)
Time
Memory
C. T. VIRTUALSOLVERRUNNS... W
See Algorithm 3
3
Co(g m))
Consistent initial iterate O(RNC)
O(NC)
4 for = 1,2,,.... while not INNESSTOPPINCCTTROONN do
-0 03-1
Py-1 : Residual vector
( + +Tw- +NO)
O(NC)
8, < POLICY()
Select action 8y via policy
(Trowcy)
((NC)
Append 8, to buffer 5 (,, RNCD
((BNC)
0j a- r5-1
0j Observation is projection of residual onto action O(NC)
O(1)
t - Ksj
First term =s Ka, W
((тк)
O(NC)
10
Append L to buffer T (T,4) e NCBB
O
11
2j 1W-W
Second term Kay=Kaj++ W 8, O(Tw NC)
((NC)
12
d, 8, -C-1
dj Search direction Normalizaionn constant Append column
O(BNC)
O(NC)
13
J 2; dj
O(NC)
O(1)
14
Q, - (Qi-1, 1 dj) NCxB
O(NC)
O(BNC)
15
C - Q
Rank approximation C2 ~ R
16
0j < fz-1 + d dj
by Updated representer weights estimate
O(NC)
O(NC)
17
m,y() - m(+) K(- X)0,
Equation (8)
O(NN.C3)
O(N.C)
18
Kiy(::) K(.)) K(: ((;)))(XXX)))
Equation (9) O(B(N+N)NNCC))
O(N3C9)
19
return GP(m..j, K.,) and 2
Instructions in blue needed for recycling (see Section 3.3). Cj is represented via its root Q2 and evaluated lazily. We thus report the runtime costs when the matrix-vector products are actually computed. The costs for evaluating the posterior GP GP (ma.j; K.,) are based No test data points X, NoXD For in-depth discussion of the computational costs, see Appendix B.3.
(element-wise) that diag( (x, ) 2 (e, æ)) for any k > j and arbitrary æ. This is because the approximate precision matrix C, grows in rank with each solver iteration. For a detailed derivation, see Appendix A.3.
Summary. Finding the posterior mode fMAP and the corresponding predictive distributions (Fquations (4) and (5)) can be viewed from different angles. Through an optimization lens, Newton updates, each maximizing local quadratic approximation of the log-posterior. From a probabilistic perspective, we solve a sequence of related GP regression problems and ITERGP enables us to propagate probabilistic estimate of the latent function throughout the entire optimization process.
For Gaussian likelihoods, the LA (Equation (1)) is exact and a single Newton step suffices. Consequently, our framework generalizes ITERGP to arbitrary log-concos logcconcave likelihoods (Theorem A.2). We now explore the role of the policy and its potential for actions tailored to specific problems (Section 3.2). We also leverage the relatedness of GP regression problems in the outer loop for further speedups (Section 3.3) and introduce mechanism to control ITERNCGP's memory usage (Section 3.4).
3.2 Policy Choice: Targeted Computations
Algorithm 2 defines a family of inference algorithms. Its instances, defined by concrete action policy, generally behave quite differently. To better understand what effect the sequence of actions S = ($1..... .8) e NCX) has on ITERNGGP, we consider the following examples.
Unit Vector Policy Subset of Data (SoD). Choosing the actions s, = to be unit vectors with all zeros except for one at entry j, corresponds to (soquentially) conditioning on the first j data points the training data in each GP regression subproblem, since K(, = k(, (1:)). Therefore this policy is equivalent to simply using subset. X1. X of the data and performing exact GP regression (e.g. via

---

Published in Transactions on Machine Learning Research (04/2025)
True posterior mean mo,
mo,1
170,10
Mo,19
GGGNCGG IteNCGP-
Figure 3: Different TterNGGP Policies Applied to GP Classification. (Left) The true posterior and its decision boundary (--). (Right) Current posterior mean estimale after 1, 10, and 19 iterations with the unit vector policy ( (Top) and the CG policy (Boldom). Shown are the data points selected by the policy in this iteration with the dot size indicating their relative weight. For IrERNCGP-Chol. data points targeted one by one and previously used data points are marked with (O). Details in Appendix C.1
mcan Ma,s ( for binary classification
a Cholesky docomposition) in each Newton itcration in Algorithm 1. This basic policy shows how actions largel compulalion illustrated in the top row of Figure
(Conjugate) Gradient Policy. Instead of targeting individual data points, linear combinations of them to target the data more globally. E.g. using the current residual 8j = T--1 = ū(f.) (f.) approximately targets those data most, where the posterior mean prediction is far off. As Wenger et al. (2022b) show, this corresponds to using conjugate gradients (CG) (Hestenes & Stiefel, 1952) to estimate the posterior mean. This policy is illustrated in the bottom row of Figure 3.
also specify weighted
3.3 Recycling: Reusing Computations
Using ITERGP with suitable policy for GP inference allows us to solve cach GP regression problem more efficiently. However, for NCGP inference, we must solve mulliple regression problems one per mode-finding step. Figure 2 suggests that GP posteriors across steps are highly similar. Leveraging this observation, we develop a novel approach, designed specifically for the NCGP setting, that efficiently recycles costly computations between outer loop steps (pscudo code in Algorithm 3).
The cost of TEERNCGP is dominated by repeated matrix-vector products with K (see Section 3.5). However, these costly operations can be recycled and used over multiple Newton steps: Consider the matrix-vector products with an action vector s in the first and second mode--inding step as example:
Step 0:
R(fo)s - Ks ( s - R(f)s== Ks ((f))
Step = 1:
Since K is independent of f., the product Ks is shared among both operations.
Virtual Solver Run. Assune we have used B action vectors (81,..., 8n) S E NC in step i = 0, and buffered the matrix-vectt vector products (Ks) = KS =: T. In the next Newton step i = 1 we apply the same actions to a new lincar system of equations. From Equation (10) we obtain
(T SM- ST with M T (KS+ w(f)) =T (T+ W).
(11)
Since r--1 ((f:) Kuj-1 ((f)) (X))

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 3: Recycling: Virtual Solver Run with Optional Compression.
Input: Buffers S,T E RCCB access to products with W-1, compression parameter At & B (optional) Output: Co, updated buffers S,T
procedure VIRTUASSOEERUNSSSTWWW
Time
Memory
2 M - S (T 5)
M + ( B Eigendeconposition UAU
O(B²)
U < :
O(B))
O(B²)
U U=(ui,..,uu)) A di AB) BBB
BxB
procedure COMPRESSION(U,A R)
U (U1:.... uR). diag A An)
Truncation R eigenpairs
O(BR)
6 S - SU, T TU
O(BRNC) Construct root Co- 2 ST ORP
Update buffers
O(RNC)
7 Qo < SA 112
O(RNC)
8 return C QuQi and S,T
Cu has rank R
Cu is never formed explicitly in memory but evaluated lazily via its root Qu: i.e. w - Cow = Qu(Q.
So, we can imilate solver with the previous actions S and construct C' without ever having to multiply with K. The associated computational costs comprise memory for the two buffers S, T well as the runtime costs for matrix- matrix-matrix products in Equation (11) and inverting M. This virtual solver run is generally orders of udenitude cheaper than running the solver from scratch with new actions (details in Appendix B.3). Within ITERGP (Algorithm 2), we can use this matrix as an initial estimate Co + C of the precision matrix. Subsequently, the algorithu can proceed as usual with new actions.
The presented recycling approach can easily be extended to all Newton steps. Whenever K is multiplied with action vector, the vector itself and the resulting vector appended to the respective buffers S and T. For each Newton step, an initial Co can be constructed via Algorithm 3.
Numerical Perspective. Crucially, the above strategy does not affect the solver's convergence properties: From a numerical linear algobra viewpoint, the strategy above is a form of subspace recycling (Parks et al., 2006). Specifically, Co, as described above, defines a deflation preconditioner (Frank & Vuik, 2001): The projection of the initial residual rp = (ğ- m) K vo for the first iterate Un- Colg-m onto the subspace [pan(S) spanned by the actions is zero (see Appendix A.4 for details). That means, the solution within the subspace SS is already perfectly identified at initialization.
Probabilistic Perspective. Via Equation (9), we can quantify the effect of C. on the total marginal uncertainty of predictions at the training data Tr(K, X, X)) = TT(K))T((CoK) Assuming observation noise W
0 and all actions in S eigenvectors of K K, it simplifies to
Tr(K,,(X, Tr Tr( -T)M).
(12)
see Appendix A.4. The second term Tr(M describes the reduction of the prior uncertainty due to C.. It can be maximized (which our goal) when S contains those eigenvectors of K with the largest eigenvalues. We take this insight as motivation for buffer compression approach that we describe next.
3.4 Compression: Memory-Efficient Beliefs
Whenever K is applied to an action vector, the buflers S, T NC grow by NC entries. To limit memory requirements for large-scale data, propose compression strategy (see Algorithm 3).
Compression via Truncation. In Algorithm 3, M- E BBxB is computed via an cigendecomposition M = UAU, such that C = Q can be represented via its matrix root Qo := SUA for efficient storage and matrix-vector multiplies. To limit memory usage, we can use a truncated eigendecomposition of M. Based on the intuition we gained from Equation (12), makes sense to keep the largest cigenvalues (to maximize the trace) and corresponding cigenvectors. Keeping the R largest eigenvalues/-vecte yields a rank R approximation M ÜAÜ of M.
8

---

Published in Transactions on Machine Learning Research (04/2025)
R 0
R
R
R 10
Figure 4: Compressed Beliefs. Recycled initial beliefs in the second Newton step (i= = 1) with means m1,0 (Top) and (co-)variance functions K1,0 (Bot- tom) using compression with different buffer sizes Re {0, 1,3, 10). Buffer size R = 0 is equivalent to not using recycling. The larger the buffer bizeraan of Co, the more expressive the belief. Details in Appendix C.1.
10,00
UR
K1.0
Compression as Re- Re-Weighting Actions. Forming C. SM-1S from the above approximation is equivalent to virtual solver run with the modified buffers S SU RNCRR T K ((UU) = TU RCCC in Equation (11). This shows that the truncated eigendecomposition effectively re- weights the previous B actions to form R new onos-and the weights are the cigenvectors from M that maximize the uncertainty reduction. The limit R on the buffer size controls the memory usage as well as the rank of Co and thereby the expressiveness of the associated belief (see Figure 4).
3.5 Cost Analysis of IterNCGP
ITERNCGP's total runtime is dominated by the repeated application of K in Algorithm 2, i.c. O(JTk), with describing the tolal number of solver iterations over all Newton steps. TK denotes the cost of a single matrix-vector product with K. Typically, тK is quadratic in the number of training data points. In terms of memory, the buffers S, T and the matrix root Q are the decisive factors with O(BCC). Without compression, their final size is B = J. Otherwise, their maximum size is given by the sum of the rank bound R and the maximum solver iterations in Algorithm 2 (Appendix B.3 provides in-depth discussion of runtime and memory costs).
4 Related Work
The Laplace approximation (Bishop, 2006; MacKay, 1992; Rue et al., 2009; Spiegelhalter & Lauritzen, 1990) is commonly used for approximate inference in (Bayesian) Generalized Linear Models. Here, we consider the function-space generalization of Bayesian Generalized Linear Models, namely non-conjugate GPs, for which multitude of approximate methods have been proposed, arguably the most popular being variational approaches (e.g. Khan et al., 2012), such as SVGP (Hensman et al., 2015; Titsias, 2009). In contrast, to address the computational shortcomings of NCGPs on large datasets, we leverage iterative methods to obtain and efficiently update low-rank approximations. Similar approaches were used previously to accelerate the conjugate Gaussian special case (Cunningham et al., 2008; Gardner al., 2018; Guhaniyogi & Dunson, 2015; Murray, 2009; Wang al., 2019; Wenger et. al., 2022a), binary classification (Zhang et al., 2014) and general Bayesian linear inverse problems (Spantini et al., 2015). Trippe et al. (2019) is closest in spirit to approach if viewed from a weight-space perspective. Their choice of low-rank projection corresponds to a specific policy in our framework. Our approach not only enables the use of policies that are more suited to the given link function, but also saves additional computation, as well as memory, via recycling and compression. In each Newton iteration, the posterior for the current regression problem is approximated via ITERGP (Wenger el al., 2022b), which internally uses a probabilistic linear solver (Cockayne et al., 2019a; Hennig, 2015; Wenger & Hennig, 2020). Therefore, ITERNCGP is probabilistic nunerical method (Cockayne et al., 2019b; Hennig et al., 2015; 2022; Oates & Sullivan, 2019): It quantifics uncertainty arising from limited computation.
5 Experiments
We apply ITERNCGP to Poisson regression problem to explore the trado-off between the number of (outer loop) mode-finding steps and (inner loop) solver iterations (Section 5.1). In Section 5.2, we demonstrate our algorithm's scalability and the impact of compression on performance.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Test: loss
0= j=0
i= j=0
i= 17,
15.0
10.0
7.5
5.0
20
2.5
0.0
0.2
0.3
0.0
0.3
1.0
Runtime in
X
JS1 1 10
Latent log rate /
Posterior (P ks,s)
Counts y
355
21520
Latent rate A
Posterior (via MC)
Figure 5: Poisson Regression with IterNCGP. (Left) Test loss performance for ITERNCGP-CG with recycling and four schodules (j & 1,5, 10 or 20) over 100, 20, 10 or 5 steps (always using the same total budget of 100 iterations). For each schedule, the median (solid line) and min/ max (shaded area) over 10 runs are reported. The crosses indicate the end of each run. (Right) Posterior GP (m.sj. kz,) for the latent log rate f (Top) and the corresponding belief about the rate (Bottom) computed via MC at three timepoints during a run of ITERNCGP. The shaded 95% credible intervals show how stopping early trades less computation for increased uncertainly. Details in Appendix C.2.
5.1 Poisson Regression
Consider count data y NO generated from a Poisson likelihood with unknown rate A: X - -- R2. The log rale is modeled by GP. See Appendix C.2 for details.
Data & Model. We generate synthetic dataset by (i) sampling the log rate from a GP with an RBF kernel (ii) transforming it into the latent rate by exponcntiation, and (iii) sampling counts yn N, from the Poisson distribution with rate X(x,). The functions f, X, and the resulting count data are shown in Figure 5 (Right). Our model uses the same RBF prior GP to avoid model mismatch.
Newton Steps vs. Solver Iterations. From a practical standpoint, the performance achicvable within a given budget of solver iterations is highly relevant: How many linear solver iterations should be performed for each regression problem before updating the problem to maximize performance? To investigate this, we use ITERNGGP-GG and distribute 100 iterations uniformly over (5, 10, 20, 100} outer loop steps. Each run uses recycling without compression and repeated 10 times.
Results. Figure 5 (Left) indicates that the strategy with single iteration per step the most efficient. An "outdated" regression problem that
explanation might be that there is reason to spend compute could be updated instead. Of course, this only applies if recycling is used, such that the effective number of actions accumulates. As long as B & N, the cost due to repeated recycling (O(N) is dwarfed by the cost of products with K (O(N2)). Figure (Right) shows an ITERNCGP-CC run with one iteration per step. As we spend more compulational resources, our estimates approach the underlying latent function and, where data is available, the uncertainty contracts.
5.2 Large-Scale GP
Classification
Here, we showcase ITERNCGP's scalability. See Appendix C.3 for details.
Data & Model. We generate = 10a data points from a Gaussian mixture model with C = 10 classes. We use the softmax likelihood and assume independent GPs (each equipped with a Matérn kernel) for the outputs of the latent function. While this experiment uses synthetic data, the latent function is not drawn from the assumed GP model and thus the kernel not perfectly identified. Also note that, if we formed K in (working) memory explicitly, this would require (NC)2. 8 byte 8000 GB (in double precision). Solving
10

---

Published in Transactions on Machine Learning Research (04/2025)
Accuracy in % f
NLL loss 4
ECE 4
SoD
Nab = 250
2.25
Nou = 500
Nob 1,000
0.6
2.00
Nuu 22000
SVGP
1.75
0.4
0.000, U=5,000
0.001 5,00000
1.50
00000, U 10,000
1.25
0.2
IteNCGG-CGG
155, R=00
1.00
555, R=10
10a
10
10a
10
10
10"
10
102 10
10
10" 10
102 10 10
Runtime in 8
Runtime in s
Runtime in s
Figure 6: Large-Scale GP Classification. Comparison of SoD, SVGP with learning rate a e {0.001, 0.01, 0.05) and U {1000, 2500, 5000, 10000; inducing points (showing only the best three runs) and ITERNCGP-CG with Re {20, 10) on a classification problem with N = 10 training points and C = 10 classes in terms of accuracy (Left), NLL loss (Center) and ECE (Right). Performance metrics over five runs and are shown as solid (training set) or dashed (test set) lines. ITERNCGP-CG performs best in all three performance metrics, with minimal memory requirements. The two variants with R E {20, 10) are visually indistinguishable, i.e. the performance is not affected by compression. Details in Appendix C.3.
averaged
the linear systems precisely, c.g. via Cholesky dccomposition, is therefore infeasible, whereas our family of methods is matrix-free and can still be applied.
Methods. We compare the subset of data (SoD) approach from Section 3.2, the popular variational approximation SVGP (Hensman et al., 2015; Titsias, 2009) and our ITERNCGP-CC. For SoD, we materialize K in memory and compute its Cholesky decomposition. Four different subset sizes are used used-the largest. one Noub = 2000 requires 3.2 GB of memory for K. For SVGP, we use the implementation provided by GPYTORCH (Gardner et al., 2018). We optimize the ELBO for 10a seconds using ADAM with batch size 1024 and determine suitable hyperparameters via grid search over the learming rate a and the number of inducing points U. Only the best three settings are included in our final benchmark. TrERNGGP-CG is applied to the full training set with recycling and R e (co, 10). The number of solver iterations is limited by & 5. We use KEOPS (Charlier et al., 2021) and GPYTORCH for fast kernel-matrix multiplies. For this work, we consider kernel hyperparameter optimization out of scope- all methods therefore use the same fixed hyperparameters. The benchmark is run on an NVIDIA A100 GPU.
Results. Figure shows the average performance of each method over five runs that use different random seeds. Once the matrix is formed in memory, the SoD approaches are very fast even with Naub 2000, they converge within 100s (all SoD runs require only two Newton steps). With incrcasing Naub: the runs reach higher accuracy at the cost of increased memory requirements. To achieve top performance, SVGP requires a large number of inducing points (at the cost of slower training). Increasing the learning rate to compensate results in instabilities these runs do not exhibit competitive performance. Both SoD and SVGP fall short of ITERNGGP-CG in all three performance metrics. Using recycling, ITERNCGP-CG maintains low loss/high accuracy throughout training, even when compression is used. reaches the lowest. final negative log-likelihood (NLL) and expected calibration error (ECE) demonstrating better uncertainty quantficatton. It is more memory-effciint than SoD (especially with compression), and, in contrast to SVGP, does not require extensive tuning.
Extension to MNIST. To demonstrate ICGPNs applicability to real-world data, we perform a similar experiment on MNIST (Lecun et al., 1998), see Appendix C.4 for details. As KEOPS scales poorly with the data dimension (D = 28² = 784 for MNIST), we revert to GPYTORCH's standard kernel implementation, which requires more memory. We thus limit the training data Naub 20,000 images. The results (Figure 7) outperforms the well-tuned SVGP basclines in terms of accuracy and NLI loss. Only the ECE of TrERNCGP is slightly worse than for SVGP. This is easily
are mostly aligned with Figure 6: ITERNCGP.CO
11

---

Published in Transactions on Machine Learning Research (04/2025)
Accuracy in % 1
NLL loss 4
ECE 4
SVGP
100
= 0.05, U = 1,000
0,0,5 U = 2,500
2.300
0.01, 10,000
80
0.05, 10,000
0.6
HGPNCGP-CG
60
2.295
$51, R=
0.4
555, m0
400
0.2
2.1
20
2.0
10-1
10
10
10a
10-1
10
10¹
10a
10-1
10"
10
10a
Runtime
in
Runtime
Runtime in 8
Figure 7: GP Classification MNIST. Comparison of SVGP with learning rate a E {0.001, 0.01, 0.05) and U E (1000, 2500, 5000, 10000) inducing points (showing only the best four runs) and TTERNCGP-CG on a classification problem with N = 20,000 training points and C' = 10 classes in terms of accuracy (Left), NLL loss (Center) and ECE (Right). Performance metrics are shown as solid (training set) or dashed (test set) lines. For ITERNGGP, the dots mark the start of a new outer-loop iteration. ITERNCGP-CG performs best in terms of accuracy and NLL loss but slightly worse in terms of ECE. Details in Appendix C.4.
explained, by the fact that one can achieve smaller ECE by accepting lower accuracy- the canonical example being& random baseline, which is perfectly calibrated.
6 Conclusion
Non-conjugate Gaussian processes (NCGPs) provide a flexible probabilistic framework encompassing, among others, GP classification and Poisson regression. Training NCGPs on large datasets, however, necessitates approximations. Our method ITERN quantifies and continuously propagates the errors caused by these approximations, in the form of uncertainty. The information collected during training is efficiently recycled and compressed, reducing runtime and memory requirements.
Limitations. A limitation of our method is directly inherited from the Laplace approximation: The inherent error in approximating the posterior with a Gaussian is not captured and generally depends on the choice of likelihood. However, under mild regularity conditions, the (relative) error of the Laplace approximation scales inversely proportional to the number of data (Bilodcau et al., 2022; Kass ct al., 1990). Since we are considering the large data regime this work one might reasonably expect the error contribution of the Laplace approximation itself to be small, relative the error contribution from approximating the MAP via Newton's method, cquivalently the error contribution of approximate GP regression.
Another limitation is the lack of a ready-to-use approach for kernel hyperparameter estimation. However, our method is entirely composed of basic linear algebra operations (see Algorithms 1 to 3). Therefore one can in principle simply differentiate with respect to the kernel hyperparameters through the entire solve. Recent work by Wenger et al. (2024) proposes an ELBO objective to perform model selection for ITERGP, which one could therefore readily apply in our setting. We leave the open question on how to optimally choose the number of Newton and ITERGP steps per hyperparameter optimization step for future work.
Future Work. So far, we have only explored the policy design space in limited fashion. The policy controls which areas of the data space are targeted and accounted for in the posterior. Tailoring the actions to the specific problem could further increase our method's 's efficiency. For classification problems, a good strategy might be not to spend compute on data points where the prediction is already definitive.
Finally, promising application for ITERNCGP may be Bayesian deep learning. A popular approach equip a neural net with uncertainty is via Laplace approximation (Khan et al., 2019; MacKay, 1991; Ritter et al., 2018), which is equivalent to GP classification problem with neural tangent kernel prior (Immer
12

---

Published in Transactions on Machine Learning Research (04/2025)
et al., 2021; Jacot et al., 2018). There, the SoD approach is regularly used (Immer et al., 2021, Sec. A2.2), for which our approach might offer significant improvements.
Acknowledgements
The authors gratefully acknowledge co-funding by the European Union (ERC, ANUBIS, 101123955). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Philipp Hennig is a member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsemciinccaatt (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC number 2064/1 - Project number 390727645; The authors further gratefully acknowledge financial support by the DFG through Project HE 7114/5-1 in SPP2298/ 1; the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FK22011S18039A) and funds from the Ministry of Science, Research and Arts of the State of Baden- Württemberg. Frank Schneider is supported by funds from the Cyber Valley Research Fund. Lukas Tatzel is grateful to the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. Jonathan Wenger was supported by the Gatsby Charitable Foundation (GAT3708), the Simons Foundation (542963), the NSF AI Institute for Artificial and Natural Intelligence (ARNI: NSF DBI 2229929) and the Kavli Foundation.
References
Blair Bilodean, Yanbo Tang, and Alex Stringer. On the Tightness of the Laplace Approximation for Statistical Inference, 2022.
Christopher M. Bishop. Pattern Recognation and Machine Learning. Springer, 2006.
Antoni B. Chan and Daxiang Dong. Generalized Gaussian process models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
Benjamin Charlier, Jean Feydy, Joan Alexis Glaunès, Crangois-David Collin, and Ghislain Durif. Kernel operations on the GPU, with autodiff, without memory overflows. Journal of Machine Learning Research, 2021.
Jon Cockayne, Chris Oates, Isse C. Ipsen, and Mark Girolami. A Bayesian conjugate gradient method. Bayesian Analysis, 2019a.
Jon Cockayne, Chris Oates, Tim J. Sullivan, and Mark Girolami. Bayesian probabilistic numerical methods. SIAM Review, 2019b.
John P Cunningham, Krishna V Shenoy, and Maneesh Sahani. Fast Gaussian process methods for point process intensity estimation. In International Conference on Machine Learning (ICML), 2008.
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stanel, Changsheng Quan, Maxim Grechkin, and William Falcon. TorchMetrics measuring reproducibility in PyTorch. Journal of Open Source Software, 2022.
Jason Frank and Cornelis Vuik. On the construction of deflation- based preconditioners. SIAM Journal on Scientific Computing, 2001.
Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU accoleration. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Rajarshi Guhaniyogi and David B. Dunson. Bayesian compressed regression. Journal of the American Statistical Association, 2015.
Philipp Hennig. Probabilistic interpretation of linear solvers. STAM Journal on Optimization, 2015.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Philipp Hennig, Mike A. Osborne, and Mark Girolami. Probabilistic mumeries and uncertainty in computations. Proceedings of the Royal Sociely of London A: Malhemationl, Physical and Engineering Sciences, 2015.
Philipp Hennig, Michael A. Osborne, and Hans P. Kersting. Probabilistic Numerics: Computation as Machine Learning. Cambridge University Press, 2022.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In International Conference on Artificial Intelligence and Statistics (ATSTATS), 2015.
Magnus Rudolph Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, 1952.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local lincarization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Robert E. Kass, Luke Tierney, and Joseph B. Kadane. The validity posterior expansions based on Laplace's method. Bayesion and Likelihood Methods in Statistics and Econometrics, 1990.
Emtiyaz Khan, Shakir Mohamed, and Kevin P Murphy. Fast Bayesian Inference for Non-Conjugate Gaussian Process Regression. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate inference turns deep networks into gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Ananya Kumar, Percy S Liang and Tengyu Ma. Verified uncertainty calibration. In Advances in Newral Information Processing Systems (NeurIPS), 2019.
Yann Lecun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
David MacKay. Bayesian model comparison and backprop nets. In Advances in Neural Information Processing Systems (NeurIPS), 1991.
David J. C. MacKay. The evidence framework applied to classification networks. Newral Computation, 1992.
David J.C. MacKay. Choice of basis for laplace approximation. Machine Learning, 1998.
lain Murray, Gaussian processes and fast matrix-vector multiplies. In Numerical Mathematics in Machine Learning Workshop (ICML), 2009.
J. A. Nelder and R. W. M. Wedderburn. Generalized linear models. Journal of the Royal Statistical Society. Series A (General), 1972.
Chris Oates and Tim J. Sullivan. A modern retrospective on probabilistic numerics. Statistics and Computing, 2019.
Michael L. Parks, Eric de Sturler, Greg Mackey, Duane D. Johnson, and Spandan Maiti. Recycling krylov subspaces for sequences of linear systems. SIAM Journal Scientific Computing, 2006.
Adam Paszke, Sam Gross, Francisco Massa, Adamn Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, perforerformecce deep learning library. In Advances in Newral Information Processing Systems (NeurIPS), 2019.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Represenlations (TCI.), 2018.
Håvard Rue, Sara Martino, and Nicolas Chopin. Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. Journal of the Royal Statistical Sociely: Series B (Statistical Methodology), 2009.
Alessio Spantini, Antti Solonen, Tiangang Cui, James Martin, Luis Tenorio, and Youssef Marzouk. Optimal low-rank approximations of Bayesian lincar inverse problems. SIAM Journal on Scientific Computing, 2015.
David J. Spiegelhalter and Steffen L. Lauritzen. Sequential updating of conditional probabilities on directed graphical structures. Networks, 1990.
Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In International Conference on Arlificial Intelligence and Statislics (AISTATS), 2009.
Brian L. Trippe, Jonathan H. Huggins, Raj Agrawal, and Tamara Broderick. LR-GLM: High-dimensional Bayesian inference using low-rank data approximations. In International Conference on Machine Learning (ICML), 2019.
Ke Alexander Wang, Geoff Pleiss, Jacob R Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact Gaussian processes on a million data points. Advances in Neural Information Processing Systems (NewrIPS), 2019.
Jonathan Wenger and Philipp Hennig. Probabilistic linear solvers for machine learning. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Jonathan Wenger, Geoff Pleiss, Philipp Hennig, John P. Cunningham, and Jacob R. Gardner. Preconditioning for scalable Gaussian process hyperparameter optimization. In International Conference on Machine Learning (ICML), 2022a.
Jonathan Wenger, Gcoff Pleiss, Marvin Pförtner, Philipp Hennig, and John P. Cunningham. Posterior and computational uncertainty in Gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Jonathan Wenger, Kaiwen Wu, Philipp Hennig, Jacob R. Gardner, Geoff Pleiss, and John P. Cunningham. Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference. In Aduances in Newral Information Processing Systems (NeurIPS), 2024.
Lijun Zhang, Mehrdad Mahdavi, Rong Jin, Tianbao Yang, and Shenghuo Zhu. Random projections for classification: A recovery approach. IEEE Transactions on Information Theory, 2014.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Supplementary Materials
The supplementary materials contain derivations for our theoretical framework and proofs for the mathematical statements in the main text. We also provide implementation specifics and describe our experimental setup in more detail.
A Mathematical Details
17
A.1 Newton's Method Sequential GP Regression
17
A.2 Our Algorithm is an Extension of ITERGP
18
A.3 The Marginal Uncertainty Decreascs in the Inner Loop
18
A.4 Virtual Solver Run
A.5 Derivatives of the Poisson Log Likelihood
21
A.6 Pseudo-Inverse of Negalive Hessian of the Log Likelihood for Multi-Class Classification
21
B Implementation Details
23
B.1 Ordering within Vectors & Matrices
23
B.2 Stopping Criteria in Algorithms 1 and 2
23
B.3 Cost Analysis of ITERNCGP
23
B.3.1 Metrix-Vector Products
24
B.3.2 Cost Analysis Algorithms 1 to 3
24
C Experimental Details
25
C.1 Binary Classification
25
C.2 Poisson Regression
26
C.3 Large-Scale GP Multi-Class Classification
27
C.4 GP Multi-Clas Classification MNIST
29
16

---

Published in Transactions on Machine Learning Research (04/2025)
A Mathematical Details
A.1 Newton's Method as Sequential GP Regression
In Section 3.1, we reinterpret the Newton itcration as a sequence of GP regression problems. More specifi- cally, we rewrite the posterior predictive mean (Equation (4)) as a GP posterior for a regression problem (Equation (6)). Here, we provide a proof for this connection.
Proposition A.1 (Reformulation of the Newton Step) Let (fi) be invertible. Using the transform g m and consequently g: S: m. the Newton step (Equation (3)) con be written
gi+1 <(K+W(())) (g: - logply ft)).
Proof. Recall from Equation (3) that
fit1 - - vV((),, with
VV((f) V log fi) K K- m) V = - K-1
where ((.) = -oolog fi) denotes the negative Hessian (with respect to f) of the log likelihood evaluated at ft. holds
fit f. -
ff+ -W(I) K
(.r....Voggge (Vlogp(y fi) K- -m))
By substracting m from both sides we obtain
g++1 9. (W(()) K (V log fi) K-
(W(f)++ ((W()) K K 9. + V logp(y fi) g-)
Vogg(y fi))
(W( (
p(y fi))
=(I+ (a) Vogp(y fi)
Voggpy f.)
KIK (" Voogpyy fi)
-K())) (g: + (( logp(y f:)).
with K
Newton's Method as Sequential GP Regression. Using the LA at f., we obtain a GP posterior (see Equations (4) and (5) in Section 2). With Proposition A.1 (ie, assuming that W(f)) exists), we can rewrite Equation (4) as
(.) = m(.) + K(,, ((, (fi-1-m
= m(.) + K(,XX)K-g 9i+1
-m()) X(, (g: - logog( f())
= =m(.) + K((,x)))((())'((f + W(f) logp(y f.) m)
= =m()) K(,, X(x m),
where ((f.) := f. + ((
f.). This proves Equation (6). Together with Equation (5), m,, defines a GP posterior for a GP regression problem with pseudo targets g(f.) observed with Gaussian noise (0, W -1)) (Rasmussen & Williams, 2006, Eqs. (2.24) and (2.38)).
17

---

Published in Transactions on Machine Learning Research (04/2025)
Equation (6) requires solving the linear system (f.) ú(fi) of size NC. Then, my,(1) ml.) + K(, x)v. In Proposition A.1, we can write Ji+1 as Di+1 Ke, i.e. fi+1 Kr + m. So, both the predictive mean m and the Newton update fi+1 follow directly from the solution v. In that sense, performing inforence and computing Newton iterates are equivalent.
What. If (fi) Is Not Invertible? For multi-class classífication, W has rank N(C- and thus W-1 does not exist. Therefore, use its pseudo-inverse Wt instead. We derive an explicit expression for Wt in Appendix A.6 which allows for efficient matrix-vecter multiplies.
A.2 Our Algorithm is an Extension of IterGP
Our algorithm TTERNCGP uses ITERGP as a core building block. ITERNCGP's outer loop (Algorithm 1) can be understood as a sequence of GP regression problems and we use ITERGP (that implements the inner loop, see Algorithm 2) for finding approximate solutions each of these problems. In the case of GP regression (i.e. with Gaussian likelihood), the outer loop collapses to single iteration and TTERNCGP show in the following.
coincides exactly with ITERGP,
Theorem A.2 (Generalization of ITERGP)
Por a Gaussian likelihood p(y f) = J, ITERNCGP converges in single Newlon slep (i.e. fi = fMAP) and ITERNCGP (Algorithm 1) coincides exactly with ITERGP (Algorithm 2).
Proof. Since the likelihood Gaussian p(y 5) = -Nyy; A,),, the log likelihood given by
1
logp(y () _ - ( y) T) -(- y) y)-
This gives rise to log-posterior (Equation (2))
(f) == logp(s x,y)
1 (f mm) K- m)
logp(y) f)
(f - v) A- - y) -f - K K- m
m)
that is quadratic in f. The first Newton iterate fi therefore coincides with bog-posterior's maximizer fi fMAP. The outer loop of ITERNCGP thus reduces to a single iteration.
How does this step look from the perspective of the ITERNCGP algorithm? First note that W(J -V log f) =- Given an initial fo, ITERNCGP computes the observation noise W W-1 (fo) A and pscudo regression targets ((f) = fo + - log fo) = fo - AA AA- y) = y. Both these quantitics are independent of the initialization fo. Thus, the first (and only) regression problem that TEERNGGP forms in the outer loop is the original regression problem (defined by labels y and the observation noise A) and ITERGP is applied to solve that regression problem. This shows that our framework recovers ITERGP in the case of Gaussian likelihood and our algorithm can thus be regarded as an extension thereof.
A.3 The Marginal Uncertainty Decreases in the Inner Loop
We claim Section 3.1 that the marginal uncertainty captured by (x, x)) e RCXC (see Equation (9)) within a Newton step decreases with each solver iteration j. Here, we provide the proof for this statement.
Proposition A.3 (The Uncertainty Decreases in the Inner Loop) For each it holds (element-wise) that diag( Kez(æ, m)) 2 diag(Ki, ( 2))) for any k 2 j and arbilrory
18

---

Published in Transactions on Machine Learning Research (04/2025)
Proof. To see this, rewrite Cj as sum of j rank-1 matrices C₁ E'=1 dedp and substitute this into Equation (9). It holds that
diag( Kig x:) diag( (ar, .))
diag( K(x,, X)d dl K(X,
=id, dd
diag(K((x,
Σ
d/
The square is applied element wise.
8=1
diag(K((x,
Σ
d
for k 3
8=1
diag( (x,,
for any X.
A.4 Virtual Solver Run
In Scction 3.3, we showed that it is possible to imitate a solver run using the previous actions on the new problem, without ever having multiply by K. The pseudo code is given in Algorithm Here, discuss the numerical and probabilistic perspective on that procedure in more detail and provide derivations for the statements in the main text.
Numerical Perspective. Let S = (.1,...,, 8B) the matrix of stacked linearly independent actions. We use C = S(S - (see Equation (10)) as an initial estimate the precision matrix in Algorithm 2. The corresponding initial residual (sce Algorithm 2) ro (g- m) Kv for the first iterate up- = Colg m) can be decomposed into Psro and (I PS) Ps S) ST is the orthogonal projection onto the subspace (Sa spanned by the actions.
Proposition A.4 (Residual in (Sa Is Zero) The orthogonal projection Psro of the initial residual ro onto span(S} is zero
Proof. It holds that
Psro Ps(ğ - m) PsKvo
Ps(ğ m) PskColj m))
Ps(ğ m) - SSS '(S K SS -S m))
-Ps
-Co
Ps(ů m) S(S S)
S (g- m
Ps
= 0.
Proposition A.4 shows that the residual in span(S) is zero. In that sense, the solution within this subspace is already perfectly identified at initialization. The remaining residual thus lies in the orthogonal complement of ( which can be targeted through additional actions. If we measure the error in the representer weights to, similar results holds, we show in the following.
Proposition A.5 (Error in Representer Weights in span(S) Is Zero) The K -orthoggonal projection of the representer weights approzimation envor Ps(v vo) onto span(S) is zero.
Proof. The Koorthogonal (orthogonal with respect to the inner product (:)))) projection onto the subspace SSa spanned by the actions is given by Ps Co K (Wenger al., 2022b, Section S2.1). It holds that
Ps(o vo) Co K vo)
19

---

Published in Transactions on Machine Learning Research (04/2025)
COKK (ÿ m) CoKCo
(ÿ ml
Kr
Colg - m) Cok CoKe
=Ps =Ps
Colg m) Co Kv
=
0,
where we used that w= -= (g m) is the solution of the GP regression linear system, vo = Co( m)) and the idempotence of the projection matrix Ps- = Ps
Probabilistic Perspective. Equation (9) describes the effect of Co from a probabilistic perspective. Initializing Co in step results in m4,0 m(.) (prior mean) and Kg,0 k(.-)) (prior covariance) since the reduction of uncertainty K(, C. ((x,)) -) is zero. This case, where no information from past steps is included, is illustrated in the first. column R 0 in Figure 4.
Special Caso. We consider a special case, where the general intricate form of the total marginal variance from Section 3.4
Tr To,,,,((X, x)) - Tr(K) - Tr(KK,
(13)
collapses. Let ANC > 0 denote the eigenvalues of K and bi..... bvc the corresponding (pairwise orthogonal) eigenvectors. We make the following two assumptions: (A1): We assume W-1 0, i.e. K. (A2): We assume that the actions coincide with a subset L S (l....,NC) of K's cigenvectors S = (bi)te. E NC
Proposition A.6 (Total Marginal Uncertainty) Under assumptions (A1) and (A2) holds that
(X, (K)) Tr(M).
Proof. Let S (br)teL NCX and A diag( RIIXILI contain subset L (1....,NC) of K's eigenpairs. The remaining eigenvectors and eigenvalues are given by S+ (bi)uer. E NCCCCC- and A+ diag((A))) IRNCCLLIXNC-HL First note that we can write the eigendecomposition of K K as sum of two components K SAS T + SSA. ST, each of which covers one part of the spectrum. It holds
S = I, S.S, = I, S =0 and S =
since K is symmetric and its cigenvectors are thus pairwise orthogonal. It follows
KS = (SAST + S,A, SA S 3 ST (SAS S.A. AS
= S KS = (SAS +SAA,S A. Plugging those expressions into Equation (13) yields
Tr(Kr,(X, X)) Tr(K) TI(KOo))
= =Tr(K)- - (SSS KS) -1 -S (T))
K)
-
Tr(K) - -USSAA-AAS
Tr(K) - TH(S SA)
TT(K - TT(A)
-Tr(K) - -Tr(M)
= > A
LEL
20

---

Published in Transactions on Machine Learning Research (04/2025)
The last equation is due to Tr(K) TH(SAS)) Tr(++A,S!) Tr(S SA) I.(S TT(A) TY(A)).
Proposition A.6 shows that the reduction of the marginal uncertainty determined by the sum of M's eigenvalues Eie, At- If contains the eigenvectors by to the largest eigenvalues (i.e. S "aligned" with the high-variance subspace of K), the remaining uncertainty Ligr. Ar is small. In contrast, if S covers the low-variance subspace of K, the uncertainty remains largely unaffected.
A.5 Derivatives of the Poisson Log Likelihood
One of our main experiments in Section 5 is Poisson regression (see Appendix C.2 for details). In order to apply ITERNGGP, we have to formulate the problem within the NOGP framework. In particular, we have to specify the derivatives of the log likelihood.
The Poisson likelihood given by
N
v exp(-An)
ply f)
I
Yn
where yn E No and X X(X) exp(s(X)) exp(f). Taking the logarithm yields
xg exp(- -dn)))
log f)
log
(un log(An) - An -log(un!))
yn!
The log likelihood's gradient and Hessian with respect to f are therefore given by
V logp(y f) = y- exp(f) and V logr(y f) - diag(exp(()),
where the exponential function is applied element-wise This implies that the log likelihood is concave which was one of the prerequisites of our algorithm (sec Scction 2.1). It follows that W(f) diag( cx)(-f)).
A.6 Pseudo-Invers of Negative Hessian of the Log Likelihood for Multi-Class Classification
For multi-class classification (sce Appendix C.3 for details), we need access to the pseudo inverse W For this be efficient, we derive an explicit form of Wt in the following and show that matrix-vector multiplies can be implemented efficiently in O(NC). Since the ordering (sce Appendix B.1) of W plays an important role in the derivation,
explicit notation in this section.
Lemma A.7 (Explicit Sseudo-Inversee for Multi-Class Classification) Consider multi- class classification, such that the log likelihood log p(y f) is given by a categorical likelihood with softmar inverse link function, then the pseudoinwerse [W e NCxN of W(f) in -orderignng is given by
(I - 11 diag( ( 111
[Wfflox=
(I- (1- diag(T f11l)
where nI = T RC denotes the output of the softmax for En, t.e. T. = =xp( E. exp(f%). The cost of one matrix-vector multiplication - [W(f)] with the pseudo-inwerse is O(NC).
Proof. By Eq. (3.38) in Rasmussen & Williams (2006), the matrix W(f) in NC-ordering is given by
[ [W]]] [ainggT))]nc II
21

---

Published in Transactions on Machine Learning Research (04/2025)
where [diag( TT) ]]]N diag( (T. .... N ................................ of. ......... x and
diag ( ,
II =
RCNN
diag(mf 11
Rewriting W()) in the CN-ordering, obtain using [diag( π JcN
diag(T)) .f.... ................................ -...... , that
T
π
[W[f]lcn = [diag(T]]oN
N
Bookkiag(ddag((m)) (T)) n. n
Now the pseudoinverse of a block-diagonal matrix is the block-diagonal of the block pseudoinverses, i.e. (ockdiag(A,)) blockdiag(AA) which can be shown by simply checking the definition criteria of the pseudo-inverse and using basic properties of block matrices. Therefore suffices to show that the block pscudoinverses are given by
(diag((T,) - ,. ).) (I - diag म. ( - 11)
for n E (1....,, NJ. We do so by checking the definition criteria of a pseudoinverse. Tet A, = diag((Tn)))))) T We begin by showing the following intermediate result:
A. I - A,
1 J (ding( T..) - , In A,
1 ) ( (.. An
(14)
Now let's verify the first criterion in the definition of the psoudoinverse. We have
A, (I _111)) diag( ग,, )) 11 An = An diag(TT An
A, diag( ,,,
(diag (T 1 TI
A, 1TT
A, (diag( (T.) T. [1
A.
where used (14). Next, we'll verify the second criterion.
1 (I c 11 diag(T n. ))
1 11 A,,
1
11 diag( n,, 1) 1
C 11
C 11
C
1 C
(I
11 diag( ,.,"" AA diag ,, )I 11
C
1
= (I - 11 (TT ) 11
where we used
diag ,. A,, I= A, "
(15)
shown above. Finally, we verify the symmetry of the product of A, and its pseudoinverse. Observe that both A, and (I - 11 diag( क. I 111T) are symmetric. Therefore we have
(A, f diag( n )I 1 1)) (I (ur)) diag(Tr, ") - )
and
((I (11- 1---1 11 diag(Tr ,,, ( (1_) A1m)),)' = An(I 11 T) diag( ,,, -)= C 11
C
C
22

---

Published in Transactions on Machine Learning Research (04/2025)
Thus if show that An and -I- ) diag(", (I δ 1 symmetry criteria of the pseudoinverse. Tt holds that
commute we have shown the remaining
A C 11)) diag( n., ' C 11 (14) A, diag( (I - [ 11 (15)
(I c =
C
well as
(I (1-1111) diag( TT, (I - ) (14) (I x dia))diagr))) n., 'A, (I (I- 1 ( 11
This completes the proof for the form the pseudoinverse. For the complexity of multiplication, note that has cost OCC), since it decomposes into two multiplications with (I - - 11 which is linear and one elementwise scaling. Therefore the cost of multiplication with the pseudoinverse consisting of N blocks has complexity (NCC).
multiplying with (1- ) dI
B Implementation Details
B.1 Ordering within Vectors & Matrices
Ordering within Vectors. By default, assume all vectors and matrices to be represented in CN-ordering, For example, the mean vector was introduced as the aggregated outputs of the mean function m: X - R for all data points m (XX)= = (m(x1)T, ..... m(xx)]) With m(x,)) T (m, m.) denoting the C' outputs for data point Fn, we can write m as m = (m), mi.,....... mi ...., m .... m.). We call that representation C:N-ordering because the superscript e moves first and the subscript n moves second. Consocutively, (m), mb.....m m............................... ..... , my ...., m corresponds to NC-ordcring.
Ordering within Matrices. The same terminology be applied to matrices, where the and columns can be represented in CN or NC-ordering, Depending on the context, different representations are beneficial. For example, in CN-ordering, W is block-diagonal (due to our iid assumption, see Section 2.1) with N blocks of size C' C on the diagonal. In contrast, when the C outputs of the hidden function are assumed to be independent of each other, K is block diagonal only in NC-ordering. So, based on the chosen ordering, different structures arise that we can exploit in subsequent computations (e.g. when we compute the inverse of W, see Appendix B.3).
B.2 Stopping Criteria in Algorithms and 2
Stopping Criterion in Am The OFTENSTOPPNCCCTTEOOO)) we use for our experiments
is based the relative change of the vector 9: f: m. When 1g, g--1ll 9 1-11-1 & 8 falls below the convergence tolerance 5 (by default, 5 1%), the loop over i terminates. Of course, other convergence criteria are also conceivable. Depending on the application one might want to customize the criterion and, for example, include the marginal uncertainty at the training data.
Stopping Criterion in Algorithm 2. We use the same INNERS TOPPINGCRTVNRI)N) as (Wenger et al., 2022b, Section S3.2): The loop over terminates the norm of the residual [ryll < max{6abes 6 9 mll is below an absolute tolerance dabs or below the scaled norm of the right-hand side - m of the linear system. By default, both tolerances iterations. The solver is also terminated when the normalization constant ЛΙ & 0. This can happen due to numerical imprecision if the linear system is badly conditioned, e.g. if some eigenvalues of the linear system are close to zero.
set to 5 Additionally,
typically specify maximum number of
B.3
Cost Analysis of IterNGGP
In this section, we investigate the computational costs of ITERNCGP in more detail. We start with a discussion of the computational costs for matrix-vent vector products with K, W-1 and C, and then analyze the runtime and memory costs of the individual algorithms (Algorithms to 3).
23

---

Published in Transactions on Machine Learning Research (04/2025)
B.3.1 Matrix- Vector Products
ITERGP is an iterative matria-free algorithm and our algorithm ITERNCGP inherits that property: The matrices K, W-1 and C₁ are evaluated lazily, i.e. matrix-veclor products are evaluated without forming the matrices in memory explicitly. This enables our algorithm to scale problems where a naive approach causes memory overflows. In Algorithms 1 to 3, the memory and runtime cost for matrix-vector products with K are denoted by μк and and by Pw-1 and Tw-1 for products with W-1
Products with K. Matrix-vector products with K can be decomposed into products with its sub-matrices. The associated memory costs ((uk) can thereby be reduced basically arbitrarily and the runtime can be improved by using specialized software libraries such as KEOPS (Charlier et al., 2021) and parallel hardware (i.e. GPUs). Still, products with K are computationally relatively expensive, since this operation is typically quadratic in the number of training data points N.
Products with W-1 (General Case). Under the assumptions the likelihood from Section 2.1, W is block-diagonal with N blocks of size C' x C (in CN-ordering, see Appendix B.1). Here, we denote these blocks by W...., WN. It can be easily verified that W-1 is also a block-diagonal matrix and the blocks on its diagonal are the inverses of WI...., WN.
Consider the matrix-vector product w W- =: w E R. In the vectors and w, we repeatodly group C consecutive entries which results in segments Wny Un BC for n 1, N, i.e.
V1
w1
W
N
UN
WN
It holds that wn = W. Un i.c. cach segment in w is the product of a single C x C' block from W W-1 with one segment from v. Computing w, thus amounts to solving a linear system of size C with cost. O(Cs). The total cost for all N segments is thus O(NC)). However, the N linear systems are independent of each other and can thus be solved in parallel. So, if appropriate computational resources are available, the total runtime complexity can be reduced to (CC)).
In general, W-1 requires ONC2) in terms of memory consumption. If needed, these costs can be reduced further to ((C²) because (as explained above), products with W-1 can be broken down into products with the individual blocks of W-1 We can perform those products sequentially such that only single block is present in memory at a time.
Products with W-1 (Special Cases). many cases, we can multiply with W-1 more efficiently. In the multi-class classification case, the runtime and memory costs for multiplication with the pseudo inverse W can be reduced O(CC) (see Appendix A.6). In the regression case (C'= = 1), W-1 diagonal matrix of size N N. The memory and runtime costs are thus in O(N). An example is Poisson regression, for which we derive the explicit form of W-1 in Appendix A.5.
Products with Cj. C2 = Q is represented via its matrix root Q, e NCxB This allows for efficient storage and meatrix-vector multiplies v CT () in O(BNC).
B.3.2 Cost Analysis Algorithms 1 to 3
The runtime and memory complexity for the operations Algorithms to 3 is given directly in the pseudo code. Here, provide some additional information for the costs that depend on the user's choices and put the costs of the individual algorithms into perspective.
Algorithm& 2 (IterGP). The runtime cost for selecting an action ( depends on the underlying policy. For Cholesky actions (s, = eÿ) or CG (s, rj-1), the runtime cost. is insignificant since no additional computations are required at all.
One iteration's 's total computational cost (without prediction) is dominated by two matrix-vector products with K in terms of runtime and O(BNC) in terms of storage requirements (for the buffers S and T as well
24

---

Published in Transactions on Machine Learning Research (04/2025)
the matrix root Q2). The initial size (i.e. the number of columns) of S, T and Qj is given by the rank limit R used in Algorithm 3. Henceforth, one column is added to each of the buffers and matrix root in each solver iteration, increasing their size to B R++ in iteration j. It is thus reasonable to include an upper bound on the iteration number in the stopping criterion of Algorithm 2.
Algorithm 3 (Virtual Solver Run with Optional Compression). The total runtime complexity of Algorithm is O(BTw 1 B2NC) i.e dominated by matrix- products involving the buffers and W-1 In terms of memory requirements, the buffers S, T, and Qo are the decisive contributors with O(RNC). The truncation of the eigendecomposition provides way to control that bound by resetting the current buffer size B to an arbitrary number R & B. In comparison to Algorithm 2, the computational costs are practically of minor importance since no multiplications with K are necessary.
Algorithm 1 (IterNCGP Outer Loop). The costs ((T)) for evaluating m on the training data depends on the choice of mcan function. For a constant mcan function, no computations are necessary, so runtime costs are negligible. This can be different e-g. for applications in Bayesian deep learning, where evaluating requires forward passes through neural network.
C Experimental Details
Throughout the paper, we use binary classification as an illustrative and supporting example (Figures 4). The two main experiments follow in Section 5: Poisson regrossion (Section 5.1, Figure 5) and large-scale GP multi-class classification (Section 5.2, Figure 6). In the following, provide additional details for all those experiments.
C.1 Binary Classification
Binary Classification with one latent function. Consider a binary classification task, i.c. C = 2. Being able to report the probability for one of the two classes is sufficient because they have to add up to one for every data point. Thus, while C'= = 2, N-dimensional vectors are typically used to describe this problem (Rasmussen & Williams, 2006, Section 3.4). Using only a single latent function is convenient for illustrative purposes, as e.g. the action vectors in Algorithm 2 are N-dimensional (not 2N dimmenssional) and thus easier to visualize.
ID Data. We use one-dimensional training set in Figure 2. X is created by sampling N = 50 data points between -3 and 5. The hidden function f is draw from GP with mean zero and a GPYTORCH (Gardner et al., 2018) RBF kernel with lengthscale 1.0 and outputscale = 5.0. For cach datapoint Ea, we sample the positive label with probability logistic[f(x..) (x.,))
2D Data. Two-dimensional data is used in Figures and 4. The data-generating process is analogous the 1D data, only now, the N = 100 training inputs are in the 2D planc: The first coordinate is sampled uniformly between 33 and 5, the second between -4 and 1. The hyperparameters of the RBF kernel are lengthscale = 1.0, outputscale = 10.0 for Figures 3 and 4 and outputscale = 20.0 for Figure 1.
Details Figure 1. In this figure, we compare two versions of our algorithm: ITERNCGP-Cho without recycling and IGERNGG with recycling and with compression (R = 10). Both runs were conducted on CPU. The computation of the NLL loss is not included in the runtime measurement. description of how the NLL loss can be computed for arbitrary C' is given in Appendix C.3.
Details Figure 2. For Figure 2, we compute sequence of precise Newton steps by using ITERNCGP with unit vector actions and & N solver iterations. Note that the Newton linear system is N-dimensional, i.c. we actually obtain f: as defined by Equation (3).
Details Figure 3. In this plot, we compare unit vector actions (ITERNCGP- and residual actions (ITERNCGP-CG) for the first Newton step (i 0) at three solver iterations j e (1, 10, 19). The true postcrior mean function Ma,s and covariance function Ku,* are computed by using ITERNCGP-CHE Chhl and j & N 100 iterations. Figure shows the covariance functions corresponding to the mean functions in Figure 3.
25

---

Published in Transactions on Machine Learning Research (04/2025)
True posterior variance Ka,-
Ko,1
Ko,10
Ko,19
IterNGGP-CG Iter Chol
Figure 8: Different Policies of IterNCGP Applicd to GP Classification. (Left) The true posterior for binary classification task
covariance K0,* covariance after 1, 10, and 19 iterations with the unit vector policy (Top) and the CG policy (Bottom). Shown are the data points selected by the policy in this iteration with the dot size indicating their relative weight. For ITERNCGP-Chol, data points are targeted one by one and previously used data points are marked with (O).
(Right) Current estimate of the posterior
The actions are visualized by scaling the training data points according to their relative weight: First, we take the absolute value of the action vector from Algorithm 2 (element-wise) and then scale its entries linearly such that the smallest entry is 0 and the largest is
Details Figure 4. In this figure, we show the initial mcan function m1,0 and covariance function K1,0 the second Newton step for different buller sizes RE {0, 1, 10). We use CG actions for the first Newton step and let the solver until convergence (this takes 19 iterations).
C.2 Poisson Regression
In Section 5.1, we apply ITERNCGP to Poisson regression to demonstrate our algorithm's ability to generalize other clogaconcave) likelihoods and to explore the trade-off between the number of (outer loop) mode-findie moddefinding steps and (inner loop) solver iterations.
Poisson Likelihood. We consider count data NY that is assumed to be generated from Poisson likelihood with unknown positive rate A: X - R,, Modeling X with a GP which may take positive and negative values, would therefore be inappropriate. However, we can use a GP for the log rate f: X - RR and rogard this as the unknown latent function. With = X(X)= = ex)(f(X)) = cxp(f), the likelihood given by
N exp(-AA) 11
N
plyl f)
yn!
The gradient and (inverse) Hessian of the log likelihood be derived in closed form, see Appendix A.5.
Data & Model. First, create X by linearly spacing N 100 points between 0 and 1. For the count data y, we sample from GP with zero mean and GPYTORCH (Gardner et al., 2018) RBF-kornel with lengthscale = 0.1 and outputscale = 5.0. That GP f represents the log-Poisson rate. We then draw counts from a Poisson distribution with rate A(x,) = exp(f(x,) for each data point in the training set. this experiment, we conduct multiple ITERNCGP runs on different training sets. These are created by re-drawing from the Poisson distributions with the same rates, i.c. the underlying GP for the log rate does not change. Our NCGP's prior uses the same RBF kernel to avoid model mismatch.
26

---

Published in Transactions on Machine Learning Research (04/2025)
IterNCGP-CG Approaches. We consider ITERNCGP-CC with four different schodules: A fixed budget of 100 iterations is distributed uniformly over 5, 10, 20 or 100 outer loop steps (see Algorithm 1), which limits the number of inner loop iterations (see Algorithm 2) to j & 20, 10, 5 or 1. For each schedule, we perform 10 runs with different training sets, see above. Each run uses recycling without compression. For this experiment, the convergence tolerance in Algorithm 1 is set to 6 = 0.001. All runs are performed on a single NVIDIA GEFORCE RTX 2080 TI 12 GB GPU.
Tracking of Performance Metrics. As performance metric, we the NLL loss. The computation of the NLL loss for the test and training set is not included in the runtime reported in the results. For the NLL loss, we approximate the integral from Section 2.3 with MC samples: For cach test datum Io, we draw 10 110", MC samples from (mm.)(x.). (Ts æo)), map those samples {fo, through the likelihood p(ye fo,s) and average. This yields a loss value for Is and we obtain the training/tent NLL loss by averaging these loss values for all data points in the training/tene /test set.
A Approximate Rate Distribution. Using ITERNCGP-CC for the Poisson regression problem results in a sequence of posteriors GP (m4.j ks.)). By drawing MC samples from those posterior GPs and mapping them through the exponential, we obtain an approxinated (skewed) distribution for the rate A. In Figure 5 (Right), we report its median and a 95 % confidence interval between the 2.5 % and 97.5 % percentile.
C.3 Scale GP Multi-Class Classification
In this experiment, we empirically evaluate ITERNCGP on a largo-scale GP multi-class classification problem to exhibit its scalability. We also investigate the impact of compression on performance.
Data. We consider Gaussian mixture problem with C' 10 classes in 3D. For each class, we sample a mean vector uniformly in [-1, 1]3 and a positive definite covariance matrix. For the covariance matrix, we first create a 3 x 3 matrix C' with entries between 0 and 1 (sampled uniformly) and compute the eigenvectors U of CCT. Then, we create three eigenvalues (1.23) uniformly between 0.001 and 0.1 and form the covariance matrix from the eigenvectors U and these eigenvalues, i.e. U diag(>1. d2, As) UT. For each class, 10 data points are sampled from the respective Gaussian distribution. This amounts to N = 10 data points in total. For testing, No = 10 data points are used (10$ per class).
Our benchmark (Figure 6) uses multiple runs for each method. The runs differ in the seed that is used sample from the Gaussians, i.e. the training and test set are different for each run (the underlying Gaussians distributions remain the same). Both the training and test set used in the first run are shown in Figure 9.
Model. We use a softmax likelihood (see Appendix A.6 for the details on the pseudo inverse W and assume independent GPs for the C' outputs of the latent function. Each of these GPs uses the zero function as the prior mean and a Matórn(?) kernel. We use the KEOPS (Charlier et al., 2021) version of the GPYTORCH (Gardner et al., 2018) kernel with lengthscale 0.05 and outputscale = 0.05.
SoD Approaches. For the SoD approaches, we create a random subset of the training data (sampling without replacement) of specific subset size Neub- We then explicitly form (f,,) for every Newton step and compute its Cholesky decomposition via PYTORCH's (Paszke et al., 2019) torch. lilalllgllgg.. choleskky (instead of using ITERGP in Algorithm 1 to ensure a competitive baseline implementation). In our experiment, we use four different subset sizes Nsub E (250, 500, 1000, 20003. Each setting is run five times with different random seeds (see above).
SVGP. SVGP (Hensman et al., 2015; Titsias, 2009) is commonly used variational method for approximative inference in non-conjugate GPs. We use GPyToRcH's (Gardner et al., 2018) SVGP implementation and optimize the ELBO for 10 seconds using ADAM with batch size 1024. The learning rate e (0.001, 0.01, 0.05) and the number of inducing points Ue (1000, 2500, 5000, 10000} are tuned via grid search (using only a single run). We use U/c inducing points per class (C- = 10 classes) and initialize them as a random subset of the training data. Within the given runtime budget, SVGP performs between 6000 (U = -0000) and 600 (U = 10000) epochs.
For each of the 12 runs, we extract 6 performance indicators: lowest training/test testt NLL loss during training, highest training/tes /est accuracy during training and training/tes expected calibration error (FCF) at the very
27

---

Published in Transactions on Machine Learning Research (04/2025)
Training data (C 10, N 100,000)
Test data (C 10, 10,000)
1.0
0.5
0.5
0.0
0.0
0.5
0.5
1.0
1.0
0.5
0.5
-1.0
0.0
-1.0
0.0
-0.5
-0.5
0.0
0.5
0.0
0.5
0.5
1.0
1.0
Figure 9: Gaussian Mixture Training and Test Data. Training data (Left) and test data (Right) for the first run. Note that the underlying Gaussians are the same for training and test set and for all runs.
end of training. For each of those 6 categories, we determine those two runs with the best performance. This procedure results in set of 3 runs in total, that are considered the "best" runs for SVGP. Only for those 3 settings, perform 5 runs each and report their mean performance in Figure 6.
NCGPCGG Approaches. For comparison, we apply our matrix-free algorithm ITERNCGP with residual actions to the full training set. We use two configurations: The first one uses recycling without compression (i.e. R = oo), the second one uses recycling with compression (R 10). The number of solver iterations per step is limitod by j & For both scttings, we perform runs with different random sceds (sce above) and report. their mean performance in Figure 6.
Tracking of Performance Metrics. As performance metrics, (NLL) loss and the expected calibration error (ECE) (Kumar et al., 2019) on both the training and test set. The computation of those six metrics is nol included in the runtime reported in the results (Figure 6).
accuracy, the negative log likelihood
First, we compute the predictive mean my,( and marginal variance diag( (K,, xx)) (see Equations (8) and (9)) for each test input æ. Then, we use the probit approximation (MacKay, 1992; Spiegelhalter Lauritzen, 1990) obtain the predictive probabilities
mig (x))
no softmax
R.C
+ diag Ki (o xx)))
where the vector division is defined clement-wise. This is an approximation of the intogral from Section 2.3. All three performance metrics are based on the predictive probabilities.
Accuracy. The prediction for H. is given by arg (axo,( i.e. by the class with the largest predictive probability. The accuracy is defined the ratio of correctly classified data.
NLL Loss. The NLL loss for x. is defined the log-probability for the actual class you i... log([Toly.). We obtain the NLL training and test loss by averaging the individual loss values for the entire training/ter test set.
ECE. The expected calibration error (ECE) (Kumar al., 2019) is a measure for the calibration of the predictive probabilities. It groups the probabilities of the predicted classes (i.e. the classification confidences) into bins and, within these bins, compares the average confidence with the actual
28

---

Published in Transactions on Machine Learning Research (04/2025)
accuracy. We use haliicllas@alllbrtttooooroo from torchmetrics (Detlefsen et al., 2022) with default parameter n binn==11
Individual runs. Figure 6 shows the average performance for each of the nine methods/vard variants over five runs. In order to show, that the observed performance differences are not due to chance, we show the individual runs in Figure 10.
Accuracy in %
NLL loss 4
ECE +
SoD
Naub = 250
2.25
Nabb 500
Nub = 1,000
2.00
Noub = 2,000
SVGP
1.75
80
0.4
0=0.01,, ==5,000
0.001, U=0,000
1.50
0=0001 U 10,000
78
1.25
0.2
IteNCGP-CG
dos
555, R=oo
1.00
8g5, 5, R=
109
10
10a
10a
10'
102
10"
10.
102
10ah
10*
Runtime in
Runtime s
Runtime s
Figure 10: Large-Scale GP Classification. Same as Figure 6, but showing all runs individually (instead of just their average).
C.4 GP Multi-Class Classification on MNIST
To demonstrate ITERNCGP's applicability to real-world datasets, we perform an additional experiment (similar to the experiment described in Appendix C.3) on a subset (Neub 20,000) of the MNIST (Lecun et al., 1998) dataset. In the following, we describe the experiment and results more detail.
Remark. For the experiment on synthetic data, we use KEOPS on top of GPYTORCH for fast kernel-matrix multiplies. However, KEOPS scales poorly with the data dimension (D 28 784 for MNIST) We therefore revert to GPyTORCH's standard implementation. This implementation of the kernel matrix-vector product is fast but causes out-of-memory errors for large datasets, which is why we limit our benchmark to Naub 20,000 training data. To fully realize the potential of our method using KEOPS, it might be advisable to apply it only to problems with data dimensions smaller than around 100.
Data & Model. We use 20,000 training and 10,000 test images from the MNIST datasct and the softmax likelihood. Our model for the latent function is multi-oulput GP which C 10 independent GPs, each of which is equipped with Matérn(?) kernel.
Kernel Hyperparameters. As first step, we determine suitable hyperparameters (the outputscale and lengthscale) for the Matérn(?) ( kernel by running GPYTORCH's SVGP implementation. We use 1000 inducing points per class, a batch size of 1024 and optimize the ELBO using ADAM with learning rate 0.001 for 30 epochs (this results in lengthscale hyperparameters with SVGP may give SVGP an advantage what performance can reach, making it a competitive baseline.
1.559934 and outputscale 0.451191). Note that choosing
SVGP and IteNNGP--G Approaches. We compare ITERNGGP-CG and SVGP both using the same fixed kernel hyperparameters (see above). TTERNCGP-CC is applied with recycling and R = cc, i.e. without compression. We exclude compression since both ITERNCGP runs converge within three iterations, see below. The number of inner loop iterations is limited by j & or j & 5, i.c. two runs are porformed. For the SVGP approach, we optimize the ELBO using ADAM with batch size 1024 for 200 seconds. The number of inducing points and the learning rate are tuned via grid search. As in Appendix C.3, we use
benchmark hatisk_hhhhh___ighhge dimension. html (accessed May
2024)
29

---

Published in Transactions on Machine Learning Research (04/2025)
U e (1000, 2500, 5000, 10000) inducing points and three different learning rates (0.001, 0.05). All 14 runs are performed on a single NVIDIA GEFORCE RTX 2080 TI 12 GB GPU.
Results. The results are shown in Figure 7. They show the two IEERNCGP runs and the best four SVGP runs (these include the best two runs for each of the six performance metrics training/ test accuracy/NLL/BCEB Our observations are mostly consistent with the results on the synthetic data (Figure 6): Both TTERNCGP runs significantly outperform the best SVGP runs in terms of NLL loss and accuracy. Only the ECE achieved by ITERNCGP is slightly worse than for SVGP. However, explained in Section 5.2, small ECE on its own is not conclusive since easily construct classifier with perfect ECE by randomly sampling predictions.
30


Reviews:
[{"review_id": "TEMPhUf5Mv", "reviewer": "Reviewer_uNBX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n1. I think the paper is a neat idea, and I quite like the connection of GPs with optimization. \n2. Generally speaking, the paper is well written.\n3. The results show that your method is considerably faster compared to typical sparse variational approaches, which is nice to see.\n\nWeaknesses\n---\n1. To understand the paper, one needs to be first quite familiar with Wenger's paper. It would be better to write this paper without totally assuming familiarity, otherwise the paper reads as a further appendix to Wenger et al. instead of a stand-alone paper. This is related to the next point as well.\n2. Section 3 is not so easy to understand, particularly in terms of the language used regarding \"actions\" and \"policy\". I understand this is the terminology used in the Wenger paper but it doesn't seem that helpful (in my opinion) to allude to reinforcement learning if none of your applications are in RL. I would introduce the concepts of action and policy, along with the examples in Sec. 3.2 first before introducing Eq. 10.\n3. If you look at the GP from the weight space view, it seems straightforward to obtain a Laplace approximation in the non-Gaussian case. Why did you not do this? Especially since you mention the Trippe et al. paper is a a weight space-variant of your paper,\n4. It seems like you use $i$ and $j$ to be iteration and policy indices as well as the total number of iterations and policy size in Sec. 3? (ex: $S_j = (s_1, \\ldots, s_j)$ has rank $j$). But then it seems you use $B$ to denote the policy size later? Please be consistent with the notation.\n5. Why do you need to further compress the $S$ and $T$ further to have rank $R$ if the computation in Alg. 1 is dominated by the eigendecomposition of a user-selected size $B$ matrix? If the size $B$ is already an issue, shouldn't you just set $B$ to be whatever $R$ would be?\n6. The method seems more involved than how it is presented in Alg.1, and many of the important details are pushed to the Algorithms in the appendix. I think they should be moved to the main text and you should spend some time guiding the reader through the implementation. Moreover, the grey subtitles of Alg. 3 are not super clear to me as to what they really mean so it would be helpful to explain some of the matrices and vectors defined here a little bit more. \n7. I would be interested to see how this method fares on real data classification problems with a considerably large number of classes (CIFAR-100, for example) in comparison to the other GP methods, since the class size seems to cause problems here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a method for accelerating Laplace approximations of GP models by extending IterGP for non-Gaussian likelihoods. The speedup comes from using matrix-free computation techniques in the optimization literature and posing Newton's method as a form of GP regression.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer to the weaknesses in the previous section. In summary, I think the paper is quite nice but the authors should spend some time rewriting Sec. 3 to be a more gentler transition from introducing the problems of non-conjugate GP regression to introducing their method.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OaWlCUnpwl", "reviewer": "Reviewer_Fjue", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The method is well inspired and shows the benefit of incorporating additional sources of uncertainty into GP modeling.\n- Connecting Newton steps with sequential GP regression is interesting and can even open new avenues of exploration for different methods using Laplace approximation.\n- The arguments of the authors are well-supported and connected to literature in numerics and probabilistic methods, thus increasing the overall familiarity of the community with these disciplines.\n\nWeaknesses:\n\n- The paper is overall well-written; however, I believe key parts could be more clearly explained to better convey what exactly the authors are proposing. I also believe that a short review of IterGP may be warranted in the background section as the proposed method is so heavily based on it.\n- It is well-known that the perfomance of GP methods heavily depend on the hyperparameters of their kernels and likelihoods. Most of the mainstream methods do hyperparameter selection using fast gradient-based optimization, instead of costly grid/random search with cross-validation. In the MNIST experiment, the hyperparameters estimated using the ELBO of variational methods is copied over to be used in IterNCGP and no clear way to estimate these hyperparameters using only IterNCGP is given. This limitation should be better communicated to the reader as it might hinder the applicability of the method in practice.\n- The empirical evaluation of the method does not go significantly beyond synthetic data. A short additional experiment on a subsampled 30% of MNIST is only shown in details in the appendix. As an example, the paper \"Large-Scale Cox Process Inference using Variational Fourier Features\" by John and Hensman (2018) includes datasets with an order of magnitude more data in lower dimensions while using well-known variational methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose an extension to the IterGP method (Wenger et al., 2023) for likelihoods that are not Gaussian. Their method is based on the Laplace approximation, obtaining a Gaussian approximation of the posterior centered on a mode of the true posterior, and uses Newton's method to find an approximation to a mode. As an extension of IterGP, the authors manage to view this procedure under the lens of probabilistic numerics, allowing the numerical approximations to contribute to the posterior variance of their approximation, therefore increasing the quality of the approximated uncertainty quantification.\n\nThe first contribution of the authors is to show the identification between performing the Newton steps of MAP estimation and the posterior computation of the approximate GP posterior for each step. Under this identification, the IterGP framework can be applied at each Newton step, thus adding a contribution to the posterior uncertainty that comes from approximating the matrix inversions required in the steps. The authors formalize this and show their derivations in the appendix.\n\nTheir second contribution comes from the notion of \"recycling\" and \"compression,\" where some computation required in each Newton step can be reused between iterations and, moreover, can be compressed by truncating the SVD decomposition of the recycled computation.\n\nThe authors test the correctness and speed of their algorithm in two synthetic datasets and show promising results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Possible typo in Section 2.3: Equations (4) and (5) might have problems with indices. $\\boldsymbol{f}\\_{i+1}$ should probably be $\\boldsymbol{f}\\_i$ in Eq. (4) since it's stated that $\\boldsymbol{f}\\_{i}\\approx\\boldsymbol{f}\\_{\\mathrm{MAP}}$.\n- The paragraph \"Newton's Method as Sequential GP Regression\" is hard to understand: As better explained in Appendix A.1, it is the case that the posterior under iteration $i$ and the computation of the next iteration of the Newton step are obtained by computing the representer weights of the fictitious regression problem with targets $\\hat{\\boldsymbol{y}}\\_i$. Unless this information is presented, it is not immediately clear what is meant by equating Newton's Method to computing the posterior sequential GP regression, after all, for NM, one obtains a vector $\\boldsymbol{f}\\_{i+1}$ while the other gives a distribution.\n- Figure 6 is missing a line for $R=\\infty$. I couldn't find the metrics for the light-colored blue line, and the text does not clarify why it is missing from the figure.\n- [Critical] More explicitly acknowledge that the current methodology does not provide a clear recipe for kernel hyperparameter estimation in the limitations.\n- [Critical] Incorporate MNIST experiments in the main paper. As the authors still have two more pages of content available, I believe there is no reason for the results of this experiment to be mentioned in the main paper but not included in it.", "broader_impact_concerns": "I believe this work does not require a broader impact statement, as I could not think of ethical issues that could be raised with it's deployment and application.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DR10VPA3NA", "reviewer": "Reviewer_oipv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe contribution to Gaussian processes is significant, providing extremely efficient computation in GPC/non-conjugate GPR inference. The approach is elegant and theoretically justified, and the mathematical derivations and proofs in the appendices are clear and easy to follow. The paper is generally well-written and does a good job of explaining some complex ideas, even if some non-standard background is necessary.\n\n## Weaknesses\nOverall, this is a very nice paper; I think addressing the weaknesses below would strengthen the paper, but did not find any critical issues.\n\n**No Proposed Hyperparameter Selection**\nThe issue of hyperparameter selection is treated as out-of-scope, but this seems like a rather big hurdle for the application of the proposed method. This is especially relevant as simultaneously learning the hyperparameters and variational distribution is an advantage of SVGPs when considering \"performance vs. time\" in a more realistic scenario that includes hyperparameter optimization.\n\nWhile I think this work has plenty of theoretical merits without including hyperparameter selection, this is a notable practical issue and merits some discussion. Even if no satisfying approach is immediately available, highlighting this as future work might be worthwhile. \n\n**Presentation of Algorithms**\nThe main algorithms, the inner and outer loop of IterNCGP, are only presented in the appendix. This caused some page-turning for me in an otherwise smooth read, as the algorithms are often referenced in the main text. I think it would improve the presentation to include Algorithms 2 and 3 in the main text.\n\n**Figure 1 Placement**\nI found Figure 1 more confusing than I found it motivating before reading the rest of the paper. In particular, the inner loop-outer loop structure is not yet discussed, which makes $i$ and $j$ somewhat difficult to understand. It is also not yet explained that the method approximates the Laplace approximation, which makes \"Newton steps\" opaque. I would encourage the authors to find ways to make this \"marquee\" figure a bit more approachable, either by changing the figure or by explaining the algorithm structure at a high level in the introduction. \n\nEven something relatively brief like \"the algorithm consists of an 'outer loop,' which iteratively solves approximate Newton steps of the Laplace approximation via an 'inner loop,' which uses probabilistic numerics to account for the uncertainty in computationally-limited solving\" would help in understanding what \"the panels show the marginal uncertainty over the latent function at Newton step $i$ and solver iteration $j$\" means.\n\n**Baseline Methods** \nWhile I find the experiments convincing, I think they could be made stronger by comparing IterNCGP to some more modern methods. For example, incorporating natural gradients in various ways may result in significantly better time vs. performance curves (Khan & Lin, 2017; Salimbeni et al. 2018). Using natural gradient descent, in particular, is already implemented in gPyTorch and should be relatively easy to test against. Besides natural gradients, some other works report substantial improvements to the wall-clock time of GP training (e.g., Dutordoir et al. (2020)), though I understand these are a bit more specialized. It would be interesting to consider the performance gains relative to these methods.\n\n**Limitations of the Laplace Approximation**\nThe accuracy of the Laplace approximation fundamentally limits the accuracy of the proposed method. While this is nicely acknowledged and briefly discussed in the conclusion, it would perhaps be beneficial to discuss this a bit more, especially since some older results suggest the Laplace approximation to be quite poor in the context of small-scale GPC (with Kuss et al. (2005) going so far as to say \"the Laplace approximation is so inaccurate that we advise against its use, especially when predictive probabilities are to be taken seriously\"; also see Nickisch & Rassmussen (2008)). I think the experimental results of this paper show this is not necessarily a concern when in the modern \"big data\" regime, but this could be pointed out as counter to the traditional logic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes IterNCGP, a method for computation-aware non-conjugate Gaussian process regression via the Laplace approximation. The method first interprets the Laplace approximation for GPs as sequential GP solves, then applies the recently proposed IterGP for computation-aware solving. Between the Newton steps of the Laplace approximation, a \"virtual solver\" is proposed to precondition linear solves using previously explored actions, providing significant performance boosts. To further reduce computational and (mostly) memory costs, the use of a truncated representation of previous actions is justified and used. The proposed method is then applied to Poisson regression and multi-class classification, where it shows extremely time-efficient performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical\n- My only critical change requested is in the abstract, where it is claimed that \"our method significantly accelerates training compared to competitive baselines [..]\". I understand that \"training\" is an overloaded term in GPR, but am afraid this claim could be misinterpreted as including hyperparameter optimization. I believe this is demonstrated only when \"training\" means \"posterior (predictive) inference,\" and the claim should be disambiguated accordingly.\n\n## Minor\n- Please consider changing Figure 1, or otherwise explaining the structure of the algorithm in the introduction, and including Algorithms 2 and 3 in the main text.\n- Please consider adding some discussion regarding hyperparameter selection.\n- Please consider including further baselines, such as the SVGP with NGD.\n- Please discuss some of the limitations regarding the Laplace approximation.\n\nI also have some smaller comments:\n- Equation (4) uses $f_{i+1}$, but I think this should be $f_i$.\n- Footnote 4 uses \"Eqn. (10)\", which is written as \"Eq. (10)\" in other places.\n- The outer and inner loops use (arbitrarily different) different`StoppingCriterion()` functions, but this is not reflected in pseudocode. Semantically, it seems more correct to specify, e.g., `OuterStoppingCriterion()` and `InnerStoppingCriterion()`.\n- On page 6, the sentence \"This virtual solver run [...] orders of magnitude cheaper, than running the solver [...]\" shouldn't have a comma.\n- In Appendix B.1, \"$CN$-orderingbecause\" is missing a space.\n- In Appendix C.3, for the probit approximation, Spiegelhalter & Lauritzen (1990) is also a standard reference.\n\n# References\n\nDutordoir, V., Durrande, N., & Hensman, J. (2020, November). Sparse Gaussian processes with spherical harmonic features. In International Conference on Machine Learning (pp. 2793-2802). PMLR.\n\nKhan, M., & Lin, W. (2017, April). Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models. In Artificial Intelligence and Statistics (pp. 878-887). PMLR.\n\nKuss, M., Rasmussen, C. E., & Herbrich, R. (2005). Assessing Approximate Inference for Binary Gaussian Process Classification. Journal of Machine Learning Research, 6(10).\n\nNickisch, H., & Rasmussen, C. E. (2008). Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9(10), 2035-2078.\n\nSalimbeni, H., Eleftheriadis, S., & Hensman, J. (2018, March). Natural gradients in practice: Non-conjugate variational inference in Gaussian process models. In International Conference on Artificial Intelligence and Statistics (pp. 689-697). PMLR.\n\nSpiegelhalter, D. J., & Lauritzen, S. L. (1990). Sequential updating of conditional probabilities on directed graphical structures. _Networks_, _20_(5), 579-605.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I and the reviewers agree that the paper presents a well-thought-out idea with clear practical benefits in scalability. The reviewers feel the authors have done a good job of improving the clarity of the paper. While the reviewers noted that they would like to see more datasets and baselines in the experiment, the benefit of the method is sufficiently demonstrated and I would not make this a condition of acceptance.
Decision: Accept (poster)