On Sketching for Gaussian Process Regression with New Statistical Guarantees
Authors: Jayesh Malaviya
Rachit Chhaya
Anirban Dasgupta
Supratim Shit
Venue: Accepted by TMLR ()

Abstract: The cubic computational complexity of Gaussian Process Regression (GPR) with respect to the number of data points is a major bottleneck to its scalability. While various approaches have been proposed to address this, few come with provable guarantees. Inspired by the success of ridge leverage score based sampling in scaling kernel ridge regression El Alaoui & Mahoney (2015), we propose a sketch-based approximation for GPR using ridge leverage scores. We provide theoretical guarantees on the approximation of the predictive mean, predictive variance, and negative log-marginal likelihood in this setting. To the best of our knowledge, these are the first theoretical guarantees for approximating the predictive variance and negative log-marginal likelihood of GPR using ridge leverage score sampling. We further show that a carefully constructed sketch of the kernel matrix preserves key statistical properties of the full GPR model with high probability. Our theoretical results are supported by empirical evaluations on real-world datasets, demonstrating strong trade-offs between accuracy and efficiency.

Published in Transactions on Machine Learning Research (03/2026)
On Sketching for Gaussian Process Regression with New Statistical Guarantees
Jayesh Malaviya*
moloniya jatgs/jjtyyyssgtg......
Deportment of Computer Science & Engineering Indian Institute of Technology, Gandhinagar
Rachit Chhaya
rachit mechtt_chhayaadaiice
Department of Computer Science & Engineering DA-IICT, Gandhinager
Anirban Dasgupta
anirbandy@Hitgm.cc
Department of Computer Science & Engineering Indian Institute of Technology, Gandhinagar
Supratim Shit
supratim Srpnatim@iiid..co oc
Deportment of Computer Science & Engineering HIT, Delhi
Reviewed on OpenReview: https:
// openreview. net/ form?? id= Nourhyu VEu
Abstract
The cubic computational complexity of Gaussian Process Regression (GPR) with respect to the number of data points is a major bottleneck to its scalability. While various approaches have been proposed address this, few come with provable guarantees. Inspired by the success of ridge leverage score based sampling in scaling kernel ridge regression El Alaoui & Mahoncy (2055), propose lkstch-based approximation for GPR using ridge leverage scores. We provide theoretical guarantees the approximation of the predictive mean, predictive variance, and negative log-marginal likelihood in this setting. To the best of our knowledge, these are the first theoretical guarantees for approximating the predictive variance and negative log- log-marginal likelihood of GPR using ridge leverage score sampling. We further show that carefully constructed sketch of the kernel matrix preserves key statistical propertics of the full GPR model with high probability. Our thcoretical results are supported by empirical evaluations on real-world datasets, demonstrating strong trade-offs bet accuracy and efficiency.
1 Introduction
Gaussian Process Regression (GPR) fundamental method in probabilistic machine learning, offering a principled non-parametric approach to modeling distributions functions Rasmussen & Williams (2005). Its strength lies in its ability to provide calibrated uncertainty estimates, which are critical in applications
such Bayesian optimization Xu et al. (2024), active learning Kapoor et al. (0077: Schreiter et al. (2015)
Tebbe et al. 2024), and reinforcement learning Biyik et al. 2020).
Gaussian Process Regression. Given a training dataset D { m where x; E Rd and y E R, we assume the oulputs are generated from a latent function f GP (m(m)))))))m)mm))))))))()))) ko(;,)), corrupted by Gaussian noise, i.e.,
8: = f(x) + Es. & ~~N(0,03))
Corresponding author.

---

Published in Transactions on Machine Learning Research (03/2026)
Here, m(-) is the prior mean function (often taken constant), and k.(:.-) is positive-defiitt kernel function parameterized by hyperparameters 0. A commonly used choice for the kernel function is the Radial Basis Function (RBF) kernel, defined as,
(-
x'12 [-1x- (-1x-~>'1
ke ko(x, = 03 exp
(1)
262
where of controls the variance and & is the lengthscale hyperparameter.
The prior over latent function values f ((, [Irn]] is multivariate Gaussian,
f~)((m(X); K)
where E R is the kernel matrix with entries Kiz = ka mz), and m(X) is the vector of prior mcans evaluated at training inputs.
Under this model, the noisy observations y R" are distributed as
y N(((mXX), o
Prediction at Test Time. For a new test point xx, the predictive distribution of y. conditioned on the training data also Gaussian,
yo x.. X, y ~ (N((w.)), Var(z.)) where the predictive mean and variance are respectively given as,
p(x.) = k. (K 1) Var(x.) ((),, - k (K I)
(2)
(3)
with k, =
Learning via Marginal Likelihood. Given the data, the task in GPR is to learn the kernel hyperpa- rameters / and noise variance of This typically done by maximizing the log marginal likelihood of the observed outputs,
1
logp(y X, = y (K+ + log det(K + + log 2n
(4)
2
This objective balances data fit. (first term), model complexity (second (erm), and normalization.
Computational Challenges. The exact computation of equation 2 to equation 4 requires (n)) time and O(n²) memory due to the inversion and determinant of the full kernel matrix Rasmussen & Williams (2005). This limits the applicability of standard GPR to small or moderate-sized datasets. Tn this work, we address this scalability bottleneck through a sketching-based based approximation using ridge leverage scores.
To handle this problem, a rich line work has focused on approximating the kernel matrix using techniques such as inducing points Snelson Ghahramani (2006), Nyström methods Williams & Seeger (2001) and randomized sketching El Alaoui & Mahoney (2015); Pilanci Wainwright/ (2017). Among these, sketching methods stand out for their ability to compress large kernel matrices into compact representations with statistical guarantees. However, existing analyses often fail to characterize the precise impact of sketching uncertainty quantification and negative log marginal likelihood of the gaussian process regression. El Alaoui & Mahoney (2015) successfully applied the ridge leverage score based sampling technique to Nystrom approximation for kernel ridge regression. However is non-trivial to extend their method to the case of GPR with provable guarantocs spocifically for the predictive variance and negative log marginal likelihood approximation. Inspired by their method, we next describe the Nystrom Sketching for Kernel Approximation specifically for GPR and then describe main contributions in this paper.
2

---

Published in Transactions on Machine Learning Research (03/2026)
2 Nystrom Sketching for Kernel Approximation in GPR
To overcome the above computational challenges, we employ Nystrom sketching, which provides a low-rank approximation K to K while preserving its essential spectral structure.
Nystrom approximation.
Let J c (I,..., n be a set m < n sampled indices. Define
C K E
W
KJJJ Rm
(5)
The classical Nystrom approximation is given by
= CWtC
(6)
where Wt denotes the Moore-Penreee pscudoinverse of W. This approximation projects the kernel matrix onto the span of the sampled columns, yielding rank-m surrogate of K.
Generalized sketching. More generally, let S E Raxm be tall, skinny sketching matrix (e.g., sampling matrix, random projection, or structured transform). The Nystrom approximation associated with S can be written
Rs = KS (SS K K
which recovers equation 0 when corresponds to column sampling.
Woodbury expansion for efficient solves. In GPR, inference requires computing (K + - where
of is the noise variance. Replacing K with K and applying the Woodbury identity with A = o. U = C,
M W-1, V = CTT, we obtain
- oe C(W + -0- (K + 11 oe --21-00
(8)
Lemma 1 (Woodbury Nystrom Solve). Let K = CW-ICT be the Nystrom approzimation of K. Then for
any R",
(K (K+0ol) = og 2 0g +C(W + to, 202 -2
99)
Proof. Apply the Woodbury matrix identity (4 + UM+ A- 11 A L U C, = W-1, CT.
with
This expression involves inverting only
matrix, substantially reducing the computational burden.
Predictive mean using Nystrom. The GPR predictive mean for a test point x, with kernel vector
k. E Rd is
Px K. -
(10)
Using the Nystrom surrogate and equation
g this becomes
oe k. og A C(W + + cTy
(11)
Analogous derivations yield similar reduction for the predictive variance.
Computational complexity. Constructing C' requires (nn)) kernel evaluations and storing it uses O(n))
memory. Forming and inverting A = W Woo-20T costs O(nm² +m²), and each test prediction costs (m²)).
This is dramatic improvement over the O(n3) time and Onn²) memory required for standard GPR.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Our Contributions. The main challenge in applying the sketched Nystrom method to GPR lies in designing efficient sketching matrix and analyzing its impact on the predictive mean, predictive variance, and negative mrg-maaaiaa likelihood (NLML). While sketching methods particularly those based on ridge leverage scores (RLS) have been well studied for Kernel Ridge Regression (KRR) El Alaoui & Mahoney
(0155; Rasmussen Williams (2005); their theoretical guarantees do not fully capture the unique proper
of GPR. KRR analyses typically bound the statistical risk of the point predictor El Alaoui & Mahoney (2015), whereas GPR's strengths lie in uncertainty quantification via predictive variance and model selection through NLML Hensman et al. (2013). These quantities, fundamental to GPR, have no direct analogues in KRR; the GPR posterior variance is distinct from the variance of the KRR estimator, and NLML is critical for hyperparameter learning. Our work bridges this gap by extending guarantees for the predictive mean and, more importantly, providing the first explicit approximation bounds for the predictive variance and NLML under RLS sketching, establishing a complete theoretical foundation for scalable, high-fidelity GPR.
Clarifying the scope of novelty. While we adopt ridge leverage score (RLS) sampling as a kernel approximation primitive, our theoretical results do not follow from prior KRR analyses El Alaoui & Mahoney 2215 In contrast to KRR, GPR requires controlling predictive variance (inverse quadratic forms) and NLML (log-determinant terms), which are absent in prior work. Addressing these quantities necessitates new spectral perturbation arguments. Thus, our contribution provides the first theoretical guarantees for RLS-based sketching in full Bayosian GPR.
To summarize, the main contributions of this work are as follows:
We propose a kernel sketching framework based on ridge leverage scores for Gaussian Process Regression (GPR).
2. We provide, to the best of our knowledge, the first theoretical guarantees for ridge leverage score based sketching specifically for the GPR problem. Specifically, we derive non-trivial bounds on the approximation error for the predictive mean, predictive variance, and negative loglliiel
3. We conduct extensive empirical evaluations across multiple real-world regression benchmarks, demon- strating the effectiveness of our method compared to standard baselines.
It is important to note that while our approach is comparable to some state-of-the-art methods in terms of runtime performance, we demonstrate that it achicves provable superior predictive quality and uncertainty calibration, thereby offering an accurate and efficient alternative to existing scalable GP techniques.
3 Related Work
Various methods have been applied to scale GPR for the big data regime; see Liu et al. (2020) and references therein. There are methods based on variational inference Hensman et al. (2013), and conjugate gradient based iterative methods Artemev et al. (2021) However, handling GPR using sampling sketching-based methods with theoretical guarantees is relatively less explored. Hayashi et al. (2020), using a novel graphon-based analysis, derive error bounds for Gaussian process subsampling via uniform random selection. However, their bounds decay slowly; for example, predictive error scales like Ollog 1/4 8) with the number of subsamples s. Fiedler et al. (2021) provide practical bounds on GPR in general; however, these bounds are not directly comparable ours.
Nystrom opprozimation Williams & Seeger (2001) reduces the computational complexity by projecting the full kernel matrix onto subspace spanned by set of inducing points. However, uniform heuristic-based selection of these points often fails to capture critical data-dependent structure, especially in high-dimensional or non- non-uniform settings.
This has led to the adoption of more principled sampling techniques based on ridge leverage scores El Alaoui & Mahoney (2015), which offer spectral guarantocs and have been successfully applied in kernel ridge regression Rudi & Rosasco| (2015); Musco & Musco (2017) and randomized matrix approximation Drincas et al. 2012). Despite their theoretical appeal, ridge leverage based Nystrom approximations have not been
4

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm of Ridge Leverage Score Sampling with Rescaling for Kernel Sketching
Kernel matrix K E R=*" noise parameter of > 0, sketch size m < n, Nystrom regularization parameter
Input:
Output:
Sketching matrix S E Rax" sketched kernel L, Rux
1: Compute ridge leverage scores: computation
[K(K + o for all i E (1....,n) nn D See Algorithm for fast.
2: Normalize scores: p; + (
3: Initialize S E Raxm as a zero matrix
for j j= to do
5: Sample index ij Categorical(pt..... Pr))
b Sample with replacement
6: Set Siy. + mmppi
b. Apply reweighting
end for
Compute sketched kernel: by KS(S (S K K
return S, Ly
widely explored in the context of Gaussian Process Regression particularly with respect to predictive quantities such as the posterior mean, variance, and marginal likelihood. Existing bounds are directly applicable GPR settings. In contrast, leverage similar sampling strategies but develop new, explicit guarantees these key predictive quantities, bridging this important gap.
Random Fourier Features (RFF) Rahimi & Recht (2007) approximate kernel functions using data-independent random feature maps, enabling scalable kernel learning without explicitly forming the kernel matrix. contrast, Nystrom-based methods construct data-dependent low-rank approximations by selecting informative subsets of the data. While RFF can be effective when kernel evaluations are expensive, it typically requires storing dense n D feature representations, often with large D to achieve high accuracy. In comparison, Nystrom methods store xxm m kernel factors with m < n, leading since RFF does not adapt to the data distribution, it may require higher-dimennsonall features to accurately capture predictive uncertainty in Gaussian Process settings. In contrast, ridge leverage score-based Nystrom sketching prioritizes statistically informative points, resulting in improved approximation of predictive variance and marginal likelihood, particularly in settings where memory efficiency and uncertainty calibration are critical.
more compact reprosentation. Morcover,
4 Algorithms
In this section, we outline the algorithms used to sketch the kernel matrix. While similar techniques have been studied in kernel ridge regression El Alaoui & Mahoney (2015). their application to Gaussian Process Regression (GPR) with explicit theoretical guarantees has not been previously established.
We use a gemeralized notion leverage scores specifically designed for the ridge regression setting, referred to as the oe-ridge leverage scores.
Definition 1. Given of > 0, the of leverage scores corresponding to kernel matric K and regulariza- tion/noise parameter of are defined as
n
Wi (l....,,
((0z)
Σ j=1
dj U2
o
Here, (a{) represents the tt diagonal entry of the matrix product K(K + +z where oy denotes the 5th eigenvalue of the kernel matrix K, and U is the orthonormal matrix of eigenvectors from its eigendeomposition. The set (1+ (l=())) (si)) serves similar role to classical leverage scores in statistics, they help identify
5

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 2 Approximate of E-Ridge Leverage Score Computation via Nystrom Sketching E1 Alaoui & Mahoney (2015)
Input: Data points {fI,..., Jn kernel function k(.,), sampling distribution {pr)i=1> sketch size m, regularization parameter o2
Output:
Approximate ridge leverage scores {4:)
Sample indices i1 f1....,,m Categorical(pt Pv)) with replacement
2: Form matrix C' Raxm such that Cj,t ((z
3: Form W E Rmxx with Wt,p = k( x.,,)
4: Compute B E R such that BB = C
D. Can use Cholesky or QR on W
5: Compute matrix = (B B + Im)m
for 1 n do
Set & B, MB; end for return (6:)
D B, the i-th row of B
influential data points that significantly impact the model output. In traditional settings, these scores are often derived from the row norms of the left singular vectors in the matrix U.
The effective dimension, denoted by den(o?), is defined as
dan(o?) = Tr (K(K+ +K(K+)2))-
where K is the kernel matrix and of 0 is the regularization/ noose parameter.
To efficiently approximate the oğ-ridge leverage scores without computing the full eigendecomposition of the
kernel matrix, adopt an approximation strategy inspired by Algorithm 2 El Alaoui & Mahoney (2015
The approximation algorithm 2 accepts as input a sampling distribution over the data points, which we set to a simple yet effective diagonal proxy where each point is sampled with probability proportional to the diagonal entry of the kernel matrix, i.e, Pt = K This choice motivated by the fact that the diagonal of K captures the self-similarity of each point and offers a compulationally efficient. surrogate for ridge leverage scores. The algorithm then selects a subset of size m, computes the corresponding kernel submatrices and W, and uses a Nyström-style factorization to produce an approximate low-rank embedding B, The resulting approximate leverage scores are given by the quadratic form l = B (BT 1) B, for each point This mothod has runtime O(n²²) and storage complexity O(nm), and provides provably accurate approximations to the true ridge leverage scores with high probability, while being scalable large dalasets where exact. score computation is infeasible.
Ku
5 Theoretical Guarantees
In this section, present main theoretical results for the predictive mean (Theorem 2 variance (Theorem 3), and negative log-likelihood (Theorem 4 under ridge leverage score based sketching. We begin by outlining the setup, notation, and assumptions common to all three theorems. For better readability, we have deferred detailed proofs to the appendix.
Let D ((.,,,))) be the datasot, where o; Rd are the input points and yi IR are the corresponding outputs. Let k(2,2) 2²) be the kernel function used in the Gaussian process regression, and let K be the kernel matrix such that Ki x,)) Let the eigenvalue decomposition of the kernel matrix be K = UEUT.
Sctup:
I if = else 0, where m < Let S E R=xm be sketching matrix (obtained in Algorithm 1 so that S₁y = mps
is obtained by probability distribution (p4) such that Vi ,n), P4 2 (3((())) 4(7)) for some (0, 1]. We define the sketch of kernel matrix L, KS(ST KS + + - K as the submatrix of K where 7 0.

---

Published in Transactions on Machine Learning Research (03/2026)
Moreover, let
D = 4 -- - SS U22/2
with + E -11+-11-'.
From, El Alaoui & Mahoney (015)) we have that, as long as the sketching matrix satisfies Amar (D) S for E (0,1) and Amaz denoting the maximum eigenvalue. Under this condition, they show that the approximation error between the original kernel matrix K and its sketched version L, is bounded as
(1=2)
03 1,
To ensure this bound holds, we follow the sketch size guarantee provided in Theorem 2 of Appendix B in El Alaoui Mahoney (2015), which characterizes the required number of samples m based on the ridge leverage score distribution is,
deff 1 x/)) log (") 3
For the mean and variance inference on test data, we express the test kernel vector as k, = Ua, where a E R" with a; = u, k., and U = [M1,..., un] is the eigenvect matrix of K. This a standard change-of- basis the eigenspace K and does not introduce any additional modeling assumptions.
Interpretation of and B. The coefficients quantify the aligmment of the test point with the spectral components of the kernel matrix, and determine how the eigenvalue spectrum influences the approximation error.
The parameter E (0, 1] controls the quality of the sampling distribution relative to the ridge leverage scores through the condition pt 2 3 3 E, lj. Larger B corresponds to more accurate leverage-based sampling (e.g., exact leverage scores when 8 = 1), while smaller B allows approximate scores and leads to a larger required sketch size m = (3)
Our analysis conditions on the observed dataset and quantifies uncertainty solely with respect the randomness of the sketching procodure. This follows the standard paradigm in randomized numerical lincar algebra Mahoney 2211);; Pilanci & Wainwright (2015); Musco & Musco (2017). where sketching is viewed computational approximation mechanism rather than statistical estimator of the data ceata-genraatng process. From a practical perspective, this form of guarantee is particularly relovant: given a fixed datasct, our results ensure that the sketched GPR solution closely approximates the exact solution with high probability over the sketch. This interpretation aligns with prior work in kernel sketching and randomized matrix methods.
5.1 Predictive Mean Estimation using Sketching
Theorem 2 (Predictive Mean Approximation under of Ridge Leverage Score Sketching). For the notations and assumptions defined in our Setup let
u(x') KT (K+ +0?
be the predictive mean of Gaussian Process Regression at a new point x* for the full kernel matria. Here k. = 11((11æ²)) k(x2, 2()2, k(In, IT and of is the noise. For the sketch of a kernel matrix L.y, the prealictive mean is,
ps(x") kT (L, + o
For the Ly obtained using Algorithm i we have
lu(x") us(x)))
?
a
l1a/l2 Amaz (Ap)
1-t)
E + o

---

Published in Transactions on Machine Learning Research (03/2026)
where, Ap = AD= set so that
T- 12, -0+ o?
-1
hold with probability at least 5, if the sketch size m is
deff 3
1 5)) log (n) + 6
n
m 2
Remark (Interpretation of a in the Predictive Mean and Variance Bounds). In the predictive mean approximation bound, the term involving of arises from expressing the test-to-(rain kernel vector k, R" in the eigenbasis of the kernel matrix K = UEUT, such that k, Ua with U k.. The coefficients a; quantify the aligmment of the test point I. with the spectral components of the training kernel and thus determine how the eigenvalue spectrum of K influences the approximation error.
Discussion. The bound depends on the energy of the test kernel vector in the eigenbasis of K, captured by E, af/(Ea + o²)². This term directly links the approximation quality to both the spectral decay of the kernel and the geometric relation between the test and training points. When the kernel spectrum decays rapidly such as for smooth kernels like RBF or high-order Matern the contributions from low-eigenvalue directions strongly attenuated, leading to tighter bounds. Similarly, larger noise variance of regularizes the influence of small eigenvalues, further stabilizing the approximation. Hence, the derived error bounds characterize how spectral compressibility of the kernel governs the fidelity of the sketched approximation without imposing additional assumptions on the distribution of the test inputs. An analogous interpretation applies to the predictive variance bound (Theorem 3 where the vector a again captures the projection of the test point onto the eigenspace of the kernel matrix.
We now specialize our predictive mean approximation to a standard nonparametric setting with polynomial spectral decay and source condition. These assumptions capture the smoothness of the underlying function and the effective complexity of the kernel through its eigenvalue decay.
Under this regime, our bounds translate into explicit learning rates that describe how the predictive mean error scales with sample size and noise. In particular, ridge leverage score acore-based Nystrom sketching achieves near-optimal rates (up to logarithmic factors) while using only a sublinear number of inducing points governed by the effective dimension.
Corollary 1 (Predictive Mean Approximation under Polynomial Spectrum and Source Condition). Let K = T with eigencolues satisfying polynomial decay
p>1,
and assume the test kernel vector satisfies the source condition of order r > 1/2:
he = Ua, af s Co! 1+2r
i= In..
Assume the Nystrom sketch satisfies the ridge leverage condition with
m 2 dea (o
(A) Tr((K +1)
and that o
Let oe x -B for 3 > 0. Then with probability at least 1 6,
(m(x.)) us(xx)) | 00 -nnn--ar-- ( ) m = llgg)
where = B[lvl]] is the expected squared norm of the observation vector. In particular, for the standard scaling of x 11 (i.e., B = 1), this yields
lm(x,) - #s(xx)) = 0 000, (r- (1- m = 0n
8

---

Published in Transactions on Machine Learning Research (03/2026)
We now strengthen our predictive mean approximation result by establishing uniform guarantee the entire input domain. While previous bounds control the error at fixed test point, uniform bounds are essential for ensuring that the approximation remains stable across all possible test inputs.
Under mild assumptions on kernel boundedness and sketching parameters, we obtain a bound on the worst-case predictive mean error over the domain. This result shows that the Nystrom approximation preserves the predictive behavior of the full Gaussian process uniformly, providing stronger guarantecs for downstream tasks that require consistent performance across the imput space.
Corollary 2 (Uniform Predictive Mean Approximation). Assume that the kernel is uniformly bounded compact domain X, i.C.,
k( x) x² =EX
sup
and that the sketching parometers satisfy
ty
1-1+1
Then, with probability at least. over the sketching rendomness, the predictive mean ervor satisfies
sup ju(x)) ps((*)) s EX
7K
1/yl2
2 17
(1 t)
o
1-147
5.2 Predictive Variance Estimation using Sketching
Theorem 3 (Predictive Variance Approximation under of Ridge Leverage Score Sketching). For the nolations and ussumplions defined our Setup lel.
Var(x*) = k(x",x")) K (K + o k
be the predictive variance of Gaussian Process Regression at new point x". Here, k- [k(x1,""), x(x),x"),.... k(Øn, xt and oe is the noise voriance. For the skelch of kernel matria Ly,
the predictive variance is,
Vars(x*) = k(aa x²) - kg (L.y + Te k.
For the T, obtained using Algorithm 7 we have
[Var(x*) Vars(z")| (121) ||a ADlll2
n
Σ a²
(Ei, + o?
where, An= (3(1 -1 + too1) set so that
hold with probability at least 1 - 8 if the sketch size m is
desf 3
71 5
m 2
log
6
We now extend the uniform analysis predictive variance, which plays a central role in uncertainty quantification in Gaussian process regression. While pointwise guarantees control the variance approximation at fixed test input, uniform bound ensures that uncertainty estimates remain reliable across the entire input domain.
Under the same boundedness and sketching assumptions, we obtain a high-probability bound on the worst-case predictive variance error. This result shows that Nystrom sketching preserves not only the predictive mean but also the uncertainty estimates uniformly, providing a principled guarantee for downstream tasks that rely on vell-calibrated uncertainty.

---

Published in Transactions on Machine Learning Research (03/2026)
Corollary 3 (Uniform Predictive Variance Approximation). Assume that the kernel is uniformly bounded compact domain X, i.e.,
sup #E
k( x) S x²
and that the sketching parometers satisfy Then, with probability at least 1 over the sketching randomness, the predictive variance error satisfies
ty
t+t
7 k33 Vn
sup 2 EX
Var(æ*) Vars (ans((x))
(1 t) o
by 1-14
5.3 Negative Log Marginal Likelihood Approximation
Theorem 4 (Negative Log Marginal Likelihood Approximation under of Ridge Leverage Score Sketching). Let K Rax be symmetric positive semi-definite kernel matriz and y E R the response vector. For of 0, the negative log marginal likelihood (NLML) be,
1 [(K = 2 T (K + o log det(K + of + " log((
The corresponding approzimale NLML, for the skelch of the kernel matria by oblained using Algorithm is given as,
1 [(L,) = 2 2/' (L, + T
1
o y + log det + o + 2
log(2.r)
Then, for anyy 0 05851. if
delf 3
71 log
m 2
then, with probobility at least 1 - 8 following inequality holds,
- C(L,) < 2(1 -) Xmmin(K + oo?
11v12 Amar (AD) + 2(1 ((1-)) Tr(Ap)
where, AD =
We now turn to the approximation of the negative log marginal likelihood (NLML), which governs hyperpa- rameter learning and model selection Gaussian process regression. Unlike predictive quantities, NLMT. a global scalar functional of the kernel matrix and does depend on test input.
Under the sketching assumptions, obtain high-probability bound the NLML approximation error, showing that Nystrom sketching preserves the training objective up to controlled perturbation. This provides principled guarantee that sketching not only maintains predictive accuracy but also faithfully approximates the objective used for learning kernel hyperparameters.
Corollary 4 (Uniform NLML Approximation). Assume that the skelching parameters satisfy
1-ttt
Then, with probability at least. 1 - 5 over the sketching randomness, for all y E TR"
[((K) - [(I,)) 2(1 - t)
o
byy
a
I-
by
o
10

---

Published in Transactions on Machine Learning Research (03/2026)
5.4 Interpretation of Ad
All bounds in our analysis share a central spectral lerm involving the malrix
AD 1 [1( - (E (E+ 1 7
of1
where E denotes the diagonal matrix of kernel eigenvalues, > 0 is the regularization parameter, t E (0, 1), and of is the noise
Spectral Dependence of the Bounds. The tightness of the predictive mean, variance, and NLML bounds is depends on the terms Amax (AD) and Tr(Ap), both of which are minimized when the spectrum of I exhibits fast decay. In such cases, AD becomes better conditioned, as low-eigenvalue directions are strongly regularized or suppressed by the additive noise. This yields tighter theoretical guarantees for the sketched approximation. Smooth kernels such as the RBF and high-v Matérn families naturally induce this spectral decay, particularly when applied to well-distributed, low-dimensional inputs typical of geostatistical data or physical simulations.
6 Experiments
All experiments were conducted on a machine equipped with an NVIDIA A100 PCTe GPU with 32 GB of memory. Our implementation is written in Python and leverages PyTorch and GPyTorch Gardner al. 2018 for efficient GPU-accelerated Gaussian Process modeling.
6.1 Datasets
We evaluate our methods four real-world regression dalasets: California Housing Pace & Barry 1997 Elevators Team (1996), Airfoil Self-Noise H. et al. (1999), and Protein Cai et al. (2003). All datasets are standardized using z-score normalization for both inputs and targets. For California Housing and Protein, we use 70%/30% train/test split; for the others, we follow an 80%/20% split.
California Housing contains 20,640 samples with real-valued features descríbing demographic and geographic attributes from the 1990 U.S. Census. The target variable the median house value in each district, making widely used benchmark for medium-scale regression tasks with heterogeneous feature distributions.
Elevators is a large-scale regression benchmark from the DELVE framework, hosted on the UCI repository. It consists of 16,599 samples with 18 continuous features capturing the dynamics of a control system, and a real-valued target representing elevator response time. The dataset exhibits moderately complex and nonlincar patterns, making it well-suited for testing scalable GP models.
Airfoil Self-Noise comprises 1,503 samples with 5 continuous features representing physical properties and operating conditions of airfoils in wind tunnel. The target is the scaled sound pressure level. Due to its small size and nonlinear behavior, it serves testbed for evaluating predictive uncertainty in low-data regimes.
Protein (also known as Protein Structure) is a large-scale regression dataset from the UCI repository with 43,730 samples and 9 physicochemical features describing the secondary structure of proteins. The target variable the root mean square deviation (RMSD) of atomic positions, which measures structural variability. This datasct widely used to benchmark scalable kernel methods due to its size, moderate dimensionality, and nonlinear structure.
6.2 Experimental Setup
We use the Radial Basis Function (RBF) kernel and Matern kernel for all three datasets. The kernel hyperparameters, including the lengthscales and variance, are initialized to 1.0 in case of RBF kernel and in
11

---

Published in Transactions on Machine Learning Research (03/2026)
Matern kernel / = 1.5 is initialized. The prior mean function is initialized constant set to 0 and is treated 8s learnable hyperparameter during training. Based on preliminary experiments, we fix the learning rate to 0.01 and train for 300 iterations across all methods. In contrast, SVGP was trained for 1000 iterations. This is because SVGP, as a variational method, optimizes an objective (the ELBO) that iteratively approximates the true posterior, a process that generally requires more iterations to stabilize than the methods that optimize the exact marginal log-likelihodd. We trained SVGP using Adam (I==0.01) and
size 1024.
6.3 Baselines
We compare our Nystrom Ridge Leverage method against a comprehensive set of basclines spanning both the coreset selection and scalable GPR literature which are described below. All methods are evaluated over progressively increasing subset sizes, covering approximately 2% to 12% of the full training set. To ensure statistical robustness and account for variability in subset construction, each experiment is repeated across 5 independent random sceds, where each seed corresponds to different data split and independently selected subset. We report the mean metric values across these trials, along with the corresponding standard deviations to reflect variability and robustness.
Uniform Subsampling. Uniform subsampling Hayashi et al. (2020): Malaviya et al. (2024) selects training points uniformly at random, independent of the data distribution or kernel structure. Although simple and computationally efficient, often fails to capture important geometric or uncertainty-relltted aspects of the
data.
Leverage Score Sampling. Leverage score sampling prioritizes points with higher statistical influence, emphasizing those contributing most to the low-rank structure of the kernel matrix Drineas et al. (2012); Zheng & Phillips (2017); Chhaya et al. (2020). This data-aware selection improves representativeness over uniform sampling and provides a foundation for more advanced sketching based approaches.
k-Means Coresct. We include a k-mcans-basod coresct bascline using the Lightweight Coreset method Bachem et al. (2018); Shit et al. (2022), which combines uniform and sensilivity-based sampling to select representative points with replacement. This efficiently approximates the data's clustering structure and scales better than exact k-means on large datasots.
Stochastic Variational Gaussian Processes (SVGP). SVGP Hensman et al. (013)) is a variational inference framework for scalable GPR that optimizes an evidence lower bound (ELBO) via stochastic gradients. It supports mini-batch training and inducing point learning, and is widely regarded as state-of-tho-art method for large-scale Gaussian Processes.
IterGP. IterGP (Wenger et al. 2022) introduces computation-awaa framework for Gaussian Process inference that explicitly models both mathematical uncertainty (due to finite data) and computational uncertainty (due to approximate inference). Unlike standard approximations such as SVGP or CG-based solvers that ignore the uncertainty introduced by limited compute, IterGP provides a combined posterior whose covariance decomposes into mathematical and computational components. This guarantees convergence of the posterior mean in RKHS norm and offers a worst-case bound on the approximation error.
Random Fourier Features (RFF). Random Fourier Features (RFF) approximate shift-invariant kernels using randomized feature maps, enabling scalable learning without explicitly forming the kernel matrix. The kernel is approximated via a finite-dimensional feature embedding of size D, followed by linear inference. In our experiments, we use D = 512 features. However, being data-independentt RFF typically requires large D to achieve high accuracy in Gaussian Process settings Rahimi & Recht (2007).
Nystrom Approximations. We compare three Nystrom based kernel approximation methods that differ in how the inducing points (columns of the kernel matrix) are selected. The uniform variant samples columns uniformly random with replacement Williams & Seeger (2001). while the leverage score variant uses sampling probabilities proportional to the standard leverage scores Gittens & Mahoney (2016). Our method
12

---

Published in Transactions on Machine Learning Research (03/2026)
employs ridge leverage score sampling, which incorporates the regularization parameter and provides data-aware, theoretically grounded alternative for constructing Nystrom approximations in GPR.
6.4 Evaluation Metrics
Model performance is evaluated using the following metrics: (i) predictive mean error, (ii) predictive variance error, (iii) root mean squared error (RMSE), (iv) Negative Log Predictive Density (NLPD), (v) Mcan Standardized Log Loss (MSLL), and (vi) Negative log likelihood (NLL). Predictive mean and variance errors are computed as the relative Eg norm difference with respect to the full Gaussian Process model, defined as respectively.
|| - Msketch /
|I |
and ||o -
Negative Log Predictive Density (NLPD) To evaluate the quality of uncertainty estimates in Gaussian Process Regression (GPR), report the Negative Log Predictive Density (NLPD). NLPD measures how well the predicted Gaussian distribution aligns with the true targets, penalizing both misestimated means and variances. Formally, for test data 31) with predictive mean b4 and variance of, NLPD is computed as:
n NLPD
(yi-Ma) 1₂) 20;
Σ 72
+ 1og(2To7)
(12)
Lower values indicate better predictive performance and better-calibrated uncertainty. NLPD is a proper scoring rule and is widely used in evaluating probabilistic regression models Artemev al. (2021) Rasmussen & Williams (2005).
Mean Standardized Log Loss (MSLL). Unlike standard error metries such as RMSE, MSLL evaluates how well the predictive distribution improves a simple baseline model (typically the empirical mean and variance of the training targets). Formally, MSLL is defined as
MSLL 1 n Σ [log p(y: z;. Perain) log Pbaseline (y/ xx)]
(13)
where p(W xy, Dirai)) denotes the model's predictive density and Poaseline (3t x.)) corresponds to the baseline predictive distribution. A negative MSLL indicates that the model outperforms the baseline in terms of log predictive density. MSLL is particularly useful because it standardizes performance across datasets with different output scales and provides a more interpretable measure of probabilistic performance than raw log likelihood (Rasmussen & Williams 2005
6.5 Results and Analysis
Our main results, presented in the figures below and detailed in the tables in the appendices, demonstrate that the proposed Nystrom ridge leverage sketching method consistently outperforms all considered basclines across datasets, most notably strong metrics such as NLPD and MSLL. Despite SVGP and IterGp being a state-of-the-art approach for scalable GPs, particularly in probabilistic modeling, our method achieves superior performance, especially in terms of Negative Log Predictive Density (NLPD), which is a proper scoring rule sensitive to both prediction accuracy and uncertainty calibration. Our method also achieves consistently lower predictive mean and variance errors, indicating that it more accurately approximates the true postcrior distribution of the Gaussian Process Regression. Notably, the NLPD gains are achicved using an efficient, approxinate version of ridge leverage score compulation Algorithm 2 showcasing the scalability and effectiveness of our approach. The best-performing results are shown in bold. We focus these representative basclines to cover the most widely used paradigms for scalable GPR subsot selection, Nystrom approximation, iterative approximation, and variational inference.
While several variational approaches to scalable Gaussian Process inference exist, we include SVGP as it remains the most widely adopted and well-ostablished representative of this class. Additionally, we compare against IterGP, a recent state-of-the-art scalable GPR method, benchmark our approach against the strongest contemporary baselines.
13

---

Published in Transactions on Machine Learning Research (03/2026)
uniifomm
leverage
kmeans
svpp
BerCp
nystrom_mm un!
nystrom_Je Jev
random If
nystrom, Lris
vs Sample (%)
Sample
Sample (%6
1.75
1.50
Predicãve Variance Enror
Predlctive Mean
AMSE
0.1
0.00
2%
4%
6%
8%
10%
2%
4%
6%
8%
10%
2%
4%
6%
8%
10%
Sample Size (%) Full
Sample Size
Figure 1: Results on UCI Elevators Dataset. Evaluation of Gaussian Process Regression methods the UCI Elevators dataset using the RBF kernel. Predictive mean error, predictive variance error, and RMSE are plotted subset size. Ridge Leverage based GPR yields the best tradeoff across metrics. All results averaged over random trials with standard deviation shown as error bars. The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
uniform
leverage
kmeans
SV3P
herrp
nystromuum
nystrom_Jev
random _f
nystrom_rs
Mean Sample Size (%)
Variance Error Size (%)
NLPD Sample Size (%)
40
1.0
30-
Predictive Mean
0.8
0.6
20
0.4
10
0.2
0.2
2%
496
6%
8%
10%
2.%
416
6%
8%
10%
25
4%
6%
8%
10%
Sample Size (%) of Full Data
Sam plo Size in (96) of Full Data
Sam pla Size in (9% of Full Data
Figure 2: Results on Protein Dataset. Performance comparison of various Gaussian Process Regression (GPR) methods on the Protein dataset using RBF kernel. The ridge leverage based sketching method achieves superior predictive variance and NLPD compared to uniform, IterGp, SVGP, RFF and other baselines, demonstrating its robustness on this high-dimensional, large-scale regression task. The reported NLPD and uncertainty metrics are averaged over random trials, with bars representing standard deviations across The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
For the California Housing dataset, we were unable to include the Nystrom (leverage) baseline, as the resulting kernel matrix approximation was not positive semi- meiindefintee, which caused instability during model training. More results are included in the appendix.
Training Time and Dataset Scale Justification. In our experiments, the SVGP baseline often required longer fraining time than exact GPR despite its theoretical scalability. This effect is prominent for moderate- scale datasets (n 15K to 30K), as GPyTorch's exact GPR leverages efficient conjugate gradient routines and optimizes only a few kernel hyperparametors, whereas SVGP jointly learns kernel and variational parameters through stochastic updates. Moreover, mini-batching and stochastic optimization introduce additional overhead at this scale, making SVGP slower in wall-clock time compared to the exact solver for moderate n, behavior also observed in prior work (Wilson & Nickisch/ 2015 Gardner et al. 2018 Pleiss et al. 2018 We therefore restrict our benchmark to the Protein datasct (n 45,730) the largest size for which full GPR remains tractable on a 32 GB GPU. Beyond this scale, storing and inverting the full kernel (On time, O(n²) memory) becomes infeasible, preventing computation of reference quantities such as predictive mean or variance errors. This regime allows meaningful and fair comparison against scalable methods while
maintaining exact GPR as ground-trute tnuth reference (Rasmussen & Williams 2005 Gardner et al. 2018 Wang et al. 2019
14

---

Published in Transactions on Machine Learning Research (03/2026)
unifom
leveraae
emwars
svge
BerGp
nystrom uni
random, If
mystromm Ifs
Sample (%)
Variance Sample (%%
Sample (%6
1.2
1.0
1:25
10
Variaance
Predictive Mcan
1,00
0.4
0.50
0.2
0.25
0.0
3%
5%
6%%
1.1
12.2%
31
5%
6%
11%
12.2%
3%
5%
6%
11%
12.2%
Sample
Sample (96) Full
Figure 3: Results Protein Dataset. Performance comparison of various Gaussian Process Regression (GPR) methods on the Protein dataset using the Matern kernel. The ridge leverage- sketching method achieves superior predictive variance and NLPD compared uniform, IterGp, SVGP, RFF and other baselines, demonstrating its robustness on this high-dimensional, large-scale regression task. The reported NLPD and uncertainty metrics are averaged over random trials, with error bars representing standard deviations across runs. The dashed horizontal line indicates the performance of the full-dalaset (exact GP) model.
GPR
Leverage Kmeans
Srop Nystrom unt
Nystrom ++-
Nystrom
Unifomm
-
Protein Dataset
Elevators Dataset
Housing Dataset
Negative Likeilrced (HLL)
3.5
3.0
2.5
2.0
1.5
1.5
1.0
100
10
10
Subset size (%
Subset size (% of data)
Subset size of data)
Figure 4: Comparison of Negative Log Likelihood (NLL) across different subset sizes and Gaussian Process approxi- mation methods on the Protein, Elevators, and Housing datasets using the RBF kernel. Each subplot reports the mean and standard deviation over five random trials. The dashed horizontal line denotes the performance of the full-dala (Exact GP) model.
7 Conclusion
We proposed a scalable Gaussian Process Regression method that combines Nystrom approximation with ridge leverage score sampling. While ridge leverage scores have been used in kernel ridge regression and matrix approximation, our work the first apply them the Gaussian Process sctting with thcoretical guarantees on predictive mean, variance, and negative log-likelihood. Our analysis shows how the quality of the approximation depends the kernel spectrum and sketch size, and our experiments demonstrate consistent improvements over existing baselines.
Broader Impact Statement
We do not foresce any potential nogative impact.
Acknowledgments
Anirban Dasgupta would like acknowledge the following grants and the corresponding funding agencies SERB-MATRICS grant, SERB-CRG research grant.
15

---

Published in Transactions on Machine Learning Research (03/2026)
References
Artem Artemev, David R Burt, and Mark der Wilk. Tighter bounds the log marginal likelihood of gaussian process regression using conjugate gradients. In International Conference Machine Learning, pp. 362-372. PMLR, 2021.
Olivier Bachem, Mario Lucic, and Andreas Krause. Scalable k-means clustering via lightweight coresets. Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 1119-1127, 2018.
Rajendra Bhatia. Matrix analysis, volume 169. Springer Science & Business Media, 2013.
Erdem Bıyık, Nicolas Huynh, Mykel J Kochenderfer, and Dorsa Sadigh. Active preference- based gaussian process regression for reward learning. ar preprint orXin:2005.0257 2020.
Stephen Boyd and Lieven Vandenberghe. Conver Oplimization. Cambridge University Press, 2004.
David R Burt, Carl Edward Rasmussen, and Mark Van Der Wilk. Convergence of sparse variational inference in gaussian processes regression. Journal of Machine Learning Research, 21 (131 ::1633, 2020.
C. S. S. Cai, D. J. Hand, N. M. Adams, K. structure dataset. htttps://archt: ics. Anagnostopoulos, and A. J. M. Ferreira. Protein Prodeenn/Terttaaryy-ttruuuur 2003. UCT Machine Learning Repository.
Danicle Calandriello, Alessandro Lazaric, and Michal Valko. Analysis of nyström method with sequential ridge leverage score sampling. In Uncertainty in Artificial Intelligence, 2016.
Yifan Chen and Yun Yang. Fast statistical leverage score approximation in kernel ridge regression. In International Conference on Artificial Intelligence and Statistics, 2935-2943. PMLR, 2021.
Rachit Chhaya, Anirban Dasgupla, and Supratim Shit. On coresets for regularized regression. In International conference on machine learning, pp. 1866-1876. PMLR, 2020.
Amir Dezfouli and Edwin V Bonilla. Scalable inference for gaussian process models with black-box box likelihoods. Advances in Neural Information Processing Systems, 28, 2015.
Kun Dong, David Eriksson, Hannes Nickisch, David Bindel, and Andrew G Wilson. Scalable log determinants for gaussian process kernel learning. Advonces in Neural Information Processing Systems, 30, 2017.
Petros Drineas, Ravi Kannan, and Michael W Mahoney. Fast monte carlo algorithms for matrices Approximating matrix multiplication. SIAM Journal on Computing, 36(1):132-157, 2006a.
Petros Drincas, Michael W Mahoney, and Shan Muthukrishnan. Sampling algorithms for 1 2 regression and applications. In Proceedings of the seventeenth annual ACM-SIAM symposium Discrete algorithm, pp. 1127-1136, 2006b.
Petros Drineas, Malik Magdon--smail, Michael W Mahoney, and David P Woodruff. Fast approximation of matrix coherence and statistical leverage. The Journal of Machine Learning Research, 13(1):3475 3506, 2012.
Ahmed El Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statistical guarantees. Advances in Neural Information Processing Systems, 28:775-7888 2015.
Tamás Erdélyi, Cameron Musco, and Christopher Musco. Fourier sparse leverage scores and approximate kernel learning. Advances in Neural Information Processing Systems, 18:109-122 2020.
Christian Fiedler, Carsten W Scherer, and Sebastian Trimpe. Practical and rigorous uncortainty bounds for gaussian process regression. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7439-7447, 2021.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Jacob R. Gardner, Geoff Pleiss, Kilian Q. Weinberger, and Andrew Gordon Wilson Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Alex Gittens and Michael W Mahoney. Revisiting the nyström method for improved large-scale machine learning. The Journal of Machine Learning Research, 17(1)3397-40411, 2016.
Brooks T. H., D. S. Pope, and M. A. Marcolini. Airfoil self-noise data set, 1999. UCI Machine Learning Repository.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximale matrix decompositions. SIAM review, 53(2):211-25) 288, 2011.
Kohei Hayashi, Masaaki Imaizumi, and Yuichi Yoshida. On random subsampling of gaussian process regression: A graphon-based analysis. In International Conference Artificial Intelligence and Statistios, pp. 2055-2065. PMLR, 2020.
James Hensman, Nicolo Fusi, and Neil Lawrence. Gaussian processes for big data. In Conference on Uncertainty in Artificial Intelligence (UAT), 2013.
Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Active learning with gaussian processes for object categorization. In 2007 IEEE 11th international conference on computer vision, pp. 1-8. IEEE, 2007.
Jihao Andreas Lin. Scalable gaussian processes: Advances in iterative methods and pathwise conditioning. arxiv preprint arXiv:2577.0883) 2025.
Jihao Andreas Lin, Sebastian Ament, Maximilian Balandat, David Eriksson, José Miguel Hernández- Lobato, and Eytan Bakshy. Scalable gaussian processes with latent kronecker structure. orXiv preprint NrXiv.2506.06895 2025.
Haitao Liu, Yew-Soon Ong, Xiaobo Shen, and Jianfei Cai. When gaussian process meets big data: A review of scalable gps. IEEE transactions neural networks and learning systems, 31(11):4455-4423 2020.
Yifan Lu, Jiayi Ma, Leyuan Fang, Xin Tian, and Junjun Jiang. Robust and scalable gaussian process regression and its applications. In Proceedingss of the IEEE/CVP conference on compuler vision and puldern recognition, pp. 21950 21959, 2023.
Michael W Mahoney, Randomized algorithms for matrices and data. Foundations and TrendsB in Machine Learning, 3(2): 2244, 2011.
Jayesh Malaviya, Anirban Dasgupta, and Rachit Chhaya Simple weak coresets for non-decomposable classification measures. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14289-14296 2024.
Cameron Musco and Christopher Musco. Recursive sampling for the nyström method. In Advunces in Neural Information Processing Systems (NeurIPS), 2017.
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33(3): 291-297, 1997.
Misha Padidar, Xinran Zhu, Leo Huang, Jacob Gardner, and David Bindel. Scaling gaussian processes with derivative information using variational inference. Advances in Neural Information Processing Systems, 34: 6442-6453, 2021.
Mert Pilanci and Martin J Wainwright. Randomized sketches of convex programs with sharp guarantees. IEEE Transactions on Information Theory, 6995506996,,, 2015.
Mert Pilanci and Martin J Wainwright. Randomized sketches of convex programs with sharp guarantees. IEFF Transactions on Information Theory, (19))5753-557855 2017.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Geoff Pleiss, Jacob Gardner, Kilian Weinberger, and Andrew Gordon Wilson. Constant-tinn time predictive distributions for gaussian processes. Tn International Conference on Machine Learning, pp. 4114-4123. PMLR, 2018.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in newral information processing systems, 20, 2007.
Carl Edward Rasmussen and Christopher KT Williams. Gaussian Processes for Machine Learning. MIT press, Cambridge, MA, 2005.
Alessandro Rudi and Lorenzo Rosasco. Less is more: Nyström computational regularization. In Advances in Neural Information Processing Systems (NewrIPS), 2015.
Jens Schreiter, Duy Nguyen-Tuong Mona Eberts, Bastian Bischoff, Heiner Markert, and Marc Toussaint. Safe exploration for active learning with gaussian processes. In Joint European conference on machine learning and knowledge discovery in dalabases, pp. 133 149. Springer, 2015.
Jiaxin Shi, Mohammad Emtiyaz. Khan, and Jun Zhu. Scalable training of inference networks for gaussian- process models. In International Conference on Machine Learning, pp. 5758-5768. PMLR, 2019.
Supratim Shit, Anirban Dasgupta, Rachit Chhaya, and Jayesh Choudhari. Online coresets for parameteric and non-parametric bregman clustering. Transactions on Machine Learning Research, 2022.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems, pp. 1257-1264, 2006.
The DELVE Team. Individual household electric power consumption data set (elevators), 1996. UCI Machine
Learning Repository.
Jörn Tebbe, Christoph Zimmer, Ansgar Steland, Markus Lange-Hegermann, and Fabian Mies. Efficiently computable safety bounds for gaussian processes in active learning. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2024.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12(4):389-42) 434, 2012.
Ke Alexander Wang, Geoff Pleiss, Jacob R. Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gor- don Wilson. Fxact gaussian processes on million data points. In Advonces in Newral Information Processing Systems (NeurIPS), 2019.
Jonathan Wenger, Geoff Pleiss, Marvin Pförtner, Philipp Hennig, and John P Cunningham. Posterior and computational uncertainty in gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Christopher KI Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In Advances in Neural Information Processing Systems, pp. 682 688, 2001.
Andrew Gordon Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International Conference on Machine Learning (ICML), 2015.
Andrew Gordon Wilson, Christoph Dann, and Hannes Nickisch. Thoughts on massively scalable gaussian processes. Xive preprint arXiv:1511.01877 2015.
David P Woodruff et al. Sketching as tool for numerical lincar algebra. Foundations and Trends0 in Theoretical Computer Science, 10(1 2):1 157, 2014.
Zhitong Xu, Haitao Wang, Jeff M Phillips, and Shandian Zhe. Standard gaussian process is all you need for high-dimensional bayesian optimization. arXiv preprint arXiv:2402.0277 02746, 2024.
Yan Zheng and Jeff M Phillips. Coresets for kernel regression. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 444, 2017.
18

---

Published in Transactions on Machine Learning Research (03/2026)
A Appendix
In this section, we present our main theoretical results for the predictive mean (Theorem 5 variance (Theorem 7 and negative log-likelihood (Theorem 8 under ridge leverage score-based sketching. We begin by outlining the setup; notation, and assumptions common all three theorems.
A.1 Common Theoretical Setup:
Let. D {(r v:))== be the dataset, where x; Rd are the inpul points and yi E R are the correspouding outputs. Let (ax,x") be the kernel function used in the Gaussian process regression, and let K be the kernel matrix n n such that Kij k(r,,zy) Let the eigenvalue decomposition of the kernel matrix, K UEUT.
1 if in clse 0, where Let S E Rexm be sketching matrix (obtained in Algorithm 1) so that Siy V map
is obtained by probability distribution (24)ssisn such that Vi [I,..,,)], pi 4(022/ Li L( for some 3 (0, 1].
We define the sketch of a kernel matrix L, = KS(S KS + + K as the submatrix of K where 0.
Moreover, let
D - D= 55 UQ
with Φ Σ(Σ -11++1)-1
From, El Alaoui & Mahoney 20155) we have that, as long as the sketching matrix satisfies Amar (D) I for t E (0,1) and Amaz denoting the maximum cigenvalue. Under this condition, they show that the approximation error between the original kernel matrix K and its sketched version L, is bounded as
1
0 O3K-L 3
To ensure this bound holds, we follow the sketch size guarantee provided in Theorem 2 of Appendix B in El Alaoui Mahoney (2015), which characterizes the required number of samples m based on the ridge leverage score distribution is,
doff 3
1
11 (")
log
For the mean and variance inference on test data, express the test kernel vector as k, Ua, where E R with u. k., and U= [M....., "n] is the eigenvector matrix of K. This standard change-of basis in the eigenspace of K and does not introduce any additional modeling assumptions.
Interpretation of and B. The coefficients a quantify the alignment of the test point with the spectral components of the kernel matrix, and determine how the cigenvalue spectrum influences the approximation error.
The parameter 3 E (0, controls the quality of the sampling distribution relative to the ridge leverage scores through the condition Pt 2 3 2 E, lj. Larger 3 corresponds to more accurate leverage- baarag sampling (c.g.. exact leverage scores when 3 = 1), while smaller B allows approximate scores and leads to larger required sketch size m = (dent)))
B Predictive Mean Estimation using Sketching
Theorem 5 (Predictive Mean Approximation under of Ridge Leverage Score Sketching). For the notations and assumptions defined in Setup let
u(x²) K (K or -1
19

---

Published in Transactions on Machine Learning Research (03/2026)
be the predictive mean of Gaussian Process Regression at new point x² for the full kernel matriz. Here k. = [k<(=1,")) k(22, x²) ((2,, ).... x") and of is the noise. For the sketch of kernel matria Try, the predictive mean is,
ps(x*) = k. (L, + +
For the boy oblained using Algorithm we have
lu(x*) us(x)))s
7
a EE + of)z
11v122
Amaz (Ap)
1
where, AD= (x[1- - o21 )
holds with probobility at least 1 - 8, if the sketch size m is
set that
delf
1
11
m
3
6
log
g
Proof.
lu(x") - (s((1))
IKT + o2
[ + (L, + -2
$ (KT (L, + of |/
|KT]((K - (L, + 1|| le
(K [I, 111 llul|2
| + +K [,)] [L, + + !1ap" ||
IK. (K - - 11 [L, + +z '1.op |lvlla
(As K - L, 1 (1z)11 1)
[KT - |1u/|2 Amaz (An)
(Putting bound on, (L, + 02
from equation 22 below)
|USEU + + -11, llev/le Amar (Ap)
1-t
(1-7
+ ([UUE++21UUT)
|| (AD)
(-22)
(0U(E++21)-10 ||wl|2 Amar (An)
[UU)] T (U(I+ 1 T 1l1 11æ/122 Amaa(AD)
(Substitute, k, Ua)
|aTUTU(> + -i 112 l1l//2 Amaz (An)
|| + 221) - 11a1|2 Amaz(AD)
The spectral norm decomposes as:
n Σ
a?
(E,s + o?))
i=1
20

---

Published in Transactions on Machine Learning Research (03/2026)
Therefore, in the final bound we have
a Dig + of
lu(=") - us(a))s 1-1)
l1ull2 Amaz (Ad)
Corollary 5 (Predictive Mean Approximation under Polynomial Spectrum and Source Condition). Let K = UUUT with eigencolues satisfying polynomial decay
x
p>1,
and assume the test kernel vector satisfies the source condition of order 1/2:
k = Ua,
Co i= l.....,
Assume the Nystrom sketch satisfies the ridge leverage condition with
m Z log(n/0), da Tr(K(K + 111-
and that 7 x o.
Let oe x nn for B > 0. Then with probability at least 1 - 8,
(m(x.) us((x))] = -0r-0-0---000-0 + =))
m (ng)))
log
where =[lvlli) is the expected squared norm of the obsercalion vector. particular, for the standard scaling o² x n- (t.e., B = 1), this yields lm(x,) - ps(xx))== ( - (r-
())
m = Onn
Proof. From Thcorem
we have
(k(x.) - (s(x))1 1-t |vll2,
Amax (Ap)
(a + of)z
Step 1: Apply Nystrom bounds. Using the ridge leverage Nystrom bounds,
Amax(A)) = ((
x oF
and / E (0,1) (which holds under the stated sketch size condition), obtain
a Σ (o, + 22)
(m(x.) - [s(
IIvile,
Step 2: Apply source condition and bound the sum. By the source condition a Co 1+2
2r
$
C
0; o
o
i=1
Let i' be the index satislying o;- x o Since oi x i we have
1" x (o 1/p
21

---

Published in Transactions on Machine Learning Research (03/2026)
Splitting the sum at it:
1+ of + o2)
2+2r
SSC
+
0
123
For i S 1": We have of 2 o so oi of o. Thus,
1+2r
2
p(2r-1)
O + 02)2
15
$51
Since 1/2, have p(r 1) O. For p(2r 1) < 1, the sum is dominated by the largest term at i = it:
Σ i 1(2 x (1-)2- x (a?)- (1--l222 1))/ (02)2r-1-1/e SSP
For - - 1) 2 1, the sum scales as
(-p(2r)1) (1')-p(2r-1) (o (02)0r-1-1/0
For i> 8t: We have o, < o.. 80 0i + oe x o Thus,
1+2r
(o 2 i p(1+22)) x (o (
Di (o
where the last step uses p(1 2r) p 1. This gives
(o))~ - (1-1(1+2)1/ = 201/-2-10 (02)2r-1-11p
Both sums yield the same scaling, so
Sx (02)2r-1-118
Therefore,
[m(x.) - us (s(x))) -
Step 3: Express in terms of sample size The effective dimension satisfies
dea(o?) = Σ
a: = (o 1/p o oz
Setting of x n=, we obtain
[u(x.) - us(xx) O O11112n (0(0- -(r -1)-
and the sketch size requirement becomes
m = O(der log n) 0n(2/e logn)
For the standard choice 3 (i.e., of x n -1), the rates simplify as stated.
Lemma 6 (Spectral lower bound). For any t (0, 1), > and all 2 0,
70
ty
(1 -1- () +2)
1 1- t+
22

---

Published in Transactions on Machine Learning Research (03/2026)
Proof. Define
70
f(o) = 0 (1- (
220
Then
72
f((o= =1
(1 t) a + 7)
Setting f'( = O gives the unique minimizer
0² γ
V1 - 7
Evaluating f(a") yields
tr f(0)) = t t+t
Since f(o) o + c0, this is the global minimum.
Corollary 6 (Uniform Predictive Mean Approximation). Assume that the kernel is uniformly bounded on compact domain X, i.e.,
sup k(x x² zEX
and that the sketching parameters satisfy
1-t+t
Then, with probability at least 1 - 8 over the sketching randomness, the predictive mean error satisfies
sup ex
(u(x*) (u((x))) 5
7K
|
(1 t) v o0
1-14
Proof. From Theorem 5 for any fixed test point x² E X,
lu(x") (s((=)) 7. 1-t Amax (AD).
11912
(14)
En + 222
where UTk. and k, = (k(21, "(x,,="""",,,,,,==""
Step 1: Bounding the spectral alignment term. We first rewrite the spectral term using the eigendecomposition =
(U (UT k.)?
D + o22
(a, + o2)2
= k U(E + - k.
k (K + o
(15)
To bound this quantity uniformly over X, exploit the eigenvalue structure. Since + oz 2 oz for all E {l....,7 n), we have
1
( + 22
of (a. oz)
23

---

Published in Transactions on Machine Learning Research (03/2026)
Therefore,
a
Ao (K T) -k,
Σ
(0 + o
7
0 i=
0j o
k [K + o 1) k
(16)
From the Gaussian Process posterior variance formula, we have
Var(f(x") () k(aa 1(xx,x²))~ k. (K + of 2
which implies
E (K + T) hi S k(x", xx) I R²
(17)
Combining equation 16 and equation 17
we obtain
sup x EX
2
(18)
Step 2: Bounding the maximum eigenvalue of A
Recall that
AD
1 {5(
?
( (E+g)=))
+o?1
-1
The i-th eigenvalue of AD is given by
(inner) oi
?
os o,
7/0
+og
(19)
(1 () + +7)
(1 () + 7)
To find (Ap), we must determine min; X (inner)
Case 1: Small eigenvalues. For o, < 7 we can approximate
70%
700
2
(1- [)()(a; +7)
(1- 1)7
1-6
so that
X
(nnnr)) 0:
01 + o 0i
1 (1-1-
1 1 )
o
to:
+ o
t
For small d, this approaches of from below.
Case 2: Large eigenvalues. For > we have
70,
70.
(1 () + 7)
(1-t)a;
so that
X (inner)
?
+ 2
t
As 0; + oo, this grows without bound.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Critical observation. Rewriting equation 19
(inner)
oil(1 - t)o tyl
+ +
(1 - () + 7))
we see that the first term negative only when Over this interval the function is increasing os, hence maximum attained at
0 0<0; < ly
1-t
70;
g(o,)
(o, + 7)
ly
di =
1 t
Substituting this value yields
tr
max g(0;) =
0.0; T
-
Therefore by above and Lemma
(inner)
ty
min A
2 0g 1- t
Amax(Ap)
(20)
17 1-t-t
0
Step 3: Combining the bounds.
Substituting equation 18 and equation 20 into equation 14
we obtain
sup |m(x*) (-((==((( )1 1-t t x" EX
|
of 1-t
which simplifies the stated bound. This completes the proof.
C Predictive Variance Estimation using Sketching
Theorem 7 (Predictive Variance Approximation under of Ridge Leverage Score Sketching). For the notations and assumptions defined in our Setup let
Var(x") =
T (K + +
be the predictive variance of Gaussian Process Regression a new point x², Here, ke [k(x1,²")) k(x,, .."..... k(0n, "²) and of is the noise variance. For the sketch of kernel matria Lyy the predictive variance is,
Vars(x") = k(a" "(a",x"))"""" k_ (T., -
For the Ly obtained using Algorithm we have
[Var(x*) -aarx=)-Varreee) Vars(e")| 7 ||a Apll2
Σ az
1
(Ee,i + oz)
where, AD (3) [I[T- 22 +
holds with probability at least 5 if the sketch size is
set so that
deff 3
71
m 2
log
6
5
25

---

Published in Transactions on Machine Learning Research (03/2026)
Proof.
[Var(=") Vars(**))
= |k(x", [[(
k(x", xx" + (Tn, + +z
| (K + + -) kim
|K. (o o |
= [K([ |kk
|4 (Tn, +
[K + +f²1) "kk
7
[5 + 1) - 'K -
(As, 03 K - L, 3 (121 T)
1
7
[L, + ofol) "UEUU + tZ
1-t
7
[, + (2
1-t
7
(Substitute,, k, Ua)
7
TU (L, + (U(+
(Since, by x lig)
(1-)
7
Σ + -1) U- to2l (U(E+ + -U U Ua
U
Σ
T-1
(
U
( 1-
7
E(E -1) + + U7)
(U(E+ + - Ua
7
T Σ (1-1) E(E + + tbo
(I (E+oZ1)
1-t
(As, Ao [Σ- (E-(1z) E(E + 71) 1 + of! )
ATDDDE= ब2 al
||اTدهI| 1 + o2
The spectral norm decomposes as:
|[E + +
Σ a?
n
1
(Eis + oz)
Therefore, the final bound we have
[Var(x")- - Vars(x*))
(1-)) ||a Aolll
Σ az
12
1
EEi + oz)
Corollary 7 (Uniform Predictive Variance Approximation). Assume that the kernel is uniformly bounded on compact domain X, i.e.,
sup k(æ x) S x² #et
26

---

Published in Transactions on Machine Learning Research (03/2026)
and that the sketching parameters satisfy
by oe
> I-ttt
Then, with probability at least 1 - 8 over the sketching randomness, the predictive variance error sotisfies
7k3 Vn
sup x'
Var(x*) Vars(x*))
(1 1) 00 0 1-
Proof. From Theorem 7 for any fixed x" Ex,
Var(=") - Vars(x*) $ - ||a |IaTablll Apll2 1| tall2
(21)
Step 1: Bounding the spectral alignment term
Using K = UEUT
( =
k. (K of - kk.
Since (o, (z) 2o 05 "0; of)
kI (K T2 k, s
h (K T) k.
oe
By the GP posterior variance identity.
k. (K + TZ S k(x* xx") ) S 162
",n","">>
hence
| + -2 S
sup x E'
o
Step 2: Bounding (aTablla
Since An diagonal in the eigenbasis,
|| Apll S Amax (Ap) ||a]|2
From the predictive mean corollary and Lemma 6
Amax(Ap) S
og
by T-t-r
Moreover,
12
k(2,, (2) 1/2
||a||2 = || Σ
S Vnk²
using [M(x,,)*) (;)
x²) s x²
Hence,
Viik²
sup T Anlla 5 Ex
of
by
1-t+1
Step 3: Combining the bounds. Substituting into equation
21 gives
Vina
sup x'6X
Var(x") - Vars(=*)) s 1- t
?
K
of
1-1+1
which completes the proof.
27

---

Published in Transactions on Machine Learning Research (03/2026)
D Negative Log Marginal Likelihood Approximation
Theorem 8 (Negative Log Marginal Likelihood Approximation under of Ridge Leverage Score Sketching).
Let K E Ruxn be a symmetric positive semi-definite kernel matrix and E R the response vector. For of > 0, the negative log marginal likelihood (NLML) be,
1 ((K) /リ (K + d log det(K + +z + "
log (2T)
2
The corresponding approzimate NLML for the sketch of the kernel matrie Tay obtained using Algorithm 1 is given as,
1 ((Ly)= = 2 (L, o21) + log Logdet(L,, o + log(2T)
1
Then, for any 0 < 1, if
deff
1
m
3
6
log
then, with probubility al least 1 - 5 following inequality holds,
(Nmmn( + oz?T
2(L))1 2(1 -t)
?
11y112 Amar (Ap) + Tr(An)
2(1 1)
where, AD AD= - +1)- of!) -
Proof. To analyze the negative log marginal likelihood (NLL), omit the additive constant term involving 3 log(2rr), as it does not affect the optimization or approximation. The resulting expression consists of two principal components: a quadratic term and a log-determinant term. We derive separate bounds for each of these components and then combine them to obtain an overall bound on the NLML approximation error.
((K) = /リ (K + d' + log det + +T²) +
" log (2rr)
2
Since, we have 0 K - Ly 1
2
bound from
Alaoui & Mahoney 2015).
1-7
28

---

Published in Transactions on Machine Learning Research (03/2026)
2 (K
T (K + y- - "000"''''(A (L,
+ of 7'y
IKI(K + oî - (L, + '
2
_ 2
(K(K + o - (4, + of -1 op 11v112
1
||(K d, () o. '11 op
1101
|1(K + oz!l) o - [,, + oī - |lv1
*1oop
2
_ 2 |1(K + oz11- - L,) op |1 + -2 11
-
7
|I(K + + |1 + oē 11111,10 11u1
2(1 t)
(As, 0 3 K La x
1-1
$ 7
11(K + oz op 111111 Amar (Ap) 2(1-t) t)
(Putting bound on, | + of () 12 op
from equation 22 below)
7 2(1 -t)
1 Ammnn (K of
11u112 Amaz (AD)
For, (In + o we can get upper bound like following,
=S= KS +11+ SF STK K
With K = USUT and R= E1/2UT 8, Ly R(R R (1+-1)-1pp -RT, we have
I, U,, 1/20
Due to the matrix inversion lemma, we have
Ly RR TRR +yT)
= 1 - -RRR +r1)-1
-I- +E++I++RA RRT -)-1
= I -
with
D -E+ RR (Σ (I)
② 1/241/207 SS S
1/2/2 U/1
and +- I(E+>1)-
If the sketching matrix S satisfics Amaz (Φ - T SSTU$1/2 = Amaz (D) < t for t E (0. 1) where Xmar denotes the maximum eigenvalue derive the lower bound for Ta the following,
29

---

Published in Transactions on Machine Learning Research (03/2026)
(0-01----00) 111-
+ 71)-1/2
t 1 11-7(++)1)) 1/2
1
1 + 71) 11/2 1/2
(As, ()- = (1=)
1-1
1-
7
( -
Now, lets put this Ly into Ly:
L =-!// L,
- US!/
(1-1) (E (E+11)- 1
= -
(12)) 2 + 711 -11- UT
= L
Therefore, = = U [x. (1. ( E -1)
U the lower bound for Toy. lerm we need lower bound on by which is L'p
For the upper bound of | + 1) 1
||LL, + - op 1
(Ln + o
U 2 ( E 71) UJ + t
(u 2 - (1--) 22 + 71) 71) - (T)
lpp
Using the pigendevomposition K = UEUT, and orthogonality of U, the operator norm simplifies as following,
Given the eigendecomposition of the kernel matrix K = UUUT, where Σ = diag(A),.... An), the following expression arises in the analysis,
(u Σ 1x-((0 (1--) M E(E ((=-12)++021 UJ 1
= =
=|< [1 1-t
- (2 (0+ +1) (2-++=1111 + +o?
op
op
Lets denote, AD = (I (I[I- -- 13(1-120 -E+
+1+ (1 + to?1)
-1 then we have,
||(,, + - - Amaa
(22)
Now, lets bound log determinant. term,
30

---

Published in Transactions on Machine Learning Research (03/2026)
2 log [K + of log [4, + |4, of
Now, we use proposition Bhatia 2013 Boyd & Vandenberghe (2004) which says for any two symmetric positive semi-definite matrix, A - and
A 3 al following inequality holds,
[log| A + A - log All < TT(AA S <aTr(A-1)
In above proposition if, A Ly + I A K by and (1=)) then,
1log |A+ Al - logg |A| = log [,, + oř (K - L2)1 - log L, + of-!l
log (K T] log (L., rz
5 Tr [L, + 1- - Ln)]
(Ly + oل -1 7
ST
7
Tr [L, + oē -1
1 t
7
Tr [L, + -
(Since, Ly t Lig and L, is PSD)
11tt t
7
-- Σ E(Σ+~I) U- + toē
Tr
U
1 t
1-
(1-))
y (1)+ + UT) -17
Tr
U
Y
Tr
3 7 ( + 71)
+ +
1
1-1
(Using the orthogonality of U)
Therefore bound for log determinant term is,
7 TI [ 111--
+ to21) )o?2) "
1 |log| (K + o. log [(I., ofD11 s 2(1-t)
Σ I 1- 1-
Since we are saying, AD = (I[[- 22 + to?
we have,
1 //2 |oog (K - o2 - log (Ly + 2i - 2(1 - t) Tr(Ap)
Now, combining both terms bound we will get overall bound
(K + o021) )
[((K) C((,)) S 2(1 - 1)
-1a1112 Amaz (AD) + Tr(Ap)
31

---

Published in Transactions on Machine Learning Research (03/2026)
Corollary 8 (Uniform NLML Approximation).
Assume that the sketching parameters satisfy
ty
0
1-t+t
Then, with probability at least 1 over the sketching randomness, for all y E R",
11y12
- - ((()) 7 2(1 - 1)
o
17
by
1-1+7
1-
Proof. From Theorem
with probability at least 1 - 6,
l1v11
[CK) [(In)) s 2(1 t)
Amax ( Tr(Ap)
Amina (K + oz
Bounding the quadratic term. Since N 0,
Amin (K + oou)) 2 oz
From the spectral analysis of AD.
Amax(AD)
0 1-t+t
17
Therefore,
11y11 Amiu (K + o]
11y13
Amax (Ap) 5
2
17
1-141
Bounding the log-determinant term. By definition,
Tr(Ap) Σ
0,
70
+ 0
As shown in the predictive mean analysis,
70;
ty
2 - +
(1- (1-t)(0: 7)
hence
Tr(Ap) S
Σ
by
1=1 o.
1-t+r
Combining the bounds. Substituting the above bounds yields the stated result.
32

---

Published in Transactions on Machine Learning Research (03/2026)
E Additional Experimental Details
In this section, we provide supplementary experimental results obtained using the Matern kernel in addition to the RBF kernel used in the main paper. These additional experiments are designed to validate the robustness of our proposed Nystrom ridge leverage score sketching method across different kernel choices.
Consistent with our findings in the main tcxt, the results with the Matern kernel confirm that our method continues to oulperform all baseline melhods across multiple datasets and evaluation metries. In particular, it achieves superior performance in terms of Negative Log Predictive Density (NLPD), proper scoring rule that captures both prediction accuracy and uncertainty calibration. Our approach also yiclds lower predictive mean and variance errors, indicating more accurate posterior approximations of the underlying Gaussian Process Regression model.
These trends hold across all evaluated datasets, and the improvements remain significant despite the change in kernel. Importantly, the opprozimate ridge leverage score algorithm (used to efficiently construct the sketch matrix) remains effective, demonstrating both scalability and predictive reliability of framework. Results from these additional experiments pest-performing entries are shown in bold. We note that while several scalable variational methods exist, SVGP is included as a strong and widely adopted representative of the variational family. Our method demonstrates consistently lower NLPD and better uncertainty calibration than SVGP, IerGP, and other baselines across all kernel configurations, highlighting its robustness and effectiveness.
presented in accompanying tables and figures, where the
Table 5 compares predictive performance and memory usage across methods. While Random Fourier Features (RFF) achieves the better NLPD and MSLL, it incurs significantly higher memory cost (over GB) due to dense high-dimensional features. In contrast, Nystrom methods provide more memory-elficient low-rank approximations.
Among them, the proposed Nystrom (ridge leverago) method achieves a strong accuracy-cffiiinn tradc-off, outperforming uniform, standard leverage sampling and other baselines. Overall, these results highlight that ridge leverage Nystrom offers more scalable alternative to RFF, delivering competitive accuracy with substantially lower memory requirements.
Table I highlights the significant memory advantages of the proposed Nystrom (Ridge Leverage) approach. While full-data GPR requires approximately 23.48 GB to store the kernel matrix for the Protein dataset. (n ~ 45,745 730), reflecting its O(2²) memory complexity, our method requires only 2.25 GB to 5.64 GB depending on the sketch size. Notably, our approach also maintains a substantially smaller memory footprint than Random Fourier Features (RFF), which consumes between 12.18 GB and 12.96 GB in this setting. This demonstrates that prioritizing statistically informative points via ridge leverage scores yiclds a more compact and scalable representation compared to the higl-dimensional, dlla--independenl features required by RFF to achieve comparable posterior accuracy.
Table 1: Memory usage (MB) across subset sizes on the Protein datasct (around 46K samples). All values are averaged over five random trials with standard deviations shown after +. Full-data GPR requires around 23.48GB.
Method
2 %
4%
6%
8%
10%
Full Data (Exact GP)
23477.220
Uniform
6709.6 3678..6.77.
1178.4 7.8
4302.0 10.6
1421.3 + 13.7
Leverage
4121.3 19.1
1109.2
4178.3 +: 7.6
4302.0 10.6
44211 13.6
KMeans
4121.9 18.7
4109.8
4178.3
4302.0 10.6
4424.3 13.7
SVGP
1756.8 1657.6
1121.5 73.2
1734.7
2372.0 73.3
3036.2 + 73.9
IerGG
2034.741.00
2034.7.1.00
2034.7
2.04.710.00
2034.7-000
Nyström (Uniform)
1953.9 75.5
79.
3753.2
4687.5 89.2
5644.0 1: 93.7
Nyström (Leverage)
2255.6 347.5
2843.8
3754.7184.9
4689.1 89.0
5645.5 93.6
Nyström (Ridge Leverage)
2253.5 + 347.7
2842.1
3753.7+
4687.7 + 89.2
5644.3 + 93.7
Random Fourier Features
12180.2 17.6
12258.7 0.0
12493.2
12727.7 0.0
12962.5 0.0
33

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Comparison negative log predictive density (NLPD), mean standardized log loss (MSLL) and
runtime (in seconds) for Gaussian Process Regression (GPR) methods on the UCI Elevators dataset using the RBF kernel. Each experiment repeated over random seeds, and we report the mean and standard deviation across runs. Lower values are better for all metrics.
Method
Subset %
NLPD
MSLL
Time (s)
Full Data
-0.7540
-0.8224
38.45
Uniform
2%
0.4791 + 0.1192
0.4107 0.1192
5.6802 0.0415
Leverage
2%
0.2869 + 0.0543
0.2185 + 0.0543
5.6877 + 0.0433
Kmeans
2%
0.4359 + 0.0824
0.3675 = 0.0824
5.6594 + 0.0470
Svgp
2%
40.7380 + 0.2973
40.6696 0.2973
114.2088 = 0.3616
Itergp
2%
-0.2166 + 0.0000
-0.2850 + 0.0000
8.7699 + 0.2120
Nystrom(unifone
2%
0.2323 + 0.0000
0.1639 + 0.0000
16.4537 + 0.0354
Nystrom((everage)
2%
-0.1789 0.0352
-0.2473 + 0.0352
17.0823 0.0296
Nystrom(ridge leverage)
2%
-0.5551 0.0034
0.6235 + 0.0034
21.1613 0.1409
Uniform
4%
0.2258 + 0.0428
0.1574 + 0.0428
6.1125 + 0.0073
Leverage
4%
0.1558 + 0.0174
0.0874 + 0.0174
6.1252 + 0.0429
Kmeans
4%
0.3369 = 0.0516
0.2685 = 0.0516
6.1207 0.0113
Svgp
4%
41.6886 + 0.2967
41.6202 0.2967
116.2481 + 0.1822
Itergp
4%
-0.3124 + 0.0000
-038808 = 0.0000
22.2070 0.1170
Nystrom(uniform)
4%
-0.4713 + 0.0492
-.53397 = 0.0492
18.6045 = 0.0160
Mystrom(leverage
4%
-0.3172 + 0.0174
00.855 + 0.0174
19.0833 + 0.0375
Nystrom leverage)
4%
0.63300 + 0.0032
0.7064 + 0.0032
21.7761 + 0.0655
Uniform
6%
0.1517 + 0.0509
0.0833 0.0509
6.4596 0.0220
Leverage
6%
0.1157 +: 0.0173
0.0473 + 0.0173
6.4745 + 0.0210
Kmeans
6%
0.3647 + 0.1634
0.2963 = 0.1634
6.4564 0.0371
Svgp
6%
40.0423 + 0.1234
39.9739 = 0.1234
118.1078
+ 0.2941
Itergp
6%
-0.3415 + 0.0000
-0.4099 = 0.0000
43.3009 0.0840
Nystrom(uniform)
6%
-0.5955 + 0.0074
-0.6009 + 0.0074
20.4320 0.0431
Nystrom((everage)
6%
-0.4420 0.0122
-0.5104 = 0.0122
20.9021 0.0344
(ridre leverage)
6%
0.6815 + 0.0021
0.7499 = 0.0021
22.7822 0.1542
Uniform
8%
0.1517 + 0.0308
0.0833 + 0.0308
7.1314 +: 0.0217
Leverage
8%
0.0987 + 0.0140
0.0303 + 0.0140
7.2971 0.0233
Kmeans
8%
0.3060 0.0824
7.2365 0.0169
Svgp
8%
38.7838 + 0.2665
38.7154 = 0.2665
122.1149 + 0.1595
Itergp
8%
-0.3629 + 0.0000
-0.4313 0.0000
70.7370 = 0.2089
Nystrom(uniform)
8%
-0.6465 + 0.0031
-0.7149 = 0.0031
20.5632 = 0.0910
Nystrom(leverage
8%
-0.5343 + 0.0088
-0.6027 = 0.0088
21.2114 0.2373
Nystrom (ridge leverage)
8%
0.7115 + 0.0021
0.7799 + 0.0021
22.5875 + 0.1829
Uniform
10%
0.1202 + 0.0399
0.0518 0.0399
7.1129 0.0575
Leverage
10%
0.0541 +: 0.0274
-0.0143 + 0.0274
7.2908 0.0144
Kmeans
10%
0.2613 + 0.0984
0.1929 + 0.0984
7.2396 + 0.0129
Svgp
10%
37.5595 + 0.0773
37.4911 0.0773
127.6543 + 0.3152
Itergp
10%
-0.3798 + 0.0000
-0.4482 + 0.0000
104.1623 = 0.1610
Nystrom(uniform)
10%
-0.6804 + 0.0022
- + 0.0022
22.1774 0.0991
Nystrom(leverage.
10%
-0.5982 0.0048
-0.6666 + 0.0048
22.7742 =
Nystrom(ridge leverage)
10%
-0.723 0.0024
0.8007 + 0.0024
23.9634 0.0825
34

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Comparison negative log predictive density (NLPD), mean standardized log loss (MSLL) and
memory (in MB) for Gaussian Process Regression (GPR) methods on the UCI California Housing datasct using the Matern kernel. Each experiment is repeated over random seeds, and report. the mean and standard deviation across runs. Lower values better for all metrics.
Method
Subset %
NLPD
MSLL
Memory (MB)
Full Data
0.8987
-0.6563
4797.03
Uniform
2%
2.0318 0.1416
0.4768 + 0.1416
1169.2865 = 638.5115
Leverage
2%
1.8690 + 0.0424
0.3140 + 0.0424
853.2596 + 6.4547
Kmeans
2%
2.0137 = 0.1869
0.4587 + 0.1869
854.2215 = 6.4547
Svgp
2%
10.9006 = 0.1611
9.3456 + 0.1611
278.02 078.02774 + 291.5613
Itergp
2%
1.0109 + 0.0000
-0.5442 + 0.0000
304.2607 + 0.0000
Nystrom(uniform)
2%
1.1239 + 0.0189
-0.4311 0.0189
507.2209 = 13.1754
Rff
2%
0.8915 0.0068
-0.6635 0.0068
2473.1115 0.0191
Nystrom(ridge leverage)
2%
1.0696 0.0072
0..854 = 0.0072
552.9800 = 77.3543
Uniform
4%
2.0952 + 0.1466
0.5402 + 0,1466
887.7567 = 6.9622
Leverage
4%
1.9214 + 0.0978
0.3663 + 0.0978
887.6729 = 6.9197
Kmeans
4%
2.0195 = 0.1899
0.4645 = 0.1899
887.7592 = 6.9622
Svgp
4%
10.7327 0.1574
9.1777 + 0.1574
245.7788 + 12.9210
Itergp
4%
0.9691 = 0.0000
-..859 = 0.0000
304.2607 = 0.0000
Nystrom(uniform)
4%
0.9720 = 0.0076
0.5830 = 0.0076
723.8914 13.8915
RMT
4%
0.8688 + 0.0034
0.6862 + 0.0034
2520.9160 + 0.0000
Nystrom(ridge leverage)
4%
0.9479 + 0.0075
0.6071 + 0.0075
724.5246 = 13.9494
Uniform
6%
2.0469 0.0724
0.4918 + 0.0724
873.5581 = 8.3471
Leverage
6%
1.8013 :: 0.0543
0.2462 + 0.0543
873.7120 + 8.4859
Kmeans
6%
1.9992 = 0.1683
0.4441 + 0.1683
874.0355 = 8.1051
Svgp
6%
10.3749 + 0.1324
8.8199 + 0.1324
371.1588 = 12.8385
Itergp
6%
0.9521 = 0.0000
-0.6029 = 0.0000
304.2607 = 0.0000
Nystrom(uniform)
6%
0.9040 + 0.0074
0.6510 + 0.0074
948.4028 + 14.9609
Rff
6%
0.8546 + 0.0019
0.7005 0.0019
2568.7104 + 0.0000
Nystrom(ridge leverage)
6%
0.8791 0.0059
0.6759 = 0.0059
948.4933 = 15.0164
Uniform
8%
1.9365 + 0.0553
0.3815 + 0.0553
889.3438 + 1.6545
Leverage
8%
1.7552 + 0.0312
0.2002 + 0.0312
889.0636 + 1.9553
Kmeans
8%
1.9944 + 0.1438
0.4394 + 0.1438
888.8565 = 1.6863
Svgp
8%
10.0673 0.1
8.5122 + 0.1195
499.9782 + 12.4930
Itergp
8%
0.9474 = 0.0000
-0.6076 0.0000
304.2607 = 0.0000
Nystrom)uniform)
8%
0.8518 = 0.0039
0.7032 = 0.0039
1177.9729 = 15.1928
Rff
8%
0.8548 = 0.0026
00.7002 = 0.0026
2616.5054 + 0.0000
Nystrom(ridge leverage)
8%
0.8387 + 0.0017
0.7164 + 0.0017
1177-5941 + 15.3684
Uniform
12%
1.8548 0.0359
0.2998 + 0.0359
934.2386 5.1368
Leverage
12%
1.6833 +:: 0.0524
0.1283 + 0.0524
934.4450 +: 5.2379
Kmeans
12%
1.8868 +- 0.0508
0.3318 + 0.0508
934.0741 = 5.3000
Svgp
12%
9.6147 + 0.1137
8.0597 + 0.1137
769.6117 + 25.6772
Itergp
12%
0.9388 + 0.0000
-0.6162 + 0.0000
304.2607 = 0.0000
Nystrom(uniform)
12%
0.8121 + 0.0012
-0.7429 0.0012
1648.8987 + 33.2553
Rff
12%
0.8455 + 0.0042
-0.7095
0.0042
2712.0942 + 0.0000
Nystrom(ridgo leverage)
12%
0.8103 0.0009
-0.7447 0.0009
1649.0237 33.2553
35

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Comparison of negative log predictive density (NLPD), mean standardized log loss (MSLL) and memory usage (in megabyles) for Gaussian Process Regression (GPR) methods on the UCI Airfoil Self-Noise dataset using the Matern kernel. Results are averaged over 5 random seeds, with standard deviations reported.
Method
Subset %
NLPD
MSLL
Memory (MB)
Full Data
2.4025
-0.9750
51.12
Uniform
2%
4.1267 + 0.3406
0.7492 + 0.3406
26.6922 = 5.0963
Leverage
2%
3.8720 = 0.0404
0.4945 + 0.0404
26.1291 = 0.1068
Kmeans
2%
3.9280 0.1716
0.5505 + 0.1716
24.1977 = 0.1063
Svgp
2%
4.7646 0.0660
1.3871 = 0.0660
19.9855 = 2.2387
Itergp
2%
3.0549 0.0000
-0.3225 + 0.0000
19.2832 + 0.0000
Nystrom(uniform)
2%
2.8919 = 0.3355
-0.4856 = 0.3355
20.7281 = 0.1014
Nystrom (leverge))
2%
3.5628 + 0.0000
0.1853 0.0000
22.7260 = 0.4168
RMT
2%
3.4720 + 0.1643
0.0945 0.
53.0475 + 0.0027
"ystrom(ridge leverage)
2%
2.6845 + 0.0065
0.6929 + 0.0065
20.9984 - 0.4168
Uniform
4%
3.9302 0.1720
0.5527 + 0.1720
24.2454 = 0.0131
Leverage
4%
3.7505 + 0.0482
0.3730 + 0.0482
26.1022 = 0.0133
Kmeans
4%
3.8046 = 0.0635
0 0.0633 0.0635
24.2459 = 0.0131
Svgp
4%
4.6540 = 0.0517
1.2765 17765=00000 0.0517
20.4402 = 0.1234
Itergp
4%
2.8807 = 0.0000
-0.4968 + 0.0000
19.2832 + 0.0000
Nystrom (uniform)
4%
2.6060 0.0017
-0.7714 0.0017
22.2289 + 0.0959
Dystrom(leverage)
4%
2.6762 0.0110
-0.7013 0.0110
22.7094 = 0.0959
Rff
4%
2.8534 = 0.0556
-0.5241 = 0.0556
53.0543 = 0.0027
(astrom(ridg' leverage)
4%
2.6024 + 0.0063
0.7751 + 0.0063
22.2401 = 0.0959
Uniform
6%
3.7681 0.0825
0.3906 + 0.0825
24.3877 = 0.0225
Leverage
6%
3.6811 0.0328
0.3037 = 0.0328
26.1542 = 0.0227
Kmeans
6%
3.8331 0.1226
0.4556 0.12226
24.3882 = 0.0225
Svgp
6%
4.3329 = 0.0323
0.9554 0.0323
22.1729 = 0.1336
Itergp
6%
2.8622 + 0.0000
-0.5153 + 0.0000
19.2832 + 0.0000
Nystrom unnffrrm)
6%
2.5551 + 0.0058
0.8224 0.0058
23.7790 = 0.1021
Nystrom(leve (leverage)
6%
2.5932 0.0074
0.7842 + 0.0074
23.8132 + 0.1021
Rff
6%
2.7679 0.0901
-0.6096 0.0901
53.0650 = 0.0047
Nystrom (ridge leverage)
6%
2.5498 0.0046
0.8277 = 0.0046
23.7907 = 0.1021
Uniform
8%
3.6592 + 0.0534
0.2817 + 0.0534
24.5618 = 0.0307
Leverage
8%
3.6385 + 0.0174
0.2610 + 0.0174
26.2272 = 0.0309
Kmeans
8%
3.7262 = 0.1350
0.3488 + 0.150
24.5623 = 0.0307
Svgp
8%
4.2478 0.0368
0.8703 + 0.0368
23.9900 = 0.1410
Itergp
8%
2.8677 0.0000
-0.5988 + 0.0000
19.2832 + 0.0000
Nystrom(unifone)
8%
2.5166 + 0.0032
0.8609 0.0032
25.3638 = 0.1064
Systrom(eeverage)
8%
2.5431 0.0058
0.8344
25.3979 = 0.1064
Rif
8%
2.7147 + 0.0553
0.6628 + 0.0553
53.0799 = 0.0062
Wysgre leverage)
8%
2.5154 + 0.0039
0.8621 + 0.0039
25.3755 = 0.1064
Uniform
10%
3.5781 0.0542
0.2007 + 0.0542
24.7744 - 0.0400
Leverage
10%
3.5993 + 0.0197
0.2218 + 0.0197
26.3231 = 0.0402
Kmeans
10%
3.6498 + 0.1140
0.2723 + 0.1140
24.7749 = 0.0400
Svgp
10%
4.0887 + 0.0631
0.7112 + 0.0631
25.9065 = 0.1508
Itergp
10%
2.8104 0.0000
-0.5671 + 0.0000
19.2832 4 0.0000
Nystrom (uniform
10%
2.4918 + 0.0028
-0.8857 0.0028
26.9956 = 0.1123
Nystrom )
10%
2.5148 0.0021
-0.8627 0.0021
27.0298 = 0.1123
RIT
10%
2.6256 0.0686
-0.7519 + 0.0686
53.0994 = 0.0082
Nystrom leverage)
10%
2.4890 + 0.0124
0.8915 + 0.0154
27.0073 = 0.1123
36

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: Comparison nogative log predictive density (NLPD), mean standardized log loss (MSLL) and memory usage (in megabytes) for Gaussian Process Regression (GPR) methods on the UCI Elevators datasct using the Matern kernel. Results averaged over random seeds, with standard deviations reported.
Method
Subset %
NLPD
MSLL
Memory (MB)
Full Data
0..485
0.9169
4055.47
Uniform
3%
0.3001 + 0.0167
0.2317 0.0167
247.6358 67.6086
Leverage
3%
0.2597 + 0.0188
0.1913 0.0188
215.1867 1 2.7777
Kmeans
3%
0.4627 = 0. 0.1838
0.3943 = 0.1138
215.1867 = 2.7777
Svgp
3%
4.5856 = 0.0548
4.5172 = 0.0548
161.6910 + 16.6343
Itergp
3%
-0.1158 0.0000
-0.1842 = 0.0000
0.0000
224.2437 = 0.0000
Nystrom(uniform)
3%
-0.7421 + 0.0086
-0.8105 + 0.0086
464.0999 16.9594
Rff
3%
0.9004+0.0000
0.9688 = 0.0023
2114.6526 + 0,0102
Nystrom(ridge loverage)
3%
-0.7195 + 0.0092
-0.7879 0.0092
482.6678 16.9594
Uniform
5%
0.2454 + 0.0407
0.1770 + 0.0407
236.0326 + 4.4180
Leverage
5%
0.2018 + 0.0127
0.1334 + 0.0127
236.0498 + 4.4249
Kmeans
5%
0.3895 + 0.0902
0.3211 0.0902
236.0340 4.4216
Svgp
5%
4.5543 + 0.0665
4.4859 0.0655
272.8118 11.1136
Itergp
5%
-0.1522 0.0000
-0.2206 0.0000
224.2437 0.0000
Nystrom(uniform)
5%
-0.7845 0.0042
-0.8529 0.0042
655.1745
12.2674
Rff
5%
0.9048 0.0012
0.9732 0.0012
2154.9487 0.0000
Nystrom(ridge leverage)
5%
-0.7510 + 0.0042
0..195 + 0.0042
674.4363 11.9642
Uniform
6%
0.2182 + 0.0321
0.1498 0.0321
250.7646 2.6318
Leverage
6%
0.1745 + 0.0132
0.1061 0.0132
251.1238 2.7003
Kmeans
6%
0.3857 + 0.1038
0.3173 + 0.1038
250.7249 + 2.6072
Svgp
6%
1.5257=00033
4.4573 + 0.0321
327.4325 = 5.6037
Itergp
6%
-0.1639 0.000
-0.2323 = 0.0000
224.2437 = 0.0000
Nystrom(uniform)
6%
+.7980
-0.8664 + 0.0049
750.4968 + 6.0018
Rif
6%
0.9062
0.9746 0.0024
2175.1685 = 0.0000
Nystrom(ridge leverage)
6%
-0.7648 0.0015
0.8332 + 0.0015
770.0260 + 6.0896
Uniform
9%
0.1695 + 0.0139
0.1011 + 0.0139
238.6730 + 5.9560
Leverage
9%
0.1019 + 0.0286
0.0335 + 0.0286
238. 088.8644- + 6.2307
Kmeans
9%
0.2867 + 0.0771
0.2183 0.0771
238.5708 5.9930
Svgp
9%
4.3862 + 0.0311
4.3178 0.0311
494.8212 15.9441
Itergp
9%
-0.2605 = 0.0000
224.2437
0.0000
Nystrom(uniform)
9%
0.82122
-0.8896 = 0.0039
1036.9442 + 19.7361
Rff
9%
0.9045
0.9729 = 0.0037
2235.8320 = 0.0000
Nystrom(ridge leverage)
9%
-0.7907 + 0.0008
0.8591 + 0.0008
1055.7627 + 19.2402
Uniform
10%
0.1382 + 0.0212
0.0698 0.0212
242.4258 0.8025
Leverage
10%
0.0663 + 0.0291
-0.0021 + 0.0291
242.4521 + 0.6862
Kmeans
10%
0.2509 + 0.0840
0.1825 0.0840
242.2792 + 1.2604
Svgp
10%
4.3341 + 0.0123
4.2657 + 0.0123
556.8342 + 4.8722
Itergp
10%
-0.1979 + 0.0000
-0.2663 + 0.0000
224.2437 + 0.0000
Nystrom()n
10%
-0.8249 + 0.0035
-0.8933 + 0.0035
1151.1535 + 7.5045
Rff
10%
-0.9079 0.0018
-0.9763 = 0.0018
2256.6812 + 0.0000
pestromm(idg'' leverage)
10%
-0.7977 0.0025
-0.8661 + 0.0025
1169.7740 + 7.4129
37

---

Published in Transactions on Machine Learning Research (03/2026)
Fall GPR Unifom
Leverage Kmeans
Sugp +
Mystrom uni
+
Random
Nystrom Iis
Protein Dataset
Elevators Dataset
Housing Dataset
3.0
Negative Likelhood
2.5
2.0
1.5
2.0
1.0
2.0
0.5
2
10
2
10
Subset size of TUll data)
Subset size 0% of full data)
Subset $20 (% of full data)
Figure 5: Comparison of Negative Log Likelihood (NLL) across different subset sizes and Gaussian Process approxi- mation methods on the Protein, Elevators, and Housing datasets using the Matern kernel. Each subplot reports the mean and standard deviation over five random trials. The dashed horizontal line denotes the performance of the full-data (Exact GP) model.
uniform
age
sygp
nystrom_urt
nystrom RMSE vs Sample Size (%%
Mean Error vs Sample Size (%)
Variance Error vs Sample Size (%)
Predictive Moan Error
0.8
Predictive Varfance Eroor
0.6
1.5
RISEE
0.4
2-
0.0
0.0
6%
Sample Size (96) Pull Data
Sample Sizee in (96) of Pull Data
Sample Size in (96) of Full Data
Figure 6: Results on California Housing Dataset. Evaluation of Gaussian Process Regression methods on the
Housing dataset using the Matern kernel. Predictive mean error, predictive variance error, and RMSE plotted versus subset size. Ridge leverage based GPR yields the best tradeoff across metrics. All results are averaged over 5 random trials, with standard deviations shown as error bars. The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
uniform
age
kmeans
sugp
IterGp
nystrom
random !f
nystrom_t
Mean Error Sample Size (%)
Variance Error Sample Size (%)
RMSE Sample Size
14
0.30
1.0
12
0.25
10
Predective Men
0.8
0.15
Predictive
0.4
0.10
0.05
356
536
6%
9%
10%
35.
5%
67%
5%
10%
39
5%
6%
9%
10%
Sample Size In (36) Full Data
Sample Size in (96) of Pull Data
Sample Size (%) Full Data
Figure 7: Results on UCI Elevators Dataset. Evaluation of Gaussian Process Regression methods on the UCI Elevators dataset using Matern kernel. Predictive mean error, predictive variance error, and RMSE are plotted versus subset size. Ridge leverage based GPR yields the best tradeoff across metrics. All results are averaged over 5 random trials, with standard deviations shown as error bars. The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
38

---

Published in Transactions on Machine Learning Research (03/2026)
uniiform
kmeans
BerCpp
uni
random
nis
Mean Error vs Sample Size (%)
Variance Error vs Sample Size (%)
RMSE vs Sample Size (%%
1.0
2.5
0.8
Prodictive Variance
2.0
2.0
RMSE
Predictive
1.0
0.2
0.0
4%
6%
0.0
49%
10%
0.0
10%
Sample Size Data
Sam pie (%) FUll Data
Sample Size in (9) of Full
Figure 8: Results on California Housing Dataset. Evaluation of Gaussian Process Regression methods on the Housing dataset using the RBF kernel. Predictive mean error, predictive variance error, and RMSE. plotted versus subset size. Ridge Leverage based GPR yields best tradeoff across metries. All results are averaged over random trials with standard deviation shown as bars. The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
uniform
leverage
kmeans
Sygp
RerGp
nystrom .uni
rystrom_
random_r _f
apstromurts
Mean Error Sample Size (%)
Variance Error Sample Size (36)
RMSE Sample Size (%)
0.150
Predictive Vartance Error
Predictive Moan
0.100
RMSE
0.050
0.000
10%
8%
10%
Sample Size In (36) of Full
Data
Sample Size In (%) of Full Data
Sample Size in (56) of Full Data
Figure 9: Results on Airfoil Self-Noise Dataset. Performance of various GPR methods on the Airfoil dataset. using the RBF kernel. Ridge leverage based sketching again outperforms uniform, SVGP and RFF in both predictive accuracy and variance estimation. All results are averaged over 5 random trials with standard deviation shown as error bars. The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
leverage
svep MSLL vs Sample Size (%)
nystrom_ini
nystrom_le
nystrom_rk
NLPD Sample Size (%)
Memory (MB) vs Sample Size (%)
4.
Memory (MB)
30
20
276
6%
8%
10%
41%
8%
10%
4%
6%
10%
Sample Size in (%) of Full Data
Sample Size in (96) of Full Data
Sample Size in (96) of Full Data
Figure 10: Results on Airfoil Self-Noise Dataset. Performance of various GPR methods on the Airfoil datasel using the RBF kemnel. All results are averaged over random trials, with standard deviations shown The dashed horizontal line indicates the performance of the full-dataset (exact GP) model. Due to the relatively small dataset size, the memory usage of the proposed and baseline approximation methods remains nearly constant across subset sizes from 2% to 10%. Nevertheless, all approximation methods consume substantially less memory than the full GP, highlighting their efficiency. RFF requires substantially higher memory to match posterior accuracy achieved by our method.
bars.
39

---

Published in Transactions on Machine Learning Research (03/2026)
nystrom lev
Sample (%6
Sample
Sample (%6
0.16
20.0
0:12
0.10
Predicáive Variance
12.5
0.08
10.0
Prodictive
0.06
7.5
0.02
0.00
0.0
Sample Size in (%) of Full Data
Sample Size (%) Full Data
Sample Size (96 of Full Data
Figure 11: Results on Airfoil Self-Noise Dataset. Performance of various Gaussian Process Regression (GPR) methods the UCI Airfoil Self-Noise dalaset using the Matern kernel. Ridge leverage based sketching consistently outperforms uniform sampling and SVGP in both predictive accuracy and variance estimation. All results are averaged 5 random trials, with standard deviations shown bars. The dashed horizontal line indicates the performance of the full-dataset (exact GP) model.
40


Reviews:
[{"review_id": "ocQL2nKlGN", "reviewer": "Reviewer_gy7Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Inspired by the ridge leverage score method from El Alaoui & Mahoney (NeurIPS 2015), the submitted paper introduces a sketch-based approximation for Gaussian process regression (GPR). The ridge leverage score method from El Alaoui et al. extends the notion of statistical leverage scores to the setting of kernel ridge regression, so it is possible to identify a sampling distribution that reduces the size of the sketch (required number of columns). In this case, we are developing a similar method for setting low-rank approximations of the kernel matrix of the GPR model. Such a low-rank approximation is used for the estimation of the (posterior) predictive mean, the variance, and the log-marginal likelihood.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper seems to be well-written and clear enough for a reader familiar with GPR models and low-rank approximations. However, the way sketching is presented and how the previous work is revisited is perhaps a bit too quick for my taste, and I had to go back to El Alaoui & Mahoney (NeurIPS 2015) to understand what was going on. In that sense, the paper is not really self-contained in its current state. Additionally, as long as I revisit such previous work, I can see that the results are very close to the ones presented in the current submission, more than what one should expect in this case.\n\nJust to give some particular examples of the previous comments here: i) the Algorithm section 4, with the definition of the ridge leverage score is the same one as in section 3.3 of El Alaoui & Mahoney (NeurIPS 2015), but less clear and adapted for the GPR data-noise parameter. The theoretical guarantees from Section 5 are the same ones introduced in Section 3.1 of El Alaoui & Mahoney, but adapted to mean, variance, and marginal likelihood in the following sections. Even the effective dimension is also taken and adapted. \n\nHaving Theorem 3 from El Alaoui & Mahoney (NeurIPS 2015), with the bound on the number of columns for the sketch approximation, and also having the sketch of the kernel matrix $L_{\\gamma}$, I see these two results re-used three times for the predictive mean, variance and log-marginal-likelihood approximation. From the perspective of a researcher insterested in GPR, sections 5.2, 5.3 and 5.4 seems a repetition, with the changes only visible on the inequality --- which is not discussed in depth.", "audience_yn": "Yes", "audience_explanation": "I do think low-rank approximations for GPR are always of interest for the TMLR audience. However, I have some concerns regarding how this one is presented in the current submission. In some way, I am afraid that the contributions are not that visible, and the work is just a result of extending the El Alaoui & Mahoney insights for the specific case of predictive means and variance and the LML approximation. In the same way, the story does not really align with well-known approximations or other papers that make advances on the Nystrom method for GPs. Additionally, when considering the inequalities, it's not really discussed in depth what are the dominant terms, or at least, not in a way that make one understand how the number of columns, structure of the sketch of the kernel matrix, or just the approximation are directly affecting the inequality.", "requested_changes": "- I would like to see a manuscript which is (at least) as self-contained as El Alaoui & Mahoney (2015).\n- The technical results around sections 5.2, 5.3, and 5.4 are somewhat repetitive, and definitions are even repeated. In many ways, one can think that the sketch of the kernel matrix is just being substituted in the original GPR terms. I would like to see this changed in some degree.\n- How inequalities are obtained, what are the main differences and contributions (different from the ones El Alaoui & Mahoney) should be much clearer in the manuscript, as well as connecting with the theoretical derivations included in the Appendix, so the reader can follow and reproduce.\n- Experimental results are kind of fine, but I am afraid that the metrics provided for the particular UCI datasets don't really show the big picture, as sparse or spectral approximations have other advantages that are omitted in this case.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "J6lY7ayzcj", "reviewer": "Reviewer_Mqyt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Main contribution is establishing error bound for Gaussian process regression when kernel matrix is approximated due computational limitations. Given $(x_i, y_i)$ , the papers estimations an error for estimating $y(x_*)$ where $x_*$ is a single test sample. The main idea is combining approximation error of kernel and using closed form solution of Gaussian processes. \nHowever there are two fundamental issues with the analysis \n- (1) the result only holds for a single test sample $(x_*)$ in my opinion all deviation bounds should have sup over $x_*$. Suppose that sketching does well on one single test sample $x_*$. Since there are infinite choice for $x_*$, we can not apply union bound. Thus, the high probability statements in theorems 2 and 3 can not be used in practice when $x_*$ is random. \n- (2) High probability statements in theorems 2 and 3 are over randomness of the sketching and again input $x_1, \\dots, x_n$ is assumed to be frozen. These bounds are only informative if they holds when $x_1, \\dots, x_n$ are random.", "claims_and_evidence_yn": "No", "claims_explanation": "In my opinion, machine learning community will not find the established analysis informative as they are mostly numerical approximation errors for a single test sample. Please check my summary for details.", "audience_yn": "No", "audience_explanation": "Please check my comments above", "requested_changes": "Minimum correction is proving that the statement of theorem 2 holds for $\\sup_{x_*} | \\mu(x_*) - \\mu_S(x_*)|$ or when $x_1, \\dots, x_n, x_*$ are i.i.d. from an unknown distribution. All other theorems require a similar correction.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nzufc5gQLc", "reviewer": "Reviewer_vqEv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackle the scalability of Gaussian Processes to large datasets using sketching. Specifically, the sketching scheme proposed relies on ridge leverage score sub-sampling, following El Alaoui & Mahoney (2015). It provides theoretical guarantees together with empirical experiments. The main contributions of the paper are:\n1) theoretically, the authors provide learning bounds for the sketched posterior mean, posterior variance and Negative Log Marginal Likelihood;\n2) empirically, the authors provide experiments on real-world datasets to demonstrate the relevance of their method.\n\nStrengths:\n- The paper is rather well written, except the theoretical guarantees where the assumptions are not clearly stated and some discussions are missing.\n- The paper is clearly motivated.\n- The claims are supported by theoretical and empirical evidences.\n\nWeaknesses:\n- The contributions of this paper are incremental results following El Alaoui & Mahoney (2015), but this should not be an issue for TMLR.\n- The related works should mention Random Fourier Features.\n- The theoretical guarantees lack discussions.\n- The paper lacks some discussions on the limitations of the frameworks and future work directions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The theoretical claims are supported with sound proofs. However, I have a few remarks regarding the theory section and requested changes (see Requested Changes section).\n\nEmpirically, the authors provide a comparison with many baselines on real-worlds GP Regression datasets. However, I think that it would be relevant to add Random Fourier Features as a baseline, since it is the most standard kernel approximation technique together with sketching. Moreover, I have a question regarding the \"sample size\" in the figures' x-axis: Is it, in your case, the sketching size m? Or does that mean that you use small percentage of the full training dataset as training points? It should be clarified.", "audience_yn": "Yes", "audience_explanation": "I think it would interest the TMLR's audience as GPs are a very famous Bayesian framework for Machine Learning tasks and their expensive cost with respect to the number of training data is a major bottleneck.", "requested_changes": "A) Related works:\n\n1) Add a small discussion with Random Fourier Features since it is the other famous method to scale kernel methods by approximating the kernel itself directly, similarly to Nystrom approximation or sketching. In which cases would you rather use sketching over RFF (in terms of data or kernel at hand)?\n\nB) Theoretical guarantees:\n\n1) Assumptions: it seems that you consider two assumptions (on (i) $K-L_{\\gamma}$ and (ii) $k_{\\*}$), you should declare them clearly and discuss them. For assumption (ii), does it mean that for any test point $x^{\\*}$, there exists such a vector $\\alpha$? Then maybe you should use $\\alpha^{\\*}$ as notation.\n\n2) You should move the remark (Interpretation of α in the Predictive Mean and Variance Bounds) earlier to discuss assumption (ii). To what extent such an assumption is realistic? Do you have any specific examples?\n\n3) Your theorems lack discussions and interpretations. In particular, theorems 2 and 4 are not discussed at all. How theorem 2 compare with existing results?\n\n4) Role of \\beta? It seems like we would like it to be close to 1 but then the $p_{i}$s seem unrealistic. Could you elaborate a bit more?\n\n5) Could you provide learning rates with respect to $n$? You could derive a Corollary assuming a polynomial eigendecay for instance and specific settings for every parameters and hyperparameters ($\\alpha$, $\\gamma$, $\\sigma_{\\xi}$, $t$...) to obtain a final rate.\n\nC) Experiments\n\n1) Since you derive learning bounds, it would be nice to provide synthetic experiments in which you control the assumptions considered, such as the eigendecay of the kernel or this $\\alpha$ to show that there exist scenarios in which these assumptions make sense, and retrieve the theoretical results on an empirical example.\n\n2) \"sample size\" in the figures' x-axis: Is it, in your case, the sketching size m? Or does that mean that you use small percentage of the full training dataset as training points? It should be clarified.\n\n3) You do not provide figures showcasing the training time of the different methods. For instance, a figure with the evaluation metrics w.r.t. training time would be relevant.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)