Title: Supervised Kernel Thinning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Kernel methods

distribution compression

non-parametric regression
Abstract: The kernel thinning algorithm of Dwivedi & Mackey (2024) provides a better-than-i.i.d. compression of a generic set of points. By generating high-fidelity coresets of size significantly smaller than the input points, KT is known to speed up unsupervised tasks like Monte Carlo integration, uncertainty quantification, and non-parametric hypothesis testing, with minimal loss in statistical accuracy. In this work, we generalize the KT algorithm to speed up supervised learning problems involving kernel methods. 
Specifically, we combine two classical algorithms---Nadaraya-Watson (NW) regression or kernel smoothing, and kernel ridge regression (KRR)---with KT to provide a quadratic speed-up in both training and inference times. We show how distribution compression with KT in each setting reduces to constructing an appropriate kernel, and introduce the Kernel-Thinned NW and Kernel-Thinned KRR estimators. We prove that KT-based regression estimators enjoy significantly superior computational efficiency over the full-data estimators and improved statistical efficiency over i.i.d. subsampling of the training data. En route, we also provide a novel multiplicative error guarantee for compressing with KT.  We validate our design choices with both simulations and real data experiments.

Supervised Kernel Thinning

Abstract

The kernel thinning algorithm of [10] provides a better-than-i.i.d. compression of a generic set of points. By generating high-fidelity coresets of size significantly smaller than the input points, KT is known to speed up unsupervised tasks like Monte Carlo integration, uncertainty quantification, and non-parametric hypothesis testing, with minimal loss in statistical accuracy. In this work, we generalize the KT algorithm to speed up supervised learning problems involving kernel methods. Specifically, we combine two classical algorithms-Nadaraya-Watson (NW) regression or kernel smoothing, and kernel ridge regression (KRR)-with KT to provide a quadratic speed-up in both training and inference times. We show how distribution compression with KT in each setting reduces to constructing an appropriate kernel, and introduce the Kernel-Thinned NW and Kernel-Thinned KRR estimators. We prove that KT-based regression estimators enjoy significantly superior computational efficiency over the full-data estimators and improved statistical efficiency over i.i.d. subsampling of the training data. En route, we also provide a novel multiplicative error guarantee for compressing with KT. We validate our design choices with both simulations and real data experiments.

1 Introduction

In supervised learning, the goal of coreset methods is to find a representative set of points on which to perform model training and inference. On the other hand, coreset methods in unsupervised learning have the goal of finding a representative set of points, which can then be utilized for a broad class of downstream tasks-from integration [10, 9] to non-parametric hypothesis testing [8]. This work aims to bridge these two research threads.

Leveraging recent advancements from compression in the unsupervised setting, we tackle the problem of non-parametric regression (formally defined in Sec. 2). Given a dataset of n i.i.d. samples, ( x i , y i ) n i =1 , we want to learn a function f such that f ( x i ) ≈ y i . The set of allowable functions is determined by the kernel function, which is a powerful building block for capturing complex, non-linear relationships. Due to its powerful performance in practice and closed-form analysis, non-parametric regression methods based on kernels (a.k.a 'kernel methods') have become a popular choice for a wide range of supervised learning tasks [13, 20, 24].

There are two popular approaches to non-parametric kernel regression. First, perhaps a more classical approach, is kernel smoothing, also referred to as Nadaraya-Watson (NW) regression. The NW estimator at a point x is effectively a smoothing of labels y i such that x i is close to x . These weights are computed using the kernel function (see Sec. 2 for formal definitions). Importantly, the NW estimator takes Θ( n ) pre-processing time (to simply store the data) and Θ( n ) inference time for each test point x ( n kernel evaluations and n simple operations).

Another popular approach is kernel ridge regression (KRR), which solves a non-parametric least squares subject to the regression function lying in the reproducing kernel Hilbert space (RKHS) of a

specified reproducing kernel function. Remarkably, KRR admits a closed-form solution via inverting the associated kernel matrix, and takes O ( n 3 ) training time and Θ( n ) inference time for each test point x .

Our goal is to overcome the computational bottlenecks of kernel methods, while retaining their favorable statistical properties. Previous attempts at using coreset methods include the work of Boutsidis et al. [4], Zheng and Phillips [32], Phillips [18], which depend on a projection type compression, having similar spirit to the celebrated Johnson-Lindenstrauss lemma, a metric preserving projection result. So accuracy and running depend unfavorably on the desired statistical error rate. Kpotufe [14] propose an algorithm to reduce the query time of the NW estimator to O (log n ) , but the algorithm requires super-linear preprocessing time.

Other lines of work exploit the structure of kernels more directly, especially in the KRR literature. A slew of techniques from numerical analysis have been developed, including work on Nyström subsampling by El Alaoui and Mahoney [11], Avron et al. [1], Díaz et al. [7]. Camoriano et al. [5] and Rudi et al. [22] combine early stopping with Nyström subsampling. Though more distant from our approach, we also note the approach of Rahimi and Recht [21] using random features, Zhang et al. [31] using Divide-and-Conquer, and Tu et al. [28] using block coordinate descent.

Our contributions. In this work, we show how coreset methods can be used to speed up both training and inference in non-parametric regression for a large class of function classes/kernels. At the heart of these algorithms is a general procedure called kernel thinning [10, 9], which provides a worst-case bound on integration error (suited for problems in the original context of unsupervised learning and MCMC simulations). In Sec. 3, we introduce a meta-algorithm that recovers our two thinned non-parametric regression methods each based on NW and KRR. We introduce the kernel-thinned Nadaraya-Watson estimator (KT-NW) and the kernel-thinned kernel ridge regression estimator (KT-KRR).

We show that KT-NW requires O ( n log 3 n ) time during training and O ( √ n ) time at inference, while achieving a mean square error (MSE) rate of n -β β + d (Thm. 1)-a strict improvement over uniform subsampling of the original input points. We show that KT-KRR requires O ( n 3 / 2 ) time during training and and O ( √ n ) time during inference, while achieving an near-minimax optimal rate of m log n n when the kernel has finite dimension (Thm. 2). We show how our KT-KRR guarantees can also be extended to the infinite-dimension setting (Thm. 3). In Sec. 5, we apply our proposed methods to both simulated and real-world data. In line with our theory, KT-NW and KT-KRR outperform standard thinning baselines in terms of accuracy while retaining favorable runtimes.

2 Problem setup

We now formally describe the non-parametric regression problem. Let x 1 , . . . , x n be i.i.d. samples from the data distribution P (with density p ) over the domain X ⊂ R d and w 1 , . . . , w n be i.i.d. samples from N (0 , 1) . In the sequel, we use ∥ · ∥ to denote the Euclidean norm unless otherwise stated. Then define the response variables y 1 , . . . , y n by the follow data generating process:

where f ⋆ : X → Y ⊂ R is the regression function and v i ≜ σw i for some noise level σ > 0 . Our task is to build an estimate for f ⋆ given the n observed points, denoted by

Nadaraya-Watson (NW) estimator. A classical approach to estimate the function f ⋆ is kernel smoothing, where one estimates the function value at a point z using a weighted average of the observed outcomes. The weight for outcome y i depends on how close x i is to the point z ; let κ : R d → R denote this weighting function such that the weight for x i is proportional to κ ( ∥ x i -z ∥ /h ) for some bandwidth parameter h > 0 . Let k : R d × R → R denote a shift-invariant kernel defined as

Then this smoothing estimator, also known as Nadaraya-Watson (NW) estimator, can be expressed as

whenever the denominator in the above display is non-zero. In the case the denominator in (2) is zero, we can make a default choice, which for simplicity here we choose as zero. We refer to the estimator (2) as FULL-NW estimator hereafter. One can easily note that FULL-NW requires O ( n ) storage for the input points and O ( n ) kernel queries for inference at each point.

Kernel ridge regression (KRR) estimator. Another popular approach to estimate f ⋆ is that of non-parametric (regularized) least squares. The solution in this approach, often called as the kernel ridge regression (KRR), is obtained by solving a least squares objective where the fitted function is posited to lie in the RKHS H of a reproducing kernel k , and a regularization term is added to the objective to avoid overfitting. 1 Overall, the KRR estimate is the solution to the following regularized least-squares objective, where λ > 0 denotes a regularization hyperparameter:

Like NW, an advantage of KRR is the existence of a closed-form solution

Notably, the estimate ̂ f full ,λ , which we refer to as the FULL-KRR estimator, can also be seen as yet another instance of weighted average of the observed outcomes. Notably, NW estimator imposes that the weights across the points sum to 1 (and are also non-negative whenever k is), KRR allows for generic weights that need not be positive (even when k is) and need not sum to 1 . We note that naïvely solving ̂ f full ,λ requires O ( n 2 ) kernel evaluations to compute the kernel matrix, O ( n 3 ) to compute a matrix inverse, and O ( n ) kernel queries for inference at each point. One of our primary goals in this work is to tackle this high computational cost of FULL-KRR.

3 Speeding up non-parametric regression

We begin with a general approach to speed up regression by thinning the input datasets. While computationally superior, a generic approach suffers from a loss of statistical accuracy motivating the need for a strategic thinning approach. To that end, we briefly review kernel thinning and finally introduced our supervised kernel thinning approach.

3.1 Thinned regression estimators: Computational and statistical tradeoffs

Our generic approach comprises two main steps. First, we compress the input data by choosing a coreset S out ⊂ S in of size n out ≜ |S out | . Second, we apply our off-the-shelf non-parametric regression methods from Sec. 2 to the compressed data. By setting n out ≪ n , we can obtain notable speed-ups over the FULL versions of NW and KRR.

Before we introduce the thinned versions of NW and KRR, let us define the following notation. Given an input sequence S in and output sequence S out , define the empirical probability measures

Thinned NW estimator. The thinned NW estimator is the analog of Full-NW except that S in is replaced by S out in (2) so that the thinned-NW estimator is given by

whenever the denominator in the display is not zero; and 0 otherwise. When compared to the FULL-NW estimator, we can easily deduce the computational advantage of this estimator: more efficient O ( n out ) storage as well as the faster O ( n out ) computation for inference at each point.

Thinned KRR estimator. Similarly, we can define the thinned KRR estimator as

given some regularization parameter λ ′ > 0 . When compared to FULL-KRR, ̂ f S out ,λ ′ has training time O ( n 3 out ) and prediction time O ( n out ) .

Abaseline approach is standard thinning, whereby we let S out be an i.i.d. sample of n out = √ n points from S in . For NW, let us call the resulting ̂ f S out (7) the standard-thinned Nadaraya-Watson (ST-NW) estimator. When n out = √ n , ST-NW achieves an excess risk rate of O ( n -β 2 β + d ) compared to the FULL-NW rate of O ( n -2 β 2 β + d ) . For KRR, let us call the resulting ̂ f S out ,λ ′ (8) the standard-thinned KRR (ST-KRR) estimator. When n out = √ n , ST-KRR achieves an excess risk rate of O ( m n out ) compared to the FULL-KRR rate of O ( m n ) . Our goal is to provide good computational benefits without trading off statistical error. Moreover, we may be able to do better by leveraging the underlying geometry of the input points and summarize of the input distribution more succinctly than i.i.d. sampling.

3.2 Background on kernel thinning

A subroutine central to our approach is kernel thinning (KT) from Dwivedi and Mackey [10, Alg. 1]. We use a variant called KT-COMPRESS++ from Shetty et al. [23, Ex. 6] (see full details in App. A), which provides similar approximation quality as the original KT algorithm of Dwivedi and Mackey [10, Alg. 1], while reducing the runtime from O ( n 2 ) to O ( n log 3 n ) . 2 Given an input kernel k ALG and input points S in , KT-COMPRESS++ outputs a coreset S KT ⊂ S in with size n out ≜ √ n ≪ n . In this work, we leverage two guarantees of KT-COMPRESS++. Informally, S KT satisfies (with high probability):

where C 1 , C 2 > 0 are constants that depend on the properties of the input kernel k ALG and the chosen failure probability of KT-COMPRESS++, R in characterizes the radius of { x i } n i =1 , and N k ALG ( B 2 ( R in ) , 1 /n out ) denotes the kernel covering number of H ( k ALG ) over the ball B 2 ( R in ) ⊂ R d at a specified tolerance (see Sec. 4.2 for formal definitions).

At its highest level, KT provides good approximation of function averages. The bound (9) (formally stated in Lem. 1) controls the worst-case point-wise error, and is near-minimax optimal by Phillips and Tai [19, Thm. 3.1]. In the sequel, we leverage this type of result to derive generalization bounds for the kernel smoothing problem. The bound (10) (formally stated in Lem. 2) controls the integration error of functions in H ( k ALG ) and is near-minimax optimal by Tolstikhin et al. [26, Thm. 1, 6]. In the sequel, we leverage this type of result to derive generalization bounds for the KRR problem.

3.3 Supervised kernel thinning

We show how the approximation results from kernel thinning can be extended to the regression setting. We construct two meta-kernels, the Nadaraya-Watson meta-kernel k NW and the ridge-regression meta-kernel k RR , which take in a base kernel k (defined over X only) and return a new kernel (defined over X × Y ). When running KT, we set this new kernel as k ALG .

3.3.1 Kernel-thinned Nadaraya-Watson regression (KT-NW)

A tempting choice of kernel for KT-NW is the kernel k itself. That is, we can thin the input points using the kernel

This choice is sub-optimal since it ignores any information in the response variable y . For our supervised learning set-up, perhaps another intuitive choice would be to use KT with

where ( x, y ) denotes the concatenation of x and y . While this helps improve performance, there remains a better option as we illustrate next.

In fact, a simple but critical observation immediately reveals a superior choice of the kernel to be used in KT for NW estimator. We can directly observe that the NW estimator is a ratio of the averages of two functions:

over the empirical distribution P in (6). Recall that KT provides a good approximation of sample means of functions in an RKHS, so it suffices to specify a 'correct' choice of the RKHS (or equivalently the 'correct' choice of the reproducing kernel). We can verify that f denom lies in the RKHS associated with kernel k ( x 1 , x 2 ) and f numer lies in the RKHS associated with kernel k ( x 1 , x 2 ) · y 1 y 2 . This motivates our definition for the Nadaraya-Watson kernel:

since then we do have f denom , f numer ∈ H ( k NW ) . Intuitively, thinning with k RR should simultaneously provide good approximation of averages of f denom and f numer over P in (see the formal argument in Sec. 4.1). When S out = KT-COMPRESS++ ( S in , k NW , δ ) , we call the resulting solution to (7) the kernel-thinned Nadaraya-Watson (KT-NW) estimator, denoted by ̂ f KT .

As we show in Fig. 1(a), this theoretically principled choice does provide practical benefits in MSE performance across sample sizes.

2 10

2 14

Input sample size

n

2

3

2

1

2 1

Mean Squared Error

k

(

x

1

,

x

2

)

k

((

x

1

y

1

), (

x

2

y

2

))

k

NW((

x

1

,

y

1

), (

x

2

,

y

2

))

2 10

y

1

), (

1

,

y

1

x

), (

2 14

2

2 0

2

Mean Squared Error

Input sample size

k

((

(

x

1

,

x

1

k

x

RR((

n

x

2

x

2

k

2

)

y

2

2

))

))

,

y

3.3.2 Kernel-thinned kernel ridge regression (KT-KRR)

While with NW estimator, the closed-form expression was a ratio of averages, the KRR estimate (4) can not be expressed as an easy function of averages. However, notice that L S in in (3) is an average of the function ℓ f : X × Y → R defined as

Thus, there may be hope of deriving a KT-powered KRR estimator by thinning L S in with the appropriate kernel. Assuming f ∈ H ( k ) , we can verify that f 2 lies in the RKHS associated with kernel k 2 ( x 1 , x 2 ) and that -2 f ( x ) y lies in the RKHS associated with kernel k ( x 1 , x 2 ) · y 1 y 2 . We now define the ridge regression kernel by

and we can verify that f 2 ( x ) -2 f ( x ) y lies in the RKHS H ( k RR ) . 3 When S out ≜ KT-COMPRESS++ ( S in , k RR , δ ) , we call the resulting solution to (8) the kernel-thinned KRR (KT-KRR) estimator with regularization parameter λ ′ > 0 , denoted ̂ f KT ,λ ′ . We note that the kernel k RR also appears in [12, Lem. 4], except our subsequent analysis comes with generalization bounds for the KT-KRR estimator. Like for NW, in Fig. 1(b) we do a comparison for KRR-MSE across many kernel choices and conclude that the choice (14) is indeed a superior choice compared to the base kernel k and the concatenated kernel (12).

4 Main results

We derive generalization bounds of our two proposed estimators. In particular, we bound the mean squared error (MSE) defined by ∥ f -f ⋆ ∥ 2 2 = E X [ ( f ( X ) -f ⋆ ( X )) 2 ] . Our first assumption is that of a well-behaved density on the covariate space. This assumption mainly simplifies our analysis of Nadaraya-Watson and kernel ridge regression, but can in principle be relaxed.

Assumption 1 (Compact support) . Suppose that X ⊂ B 2 ( R in ) ⊂ R d for some R in > 0 and that the density p satisfies 0 < p min ≤ p ( x ) ≤ p max for all x ∈ X .

4.1 KT-NW

For the analysis of the NW estimator, we define function complexity in terms of Hölder smoothness following prior work [27].

Definition 1. For L > 0 and β ∈ (0 , 1] , a function f : X → R is ( β, L ) -Hölder if for all x 1 , x 2 ∈ X , | f ( x 1 ) -f ( x 2 ) | ≤ L ∥ x 1 -x 2 ∥ β .

Our next assumption is that on the kernel. Whereas typically the NW estimator does not require a reproducing kernel, our KT-NW estimator requires that k be reproducing to allow for valid analysis.

Assumption 2 (Shift-invariant kernel) . k is a reproducing kernel function (i.e., symmetric and positive semidefinite) defined by k ( x 1 , x 2 ) ≜ κ ( ∥ x 1 -x 2 ∥ /h ) , where h > 0 and κ : R → R is bounded by 1, L κ -Lipschitz, square-integrable, and satisfies:

Additionally, there must exist constants c 1 , c 2 > 0 such that

When k is a compact kernel, such as Wendland, Sinc, and B-spline, Assum. 2 is easily satisfied. In App. G, we prove that Gaussian and Matérn (with ν > d/ 2 + 1 ) kernels also satisfy Assum. 2. We now present our main result for the KT-NW estimator. See App. B for the proof.

Theorem 1 (KT-NW) . Suppose that Assum. 1 and 2 hold and that f ⋆ ∈ Σ( β, L f ) with β ∈ (0 , 1] . Then for any fixed δ ∈ (0 , 1] , the KT-NW estimator (7) with n out = √ n and bandwidth h = n -1 2 β +2 d satisfies

with probability at least 1 -δ , for some positive constant C that does not depend on n .

Tsybakov and Tsybakov [27], Belkin et al. [3] show that FULL-NW achieves a rate of O ( n -2 β 2 β + d ) , which is minimax optimal for the ( β, L ) -Hölder function class. Compared to the ST-NW rate of n -β 2 β + d , KT-NW achieves strictly better rates for all β > 0 and d > 0 , while retaining ST-NW's fast query time of O ( √ n ) . Note that our method KT-NW has a training time of O ( n log 3 n ) , which is not much more than simply storing the input points.

4.2 KT-KRR

We present our main result for KT-KRR using finite-rank kernels. This class of RKHS includes linear functions and polynomial function classes.

Theorem 2 (KT-KRR for finite-dimensional RKHS) . Assume f ⋆ ∈ H ( k ) , Assum. 1 is satisfied, and k has rank m ∈ N . Let ̂ f KT ,λ ′ denote the KT-KRR estimator with regularization parameter

λ ′ = O ( m log n out n ∧ n 2 out ) . Then with probability at least 1 -2 δ -2 e -∥ f ⋆ ∥ 2 k c 1 ( ∥ f ⋆ ∥ 2 k + σ 2 ) , the following holds:

for some constant C that does not depend on n or n out .

See App. C for the proof. Under the same assumptions, Wainwright [29, Ex. 13.19] showed that the Full-KRR estimator ̂ f full ,λ achieves the minimax optimal rate of O ( m/n ) in O ( n 3 ) runtime. When n out = √ n log c n , the KT-KRR error rates from Thm. 2 match this minimax rate in ˜ O ( n 3 / 2 ) time, a (near) quadratic improvement over the Full-KRR. On the other hand, standard thinning-KRR with similar-sized output achieves a quadratically poor MSE of order m √ n .

Our method and theory also extend to the setting of infinte-dimensional kernels. To formalize this, we first introduce the notion of kernel covering number.

Definition 2 (Covering number) . For a kernel k : Z × Z → R with B k ≜ { f ∈ H : ∥ f ∥ H ≤ 1 } , a set A ⊂ Z and ϵ > 0 , the covering number N k ( A , ϵ ) is the minimum cardinality of all sets C ⊂ B k satisfying B k ⊂ ⋃ h ∈C { g ∈ B k : sup x ∈A | h ( x ) -g ( x ) | ≤ ϵ } .

We consider two general classes of kernels.

Assumption 3. For some C d > 0 , all r > 0 and ϵ ∈ (0 , 1) , and B 2 ( r ) = { x ∈ R d : ∥ x ∥ 2 ≤ r } , a kernel k is

LOGGROWTH ( α, β ) when log N k ( B 2 ( r ) , ϵ ) ≤ C d log( e/ϵ ) α ( r +1) β with α, β > 0 and POLYGROWTH ( α, β ) when log N k ( B 2 ( r ) , ϵ ) ≤ C d (1 /ϵ ) α ( r +1) β with α < 2 .

We highlight that the definitions above cover several popular kernels: LOGGROWTH kernels include finite-rank kernels and analytic kernels, like Gaussian, inverse multiquadratic (IMQ), and sinc [9, Prop. 2], while POLYGROWTH kernels includes finitely-many continuously differentiable kernels, like Matérn and B-spline [9, Prop. 3]. For clarity, here we present our guarantee for LOGGROWTH kernels and defer the other case to App. E.

Theorem 3 (KT-KRR guarantee for infinite-dimensional RKHS) . Suppose Assum. 1 is satisfied and k is LOGGROWTH ( α, β ) (Assum. 3). Then ̂ f KT ,λ ′ with λ ′ = O (1 /n out ) satisfies the following

-∥ f ⋆ ∥ 2 k log α n 2

bound with probability at least 1 -2 δ -2 e c 1 ( ∥ f ⋆ ∥ k + σ 2 ) :

for some constant C that does not depend on n or n out .

See App. E for the proof. When n out = √ n , ST-KRR achieves an excess risk rate of n -1 / 2 log α n for k satisfying LOGGROWTH ( α, β ) . While KT-KRR does not achieve a strictly better excess risk rate bound over ST-KRR, we see that in practice, KT-KRR still obtains an empirical advantage. Obtaining a sharper error rate for the infinite-dimensional kernel setting is an exciting venue for future work.

5 Experimental results

We now present experiments on simulated and real-world data. On real-world data, we compare our KT-KRR estimator with several state-of-the-art KRR methods, including Nyström subsampling-based methods and KRR pre-conditioning methods. All our experiments were run on a machine with 8 CPU cores and 100 GB RAM. Our code can be found at https://github.com/ag2435/npr .

5.1 Simulation studies

We begin with some simulation experiments. For simplicity, let X = R and P = Unif [ - √ 3 , √ 3] so that Var[ X ] = 1 . We set

and follow (1) to generate ( y i ) n i =1 (see Fig. 2). We let the input sample size n vary between 2 8 , 2 10 , 2 12 , 2 14 and set the output coreset size to be n out = √ n in all cases. For NW, we use the Wendland (0) kernel defined by

For KRR, we use the Gaussian kernel defined by

1

0

1

5

0

5

We select the bandwidth h and regularization parameter λ ′ (for KRR) using grid search. Specifically, we use a held-out validation set of size 10 4 and run each parameter configuration 100 times to estimate the validation MSE since KT-KRR and ST-KRR are random.

Ablation study. In Fig. 1, we compare thinning with our proposed meta-kernel k ALG = k NW to thinning with the baseline meta-kernels (11) and (12). For our particular regression function (21), thinning with (11) outperforms thinning with (12). We hypothesize that the latter kernel is not robust to the scaling of the response variables. By inspecting (22), we see that ∥ ( x 1 , y 1 ) -( x 2 , y 2 ) ∥ 2 is heavily determined by the y i values when they are large compared to the values of x i -as is the case on the right side of Fig. 2 (when X > 0 ). Since P is a uniform distribution, thinning with (11) evenly subsamples points along the input domain X , even though accuractely learning the left side of Fig. 2 (when X < 0 ) is not needed for effective prediction since it is primarily noise. Validating our theory from Thm. 1, the best performance is obtained when thinning with k NW (13), which avoids evenly subsampling points along the input domain and correctly exploits the dependence between X and Y .

In Fig. 1, we perform a similar ablation for KRR. Again we observe that thinning with k ALG (( x 1 , y 1 ) , ( x 2 , y 2 )) = k ( x 1 , x 2 ) outperforms thinning with k ALG (( x 1 , y 1 ) , ( x 2 , y 2 )) = k (( x 1 , y 1 ) , ( x 2 , y 2 )) , while thinning with k ALG = k RR achieves the best performance.

Comparison with FULL, ST, RPCHOLESKY. In Fig. 3(a), we compare the MSE of KT-NW to FULL-NW, ST-NW (a.k.a 'Subsample'), and RPCHOLESKY-NW across four values of n . This last method uses the pivot points from RPCHOLESKY as the output coreset S out . At all n we evaluated, KT-NW achieves lower MSE than ST-NW and RPCHOLESKY-NW. FULL-NW achieves the lowest MSE across the board, but it suffers from significantly worse run times, especially at test time. Owing to its O ( n log 3 n ) runtime, KT-NW is significantly faster than RPCHOLESKY-NW for training and nearly matches ST-NW in both training and testing time. We hypothesize that RPCHOLESKY-while it provides a good low-rank approximation of the kernel matrix-is not designed to preserve averages.

In Fig. 3(b), we compare the MSE of KT-KRR to FULL-KRR, ST-KRR (a.k.a 'Subsample'), and the RPCHOLESKY-KRR method from Chen et al. [6, Sec. 4.2.2], which uses RPCHOLESKY to select

2 8

2 10

2 12

2 14

Input sample size

n

2

6

2

4

2

2

2 0

Mean Squared Error

n

0.73

n

0.25

n

0.36

n

0.67

0.0

0.1

0.2

0.3

0.4

Train time (s)

0

1

2

3

4

5

Test time (s)

n

=2 10

n

=2 14

n

=2 10

n

=2 14

Full

Subsample

RPCholesky

KT (Ours)

2 8

2 10

2 12

2 14

Input sample size

n

2

7

2

5

2

3

2

1

2 1

Mean Squared Error

n

1.24

n

0.30

n

0.37

0.001

0.03

1

32

Train time (s)

0.0

0.5

1.0

1.5

2.0

Test time (s)

n

=2 10

n

=2 14

Full

Subsample

RPCholesky

KT (Ours)

(b) Kernel ridge regression estimator with Gaussian kernel (23).

landmark points for the restricted KRR problem. We observe that KT-KRR achieves lower MSE than ST-KRR, but higher MSE than RPCHOLESKY-KRR and FULL-KRR. In Fig. 3(b), we also observe that KT-KRR is orders of magnitude faster than FULL-KRR across the board, with runtime comparable to ST-KRR and RPCHOLESKY-KRR in both training and testing.

5.2 Real data experiments

We now present experiments on real-world data using two popular datasets: the California Housing regression dataset from Pace and Barry [17] ( https://scikit-learn.org/1.5/datasets/ real_world.html#california-housing-dataset ; BSD-3-Clause license) and the SUSY binary classification dataset from Baldi et al. [2] ( https://archive.ics.uci.edu/dataset/ 279/susy ; CC-BY-4.0 license).

California Housing dataset ( d = 8 , N = 2 × 10 4 ). Tab. 1(a) compares the test MSE, train times, and test times. We normalize the input features by subtracting the mean and dividing by the standard deviation and use a 80-20 train-test split. For all methods, we use the Gaussian kernel (23) with bandwidth h = 10 . We use λ = λ ′ = 10 -3 for FULL-KRR, ST-KRR, and KT-KRR and λ = 10 -5 for RPCHOLESKY-KRR. On this dataset, KT-KRR lies between ST-KRR and RPCHOLESKY-KRR in terms of test MSE. When n out = √ n , RPCHOLESKY pivot selection takes O ( n 2 ) time by Chen et al. [6, Alg. 2], compared to KT-COMPRESS++, which compresses the input points in only O ( n log 3 n ) time. This difference in big-O runtime is reflected in our empirical results, where we see KT-KRR take 0.0153s versus 0.3237s for RPCHOLESKY-KRR.

SUSY dataset ( d = 18 , N = 5 × 10 6 ). Tab. 1(b) compares our proposed method KT-KRR (with h = 10 , λ ′ = 10 -1 ) to several large-scale kernel methods, namely RPCholesky preconditioning [7], FALKON [22], and Conjugate Gradient (all with h = 10 , λ = 10 -3 ) in terms of test classification error and training times. For the baseline methods, we use the Matlab implementation provided by Díaz et al. [7] ( https://github.com/ eepperly/Robust-randomized-preconditioning-for-kernel-ridge-regression ). In

(b) SUSY dataset.

our experiment, we use 4 × 10 6 points for training and the remaining 10 6 points for testing. As is common practice for classification tasks, we use the Laplace kernel defined by k ( x 1 , x 2 ) ≜ exp( -∥ x 1 -x 2 ∥ 2 /h ) . All parameters are chosen with cross-validation.

We observe that KT-KRR achieves test MSE between ST-KRR and RPCHOLESKY preconditioning with training time almost half that of RPCHOLESKY preconditioning. Notably, our Cython implementation of KT-COMPRESS++ thinned the four million training samples in only 1.7 seconds on a single CPU core-with further speed-ups to be gained from parallelizing on a GPU in the future.

6 Conclusions

In this work, we introduce a meta-algorithm for speeding up two estimators from non-parametric regression, namely the Nadaraya-Watson and Kernel Ridge Regression estimators. Our method inherits the favorable computational efficiency of the underlying Kernel Thinning algorithm and stands to benefit from further advancements in unsupervised learning compression methods.

The KT guarantees provided in this work apply only when f ⋆ ∈ H ( k ) for some base kernel k . In practice, choosing a good kernel k is indeed a challenge common to all prior work. Our framework is friendly to recent developments in kernel selection to handle this problem: Dwivedi and Mackey [9, Cor. 1] provide integration-error guarantees for KT when f ⋆ / ∈ H ( k ) . Moreover, there are recent results on finding the best kernel (e.g., for hypothesis testing [8, Sec. 4.2]). Radhakrishnan et al. [20] introduce the Recursive Feature Machine, which uses a parameterized kernel k M ( x 1 , x 2 ) ≜ exp( -( x 1 -x 2 ) ⊤ M ( x 1 -x 2 ) / (2 h 2 )) , and propose an efficient method to learn the matrix parameter M via the average gradient outer product estimator. An exciting future direction would be to combine these parameterized (or "learned") kernels with our proposed KT methods for non-parametric regression.

References

Haim Avron, Kenneth L Clarkson, and David P Woodruff. Faster kernel ridge regression using sketching and preconditioning. SIAM Journal on Matrix Analysis and Applications , 38(4): 1116-1138, 2017. (Cited on page 2.)

Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications , 5(1):4308, 2014. (Cited on page 9.)

Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1611-1619. PMLR, 2019. (Cited on pages 7 and 21.)

Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Near-optimal coresets for least-squares regression. IEEE transactions on information theory , 59(10):6880-6892, 2013. (Cited on page 2.)

Raffaello Camoriano, Tomás Angles, Alessandro Rudi, and Lorenzo Rosasco. Nytro: When subsampling meets early stopping. In Artificial Intelligence and Statistics , pages 1403-1411. PMLR, 2016. (Cited on page 2.)

Yifan Chen, Ethan N Epperly, Joel A Tropp, and Robert J Webber. Randomly pivoted cholesky: Practical approximation of a kernel matrix with few entry evaluations. arXiv preprint arXiv:2207.06503 , 2022. (Cited on pages 8 and 9.)

Mateo Díaz, Ethan N Epperly, Zachary Frangella, Joel A Tropp, and Robert J Webber. Robust, randomized preconditioning for kernel ridge regression. arXiv preprint arXiv:2304.12465 , 2023. (Cited on pages 2 and 9.)

Carles Domingo-Enrich, Raaz Dwivedi, and Lester Mackey. Compress then test: Powerful kernel testing in near-linear time. arXiv preprint arXiv:2301.05974 , 2023. (Cited on pages 1 and 10.)

Raaz Dwivedi and Lester Mackey. Generalized kernel thinning. In International Conference on Learning Representations , 2022. (Cited on pages 1, 2, 7, 10, 14, and 15.)

Raaz Dwivedi and Lester Mackey. Kernel thinning. Journal of Machine Learning Research , 25 (152):1-77, 2024. (Cited on pages 1, 2, 4, 13, 14, 18, 24, 32, 48, and 49.)

Ahmed El Alaoui and Michael W Mahoney. Fast randomized kernel methods with statistical guarantees. stat , 1050:2, 2014. (Cited on page 2.)

Steffen Grünewälder. Compressed empirical measures (in finite dimensions). arXiv preprint arXiv:2204.08847 , 2022. (Cited on pages 6 and 46.)

Ming-Yueh Huang and Shu Yang. Robust inference of conditional average treatment effects using dimension reduction. Statistica Sinica , 32(Suppl):547, 2022. (Cited on page 1.)

Samory Kpotufe. Fast, smooth and adaptive regression in metric spaces. Advances in Neural Information Processing Systems , 22, 2009. (Cited on page 2.)

Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of statistics , pages 1302-1338, 2000. (Cited on page 48.)

Lingxiao Li, Raaz Dwivedi, and Lester Mackey. Debiased distribution compression. arXiv preprint arXiv:2404.12290 , 2024. (Cited on pages 37 and 39.)

R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters , 33(3):291-297, 1997. (Cited on page 9.)

Jeff M Phillips. Coresets and sketches. In Handbook of discrete and computational geometry , pages 1269-1288. Chapman and Hall/CRC, 2017. (Cited on page 2.)

Jeff M Phillips and Wai Ming Tai. Improved coresets for kernel density estimates. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2718-2727. SIAM, 2018. (Cited on page 4.)

Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Feature learning in neural networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212.13881 , 2022. (Cited on pages 1 and 10.)

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007. (Cited on page 2.)

Alessandro Rudi, Luigi Carratino, and Lorenzo Rosasco. Falkon: An optimal large scale kernel method. Advances in neural information processing systems , 30, 2017. (Cited on pages 2 and 9.)

Abhishek Shetty, Raaz Dwivedi, and Lester Mackey. Distribution compression in near-linear time. In International Conference on Learning Representations , 2022. (Cited on pages 4, 13, 14, and 15.)

Rahul Singh, Liyuan Xu, and Arthur Gretton. Sequential kernel embedding for mediated and time-varying dose response curves. arXiv preprint arXiv:2111.03950 , 2021. (Cited on page 1.)

Ingo Steinwart and Simon Fischer. A closer look at covering number bounds for gaussian kernels. Journal of Complexity , 62:101513, 2021. (Cited on page 29.)

Ilya Tolstikhin, Bharath K Sriperumbudur, Krikamol Mu, et al. Minimax estimation of kernel mean embeddings. Journal of Machine Learning Research , 18(86):1-47, 2017. (Cited on page 4.)

Alexandre B Tsybakov and Alexandre B Tsybakov. Nonparametric estimators. Introduction to Nonparametric Estimation , pages 1-76, 2009. (Cited on pages 6 and 7.)

Stephen Tu, Rebecca Roelofs, Shivaram Venkataraman, and Benjamin Recht. Large scale kernel learning using block coordinate descent. arXiv preprint arXiv:1602.05310 , 2016. (Cited on page 2.)

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019. (Cited on pages 7, 21, 22, 23, 28, 32, 33, 34, 36, 37, 40, 41, 42, 44, 45, and 46.)

Holger Wendland. Scattered data approximation , volume 17. Cambridge university press, 2004. (Cited on page 48.)

Yuchen Zhang, John Duchi, and Martin Wainwright. Divide and conquer kernel ridge regression: Adistributed algorithm with minimax optimal rates. The Journal of Machine Learning Research , 16(1):3299-3340, 2015. (Cited on page 2.)

Yan Zheng and Jeff M Phillips. Coresets for kernel regression. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 645-654, 2017. (Cited on page 2.)

A Background on KT-COMPRESS++

This section details the KT-COMPRESS++ algorithm of Shetty et al. [23, Ex. 6]. In a nutshell, KT-COMPRESS (Alg. 2) takes as input a point sequence of size n , a compression level g , a (reproducing) kernel functions k ALG , and a failure probability δ . KT-COMPRESS++ first runs the KT-COMPRESS ( g ) algorithm of Shetty et al. [23, Ex. 4] to produce an intermediate coreset of size 2 g √ n . Next, the KT algorithm is run on the intermediate coreset to produce a final output of size √ n .

KT-COMPRESS proceeds by calling the recursive procedure COMPRESS, which uses KT with kernels k ALG as an intermediate halving algorithm. The KT algorithm itself consists of two subroutines: (1) KT-SPLIT (Alg. 3a), which splits a given input point sequence into two equal halves with small approximation error in the k ALG reproducing kernel Hilbert space and (2) KT-SWAP (Alg. 3b), which selects the best approximation amongst the KT-SPLIT coresets and a baseline coreset (that simply selects every other point in the sequence) and then iteratively refines the selected coreset by swapping out each element in turn for the non-coreset point that most improves MMD k ALG error. As in Shetty et al. [23, Rem. 3], we symmetrize the output of KT by returning either the KT coreset or its complement with equal probability.

Following Shetty et al. [23, Ex. 6], we always default to g = ⌈ log 2 log n +3 . 1 ⌉ so that KT-COMPRESS++ has an overall runtime of O ( n log 3 n ) . For the sake of simplicity, we drop any dependence on g in the main paper.

Algorithm 1: KT-COMPRESS++ - Identify coreset of size √ n

Input: point sequence S of size n , compression level g , kernel k , failure probability δ

return S C++

Algorithm 2: KT-COMPRESS - Identify coreset of size 2 g √ n

Input: point sequence S in of size n , compression level g , kernel k ALG , failure probability δ

function COMPRESS ( S , g , k ALG , δ ) :

Partition S into four arbitrary subsequences {S i } 4 i =1 each of size n/ 4

for

i

= 1

S

end

˜ S ← CONCATENATE ( ˜ S 1 , ˜ S 2 , ˜ S 3 , ˜ S 4 ) // combine the coresets to obtain a coreset of size 2 · 2 g · √ |S| return KT ( ˜ S , k ALG , | ˜ S| 2 δ ) // halve the coreset to size 2 g √ |S| via symmetrized kernel thinning

function KT ( S , k ALG , δ ) :

// Identify kernel thinning coreset containing ⌊|S| / 2 ⌋ input points

S KT ← KT-SWAP ( k ALG , KT-SPLIT ( k ALG , S , δ ))

return S KT with probability 1 2 and the complementary coreset S \ S KT otherwise

Define the event

i

˜

,

2

,

3

,

4

do

COMPRESS

(

S

i

,

g

,

k

ALG

, δ

)

// run COMPRESS recursively to return coresets of size

2

g

·

√

|S|

4

Dwivedi and Mackey [10, Thm. 1, Rmk. 4] show that

We restate [10, Thm. 4] in our notation:

Lemma 1 ( L ∞ guarantee for KT-COMPRESS++) . Let Z ⊂ R d and consider a reproducing kernel k ALG : Z × Z → R . Assume n/n out ∈ 2 N . Let S KT ≜ KT-COMPRESS++ ( S in , g , k ALG , δ ) and

Algorithm 3a: KT-SPLIT Divide points into candidate coresets of size ⌊ n/ 2 ⌋

Input: kernel k split, point sequence S in = ( x i ) n i =1 , failure probability δ

return ( S (1) , S (2) ) , candidate coresets of size ⌊ n/ 2 ⌋

Algorithm 3b: KT-SWAP - Identify and refine the best candidate coreset

Input: kernel k ALG , point sequence S in = ( x i ) n i =1 , candidate coresets ( S (1) , S (2) )

// Compare to baseline (e.g., standard thinning)

// Swap out each point in S KT for best alternative in S in while ensuring no point is repeated in S KT for i = 1 , . . . , ⌊ n/ 2 ⌋ do

S

KT

[

i

]

argmin

z

∈{S

KT

[

i

]

}∪

(

S

in

\S

KT

)

MMD

k

ALG

(

S

in

,

S

KT

with

S

KT

[

i

] =

z

)

end return S KT , refined coreset of size ⌊ n/ 2 ⌋

define P in ≜ 1 n ∑ z ∈S in δ z and Q out ≜ 1 n out ∑ z ∈S KT δ z . Then on event E KT ,δ , the following bound holds:

[ √

1

δ

∥

L

k

k

ALG

ALG

∥

∞

k

ALG

,n

for some universal positive constant c .

Proof. The claim follows by replacing k [10, Thm. 4] with k ALG , replacing the sub-Gaussian constant of KT with that of KT-COMPRESS++ in [23, Ex. 5], and replacing ∥ k ALG ∥ ∞ with ∥ k ALG ∥ ∞ , in ≜ sup z ∈S in k ALG ( z, z ) throughout.

We restate [9, Thm. 2] in our notation:

√

d

(

) ]

log

(

)

+

log

1 +

(

R

+

R

)

,

Lemma 2 (MMD guarantee for KT-COMPRESS++) . Let Z ⊂ R d and consider a reproducing kernel k ALG : Z×Z → R . Assume n/n out ∈ 2 N . Let S KT ≜ KT-COMPRESS++ ( k ALG , g )( S in ) and define P in ≜ 1 n ∑ z ∈S in δ z and Q out ≜ 1 n out ∑ z ∈S KT δ z . Then on event E KT ,δ , the following bound holds:

for some universal postive constant c .

Proof. The claim follows from replacing k in [9, Thm. 2] with k ALG and replacing the sub-Gaussian constant of KT with that of KT-COMPRESS++ in [23, Ex. 5].

B Proof of Thm. 1: KT-NW

Our primary goal is to bound E S in [( ̂ f KT ( x 0 ) -f ⋆ ( x 0 )) 2 ] for a fixed x 0 ∈ X . Once we have this bound, bounding ∥ ̂ f KT -f ⋆ ∥ 2 2 is as straightforward as integrating over x 0 ∈ X . Consider the following decomposition:

Define the random variables

Since X i are i.i.d. samples from P , it follows that η i are i.i.d. Bernoulli random variables with parameter

where c 0 > 0 depends only on d and κ (see Assum. 2). Denote the denominator terms in ̂ f and ̂ f KT by

respectively, and the numerator terms in ̂ f and ̂ f KT by

respectively.

We now consider two cases depending on the event E .

Case I: Suppose event E c is satisfied. It follows from (34) that ̂ p ( x 0 ) = 0 , in which case ̂ f ( x 0 ) = 0 . Since S out ⊂ S in , it necessarily follows that ̂ p KT ( x 0 ) = 0 and ̂ f KT ( x 0 ) = 0 . Thus, we can bound (30) and (31) by

Also define the event

for some positive constant C that does not depend on n . Note that these are low-order terms compared to the rest of the calculations, so we may ignore them in the final bound.

Case II: Otherwise, we may assume event E is satisfied. Let us first bound (30). On event E KT ,δ (24), we claim that

We defer the proof to App. B.1.

Letting X ≜ ( X 1 , . . . , X n ) and Y ≜ ( Y 1 , . . . , Y n ) denote the x and y components of S in , respectively, we can further decompose (31) by

where the first RHS term corresponds to the variance and the second RHS term corresponds to the bias. We claim that

for some constant C > 0 that does not depend on either n or h . We defer the proofs to App. B.2 and B.3. Combining (36) to (38), we have

Note that h d ≤ 1 , so the Cd log 2 n nh 2 d term dominates the C nh d term. Thus, the optimal choice of bandwidth h comes from balancing

Finally, we must verify our growth rate assumption on p in (36) is satisfied. Since β > 0 , we have

Plugging (40) into (39) yields the advertised bound (18).

B.1 Proof of claim (36)

We first provide a generic result for approximating the numerator and denominator terms defined in (34) and (35).

Lemma 3 (Simultaneous L ∞ bound using KT-COMPRESS++ with k NW ) . Suppose k satisfies Assum. 2. Given S in , the following bounds hold on the event E KT ,δ :

where c a , c p > 0 are constants that do not depend on d or n .

See App. B.1.1 for the proof. In the sequel, we will simply treat the log(1 /δ ) term as a constant, meaning the log n terms dominate in the expressions.

With this lemma in hand, let us prove the claim (36). Define the following events:

̸

On event E , consider the following decomposition:

Bounding (43). Note that almost surely, we have

Thus, we have

where (i) follows from subtracting np from both sides of the probability statement and (ii) follows from concentration of Bernoulli random variables (see App. B.2).

Bounding (44). Note that on event E KT ,δ ∩ C , we have

where step (i) follows from applying (41) and substituting ̂ p ≥ p 2 . Hence the events E KT ,δ and A∩C are mutually exclusive with probability 1 , thereby yielding

Bounding (45). On the event B ∩ C , we have ̂ f KT ( x 0 ) = ̂ A KT ( x 0 ) ̂ p KT ( x 0 ) and ̂ f ( x 0 ) = ̂ A ( x 0 ) ̂ p ( x 0 ) , which yields

We can invoke (41) and (42) to bound | ̂ p ( x 0 ) -̂ p KT ( x 0 ) | and | ̂ A KT ( x 0 ) -̂ A ( x 0 ) | respectively. Thus, we have

Finally, we have

where the last inequality follows from Assum. 2 for some universal positive constant c ′ .

for some positive constant C that does not depend on n , where step (i) uses the fact that ̂ A ( x 0 ) ̂ p ( x 0 ) ≤ Y max and step (ii) uses the lower bound on ̂ p KT ( x 0 ) from (46). Combining (43) to (45), we have

Note that the second term dominates so that we may drop the first term with slight change to the value of the constant C in the bound (36).

B.1.1 Proof of Lem. 3: Simultaneous L ∞ bound using KT-COMPRESS++ with k NW

We first decompose k NW as

and note that

This fact will be useful later for proving simultaneous L ∞ approximation guarantees for ̂ A and ̂ p . Given that k satisfies Assum. 2, we want to show that k NW defined by (13) satisfies the Lipschitz and tail decay properties, so that we may apply Lem. 1. Note that

We claim that kernel k NW satisfies

By [10, Rmk. 8], we have

where κ † is defined by (15). Applying (50) and (51), we have

Since Assum. 1 is satisfied, R is constant. Applying (54) to M k NW ( n, n out , d, δ, R ) as defined by (26), we have the bound

for some positive constant c ′′ . Substituting this into (25), we have

for some positive constants c 1 , c 2 . By definition,

Define k 1 and k 2 by (47) and (48), respectively, and note that k 1 ( · , z ) , k 2 ( · , z ) ∈ H ( k NW ) for all z ∈ Z . We want to show that

which would imply (41) and (42) (after simplifying all terms besides n , d , and δ ).

The first inequality follows from replacing all occurrences of the test function k NW ( · , ( x, y )) in the proof of Lem. 1 with the function k 1 ( · , x ) and noting that ⟨ k NW ( · , ( x i , y i )) , k 1 ( · , ( x, y )) ⟩ k NW = ⟨ k 1 ( · , x i ) , k 1 ( · , x ) ⟩ k 1 from the fact that H ( k NW ) = H ( k 1 ) ⊕H ( k 2 ) (49).

The second inequality follows from replacing all occurrences of the test function k NW ( · , ( x, y )) in the proof of Lem. 1 with the function k 2 ( · , x ) and noting that ⟨ k NW ( · , ( x i , y i )) , k 2 ( · , ( x, y )) ⟩ k NW = ⟨ k 2 ( · , ( x i , y i )) , k 2 ( · , ( x, y )) ⟩ k 2 , again from the fact that H ( k NW ) = H ( k 1 ) ⊕H ( k 2 ) (49).

Proof of claim (51). We leverage the fact that the Lipschitz constants defined by (27) satisfies the following additivity property. Letting Z = S in , we have

We proceed to bound L k 1 and L k 2 separately. Note that

Applying the definition (27) to L k 2 , we have

Putting together the pieces yields the claimed bound.

Proof of claim (52). We aim to show that R k NW ,n is not much larger than R k ,n . Note that k NW can be rewritten as

We define the sets

noting that by definition (28).

Suppose r ∈ Γ . Then for any ( x 1 , y 1 ) , ( x 2 , y 2 ) such that

it must follow that ∥ x 1 -x 2 ∥ 2 ≥ r 2 (since ∥ y 1 -y 2 ∥ 2 ≤ 4 Y 2 max by triangle inequality). Since r satisfies (55), it must follow that

meaning √ r 2 +4 Y 2 max ∈ Γ ⋆ , where recall Γ ⋆ is defined by (56). Thus, we have

as desired.

B.2 Proof of claim (37)

Suppose event E (32) is satisfied. Define the shorthand for the variance:

Conditioned on X 1 = x 1 , X 2 = x 2 , . . . , X n = x n , we have

where we have used the fact that E [ Y | X = · ] = f ⋆ ( · ) by assumption (1). Note that on event E , we have

where recall v 1 , . . . , v n are i.i.d. random variables with Var[ v i ] = σ 2 by (1). Taking the expectation w.r.t. X 1 , . . . , X n and leveraging symmetry, we have

σ 2 X can be bounded by

where step (i) follows from the fact that k 2 ( X 1 ,x 0 ) ( ∑ n i =1 k ( X i ,x 0 ) ) 2 ≤ 1 . Using Bernstein's inequality [29, Prop. 2.14], the first term can be bounded by

for some universal positive constant c . Applying the fact that p is bounded by Assum. 1 and κ is square-integrable by Assum. 2, we can bound the second term by

for some positive constant c 2 that does not depend on either h or n . Substituting these expressions into (58) yields a bound on E X [ σ 2 ( x 0 ; X ) I [ E ] ] as desired.

B.3 Proof of claim (38)

Define the following shorthand for the bias:

We state a more detailed version of the claim.

Lemma 4 (Bias of Nadaraya-Watson) . Suppose Assum. 1 and 2 are satisfied and the event E (32) holds. If f ⋆ ∈ Σ( β, L f ) for β ∈ (0 , 1] , L f > 0 , then with high probability, the following statements hold true uniformly for all x 0 ∈ X :

(I.a) If k is compactly supported, then b ( x 0 ; X ) ≤ L f h β .

(I.b) Otherwise, if k is non-compactly supported, we have b ( x 0 ; X ) ≤ c · L f h β log n for some positive constant c that does not depend on n .

For completeness, we first state the proof of claim (I.a) from Belkin et al. [3, Lem. 2]. On the event E (32), we have

where step (i) follows from our assumption that f ⋆ ∈ Σ( β, L f ) and step (ii) follows from our assumption that k is compactly supported, so k ( x, x 0 ) = 0 whenever ∥ x -x 0 ∥ > h . We now proceed to proving claim (I.b).

Step I: Decomposing the bias. For any x 0 ∈ X , define the random variables

where

when p X ( · ) has compact support under Assum. 1 and h is defined by (40). We call these random variables the kernel empirical means . We can directly verify that

With these definitions in hand, we may rewrite b ( x 0 ; X ) in terms of A 0 ( x 0 ) , A 1 ( x 0 ) , . . . , A T ( x 0 ) as follows:

where step (i) follows from the fact that f ⋆ ( X i ) -f ⋆ ( x 0 ) ≤ L f ∥ X i -x 0 ∥ β by Def. 1 and step (ii) follows from the fact that ∥ X i -x 0 ∥ ≤ 2 j h under the indicator functions. Applying (59) to (62) and (60) to (63), we have

For any z ∈ X , define the following shorthands:

We can directly verify that

Under Assum. 1, a i ( z ) can be computed directly from the decay properties of k as follows:

Remark 1. Note that for fixed x 0 ∈ X , A i ( x 0 ) , Hoeffding's inequality implies that A i ( x 0 ) concentrates around a i ( x 0 ) for all i = 0 , 1 , . . . , T with high probability. Assuming that k decays sufficiently rapidly (so that a j ( x 0 )2 jβ = O ( a 0 ( x 0 )) for all j ∈ [ T ] ), we have

It remains to show that this bound holds uniformly over all x 0 ∈ X with high probability.

Step II: Uniform concentration of empirical kernel means. Fix ϵ ∈ (0 , h 2 ] and let C 0 be a ϵ -cover of X w.r.t. the Euclidean norm (see [29, Def. 5.1] for the detailed definition). We will optimize the choice of ϵ at the end of the proof. By [29, Lem. 5.7], we have

We now introduce a slightly modified version of (65) and (66) defined as

respectively. Using the fact that ϵ ≤ h/ 2 , Assum. 1 and 2, and a change of variables, we have

where ω d -1 ≜ 2 π d/ 2 / Γ( d/ 2) denotes the surface area of the unit sphere in R d . Moreover, we consider the following concentration events:

where ∆ 0 , ∆ 1 , . . . , ∆ T are positive scalars to be chosen later.

For any scalars 0 < a < b and z ∈ X , further denote the empirical mass contained within the ball or annulus centered at point z by

We now introduce a lemma that bounds the empirical kernel mean restricted to each ball and annuli:

Lemma 5 (Uniform concentration of empirical kernel means) . Suppose event ⋂ z ∈C 0 ⋂ T i =0 A ϵ i ( z ) is satisfied. Then for every x 0 ∈ X , there exists x ′ 0 ∈ C 0 such that ∥ x 0 -x ′ 0 ∥ ≤ ϵ and the following bounds hold simultaneously:

Here, L k and κ ( · ) are properties of k from Assum. 2.

See App. B.3.1 for the proof. This result suggests that we can control the error between the empirical kernel mean, A i ( x 0 ) , and its (shifted) population counterpart, a ϵ j ( x ′ 0 ) , by choosing ϵ to be small and leveraging the decay property of κ . To bound the Mass x ′ 0 ( h -ϵ ) and Mass x ′ 0 (2 j -1 h -ϵ, 2 j h + ϵ ) terms, we define the following events:

where t 0 , t 1 , . . . , t T are positive scalars to be chosen later. For any scalars 0 < a < b , we may apply Assum. 1 to obtain

where Vol( r ) ≜ π d / Γ( d/ 2 + 1) r d denotes the volume of the Euclidean ball B 2 ( r ) ⊂ R d .

Step III: Putting things together. For any z ∈ C 0 , note that

and therefore these random variables are sub-Gaussian with parameter σ = 1 2 ∥ k ∥ ∞ . Thus, we may bound (68) and (69) using Hoeffding's bound [29, Prop. 2.5], obtaining

Again by Hoeffding's bound [29, Prop. 2.5], we have

Next, it remains to choose the values of ϵ , ∆ 0 , . . . , ∆ T and t 0 , . . . , t T . Consider

Combining (70), (72), (74), (76), and (78), we have on the event E 0 ≜ ⋂ z ∈C 0 ( A ϵ 0 ( z ) ∩ M 0 ( z )) the following bound:

where step (i) follows from setting

Similarly, for each j = 1 , 2 , . . . , T , we may combine (71), (73), (74), (77), and (79) to obtain a bound on A j ( x 0 ) . Hence, on the E j ≜ ⋂ z ∈C 0 ( A ϵ j ( z ) ∩ M j ( z )) , we have

where step (i) employs the fact that ϵ ≤ h/ 2 . Note that for j = 1 , 2 , . . . , T :

where the last inequality comes from the fact that

by Assum. 2. Substituting this bound into (64), we have

as desired. Finally, it remains to control the probability that this bound on b ( x 0 ; X ) holds. Applying union-bound, we have

for some universal positive constants c 1 , c 2 that do not depend on n . Note that by (80), we have (1 + 2 R ϵ ) d = O ((1 + 1 h ) d ) . Setting h = cn -1 2 β +2 d as in (40) and recalling that T = O (log n ) , we have P ( E c 0 ∪ . . . ∪ E c T ) ≪ 1 for sufficiently large n . This completes the proof of Lem. 4.

B.3.1 Proof of Lem. 5: Uniform concentration of empirical kernel means

Fix any x 0 ∈ X . By definition of C 0 , there exists x ′ 0 ∈ C 0 such that ∥ x 0 -x ′ 0 ∥ ≤ ϵ . Consider any x ∈ B 2 ( h -ϵ ; x ′ 0 ) 4 . By triangle inequality, ∥ x -x 0 ∥ ≤ ∥ x -x ′ 0 ∥ + ∥ x ′ 0 -x 0 ∥ ≤ ( h -ϵ ) + ϵ . Therefore, we have

We now introduce the following lemma, which is a finite-sample version of [10, Lem. 8]:

Lemma 6 ( L ∞ bound on L 2 kernel error (finite sample)) . Consider any bounded kernel k ∈ L 2 , ∞ , points x 0 , x ′ 0 ∈ X , and function g ∈ L 2 ( P n ) . For any r, a, b ≥ 0 with a + b = 1 , points x 1 , . . . , x n ∈ X , and set A ⊂ X , we have

where

See App. B.3.2 for the proof. We now use this lemma to prove each of the claims in Lem. 5.

Proof of claim (70). We consider the following basic decomposition:

Note that the term (84) can be directly bounded using the definition of (68). Next, we bound the term (85). Define

so that ∥ g ∥ A,L 2 ( P n ) = Mass 1 2 x ′ 0 ( h -ϵ ) . By property (82), we have

Observe that

To upper bound (87), we now apply Lem. 6 with r = h , a = 1 2 , and b = 1 2 to obtain

where the last inequality uses the fact that

from the Lipschitz property of k (see Assum. 2) and the fact that

Substituting the bounds (68), (87), and (88) into the decomposition (83) yields the desired claim.

Proof of claim (71). We consider the basic decomposition:

Note that the term (90) can be directly bounded using the definition of (69). Next, we bound the term (91). By triangle inequality, we have

which implies that B 2 (2 j h ; x 0 ) ⊂ B 2 (2 j h + ϵ ; x ′ 0 ) . Moreover, we can verify that B 2 (2 j -1 h -ϵ ; x ′ 0 ) ⊂ B 2 (2 j -1 h ; x 0 ) . Hence,

Now define

so that ∥ g ∥ A,L 2 ( P n ) = Mass 1 2 x ′ (2 j -1 h -ϵ, 2 j h + ϵ ) . Hence, (91) can be bounded by

where the last inequality follows from applying Lem. 6 with r = 2(2 j h + ϵ ) and a = b = 1 2 . By triangle inequality, we have

where step (i) follows from the definition of A (93). Since κ is monotonically decreasing by Assum. 2, we have κ ( ∥ X i -x 0 ∥ h ) ≤ κ ( 2 j -1 h -2 ϵ h ) . Thus, we have

where step (i) again uses the fact that κ is monotonically decreasing. Moreover, note that

Substituting the bounds (69), (94), and (95) into the decomposition (89) yields the desired claim.

B.3.2 Proof of Lem. 6: L ∞ bound on L 2 kernel error (finite sample)

Define the restrictions

so that k ( z, x 0 ) = k r ( z, x 0 ) + k ( c ) r ( z, x 0 ) and k ( z, x ′ 0 ) = k r ( z, x ′ 0 ) + k ( c ) r ( z, x ′ 0 ) . We now apply the triangle inequality and Hölder's inequality to obtain

where ∥ · ∥ L 1 ( P n ) is defined by ∥ f ∥ L 1 ( P n ) ≜ 1 n ∑ n i =1 | f ( x i ) | . To bound the term (96), we apply Cauchy-Schwarz to g r ∈ L 1 ( P n ) ∩ L 2 ( P n ) to obtain

Next, we bound the term (97). For any x ⋆ ∈ { x 0 , x ′ 0 } and x i satisfying ∥ x i -x ′ 0 ∥ ≥ r and scalars a, b ∈ [0 , 1] such that a + b = 1 , either ∥ x ⋆ -x i ∥ ≥ ar or ∥ x ⋆ -x ′ 0 ∥ > br . Hence,

where we define δ x 0 ( z ) ≜ I [ z = x 0 ] and δ x ′ 0 ( z ) ≜ I [ z = x ′ 0 ] .

Bounding (98). Exchanging the order of the summations and applying Cauchy-Schwarz, we have

Bounding (99). Rearranging the terms and applying Cauchy-Schwarz, we have

C Proof of Thm. 2: KT-KRR for finite-dimensional RKHS

We rely on the localized Gaussian/Rademacher analysis of KRR from prior work [29]. Define the Gaussian critical radius ε n > 0 to be the smallest positive solution to the inequality

B H (3) is the ∥·∥ k -ball of radius 3 and w i i.i.d. ∼ N (0 , 1) .

Assumption 4. Assume that ∥ f ⋆ ∥ k ∈ B k ( R ) and ̂ f KT ,λ ′ ∈ B k ( c † R ) , for some constant c † > 0 .

Note that for any g ∈ B k ( c † R ) , we have

where step (i) follows from Cauchy-Schwarz. Thus, the function class B k ( c † R ) is B -uniformly bounded. Now define the Rademacher critical radius δ n > 0 to be the smallest positive solution to the inequality

and ν i = ± 1 each with probability 1 / 2 .

Finally, we use the following shorthand to control the KT approximation error term,

and W k is an inflaction factor defined in (29) that scales with the covering number N k (see Def. 2). With these definitions in place, we are ready to state a detailed version of Thm. 2:

Theorem 4 (KT-KRR for finite-dimensional RKHS, detailed) . Suppose the kernel operator associated with k and P has eigenvalues µ 1 ≥ . . . ≥ µ m > 0 (by Mercer's theorem). Define C m ≜ 1 /µ m . Let ε n and δ n denote the solutions to (101) and (100) , respectively. Further assume 5

Let ̂ f KT ,λ ′ denote the KT-KRR estimator with regularization parameter

Then with probability at least 1 -2 δ -2 e -nδ 2 n c 1 ( b 2 + σ 2 ) , we have

where recall δ is the success probability of KT-COMPRESS++ (24) .

See App. D for the proof. We set λ ′ = 2 ξ 2 n , so that (105) becomes

It remains to bound the quantities ε n (100), δ n (101), and η n, k (102). We claim that

Thus, we have

for some universal positive constants c 0 , c 1 , c 2 . Now set

for some universal positive constant c ′ . Substituting this into (106) leads to the advertised bound (19).

Proof of claim (107). For finite rank kernels, ˆ µ j = 0 for j > m . Thus, we have √ 2 n √ ∑ n j =1 min { ε 2 , ˆ µ j } = √ 2 n √ mε 2 . From the critical radius condition (100), we want √ 2 n √ mε 2 ≤ R 4 σ ε 2 , so we may set ε n ≃ σ R √ m n .

Proof of claim (108). By similar logic as above, we have √ 2 n √ ∑ n j =1 min { δ 2 , µ j } = √ 2 n √ mδ 2 . From the critical radius condition (101), we want √ 2 n √ mδ 2 ≤ 1 b δ 2 , so we may set δ n ≃ b √ 2 √ m n .

Proof of claim (109). Consider the linear operator T : H → R m that maps a function to the coefficients in the vector space spanned by { ϕ i } m i =1 . Note that

Since the image of T has dimension m , we have rank( T ) ≤ m . Moreover, ∥ k ∥ ∞ ≤ µ 1 · R 2 in . Now we can invoke [25, Eq. 14] with ϵ = a /n out to obtain

Taking the log on both sides and substituting this bound into (102), we have

for some positive constant c that doesn't depend on m,n out , δ .

D Proof of Thm. 4: KT-KRR for finite-dimensional RKHS, detailed

We rescale our observation model (1) by ∥ f ⋆ ∥ k , so that the noise variance is ( σ/ ∥ f ⋆ ∥ k ) 2 and our new regression function satisfies ∥ f ⋆ ∥ k = 1 . Our final prediction error should then be multiplied by ∥ f ⋆ ∥ 2 k to recover a result for the original problem. For simplicity, denote

For notational convenience, define an event

and our goal is to show that E occurs with high-probability in terms of P , the probability regarding all the randomness. For that end, we introduce several events that are used throughout,

where δ n is defined in (101) and H is the RKHS generated by k hence star-shaped. Further, we introduce two technical events A KT ( u ) , B KT defined in (123) and (136) respectively, which are proven to occur with small probability, and define a shorthand

Equipped with these shorthands, observe the following inequality,

where the second inequality is because E c lower ⊆ E c lower ∩ E due to the assumption λ ′ ≥ 2 ξ 2 n ≥ 2 δ 2 n .

If we are able to show the set inclusion {E good ∩E lower } ⊆ {E ∩ E lower } and that P ( E c good ) is small, we are able refine (111) to the following

where the last quantity 1 -P ( E c good ) would be large.

To complete this proof strategy, we claim the set inclusion

to hold and prove it in App. D.1 and further claim

which verify in App. D.2.

Putting the pieces together, claims (112) and (113) collectively implies

as desired.

D.1 Proof of claim (112)

There are several intermediary steps we take to show the set inclusion of interest (112). We introduce the shorthand

By invoking Propositions and basic inequalities to come, we successively show the following chain of set inclusions

Note that step (117) is achieved trivially by dropping E good. Further note that (115) is the crucial intermediary step after which we may apply uniform concentration across n independent samples. Proof of (115) leverages on the Proposition to come (Prop. 1) that allows ∥ · ∥ n and ∥ · ∥ n out to be exchangeable for finite rank kernels.

Recovering step (114) Since ̂ f KT ,λ ′ and f ⋆ are optimal and feasible, respectively for the central optimization problem of interest

we have the basic inequality

With some algebra , may refine (118) to

where ̂ ∆ KT = ̂ f KT ,λ ′ -f ⋆ .Suppose that ∥ ̂ ∆ KT ∥ n out < ξ n , then we trivially recover (114) by adding λ ′ > 0 . Thus, we assume that ∥ ̂ ∆ KT ∥ n out ≥ ξ n .

Under the assumption ∥ ̂ ∆ KT ∥ n out ≥ ξ n , which is without loss of generality, we utilize the basic inequality (119) and control its stochastic component

with a careful case work to follow, which is technical by nature.

Case where ∥ ̂ f KT ,λ ′ ∥ k ≤ 2 : Under such case, we introduce a technical event

for any star-shaped function class F ⊂ H . Since ∥ f ⋆ ∥ k = 1 , triangle inequality implies ∥ ̂ ∆ KT ∥ k ≤ ∥ ̂ f KT ,λ ′ ∥ k + ∥ f ⋆ ∥ k ≤ 3 . Moreover, on the event E lower ( ⊆ E good), we have ∥ ̂ ∆ KT ∥ 2 > δ n . Thus, we may apply ̂ ∆ KT to the event A c KT ( ξ n ) with F = B H (3) (i.e., the H -ball of radius 3) to attain

Upper bounding the stochastic component of the basic inequality (119) by (120) and dropping the -∥ ̂ f KT ,λ ′ ∥ 2 k term in (119), we have

As a last step under the case ∥ ̂ f KT ,λ ′ ∥ k ≤ 2 , apply the quadratic formula (specifically, if a, b ≥ 0 and x 2 -ax -b ≤ 0 , then x ≤ a 2 + b ) to obtain

Case where ∥ ̂ f KT ,λ ′ ∥ k > 2 : Under such case, by assumption we have ∥ ̂ f KT ,λ ′ ∥ k > 2 > 1 ≥ ∥ f ⋆ ∥ k . Thus, we may derive the following

which further implies the following inequality

Further writing ̂ f KT ,λ ′ = f ⋆ + ̂ ∆ KT and noting that ∥ ̂ ∆ KT ∥ k -∥ f ⋆ ∥ k ≤ ∥ ̂ f KT ,λ ′ ∥ k holds through triangle inequality, we may further refine (121) as

so that the basic inequality in (119) reduces to

We again introduce a technical event that controls the stochastic component of (122), which is

for a star-shaped function class F ⊂ H .

By triangle inequality, we have ∥ ̂ ∆ KT ∥ k ≥ ∥ ̂ f KT ,λ ′ ∥ k -∥ f ⋆ ∥ k > 1 , and on event E lower ( ⊂ E good), we have ∥ ̂ ∆ KT ∥ 2 > δ n . Thus, we may apply g = ̂ ∆ KT to the event B c KT , and the resulting refined basic inequality is

where the second inequality is due to the assumption that λ ′ ≥ 2 ξ 2 n . We apply the quadratic formula (specifically, if a, b ≥ 0 and x 2 -ax -b ≤ 0 , then x ≤ a 2 + b ) to obtain

Putting the pieces together, we have shown

which is sufficient to recover (114).

Recovering step (115) We now upgrade events

by exploiting the events E conc ∩ E KT ,δ (subset of E good) that were otherwise not used when recovering (114). For this end, the following result is a crucial ingredient, which shows that ∥·∥ n out and ∥·∥ n are essentially exchangeable with high-probability,

Proposition 1 (Multiplicative guarantee for KT-COMPRESS++ with k RR ) . Let C m ≜ 1 /µ m and suppose δ n satisfies (104) . Then on event E KT ,δ ∩ E conc, where E KT ,δ and E conc are defined in (24) and (110) respectively, we have

uniformly over all g ∈ H such that ∥ g ∥ 2 > δ n .

See App. D.3 for the proof.

An immediate consequence of Prop. 1 is that

on the event E KT ,δ ∩ E conc ∩ E lower , , which is sufficient to recover (115).

Recovering step (116) Our last step is to show

Such result can be immediately shown on the event E conc by observing that ̂ ∆ KT ∈ H , by our assumption that f ⋆ ∈ H and by the definition

D.2 Proof of claim (113)

It suffices to show the appropriate bounds for the following four probability terms

We know from [10] that P ( E c KT ,δ |S in ) ≤ δ and then we may apply [29, Thm. 14.1] to obtain a high probability statement,

Now we present two Lemmas that bound the P ( · | S in ) probability of events A KT ( ξ n ) and B KT ,

Fix the shorthand

Lemma 7 (Controlling bad event when ∥ ̂ f KT ,λ ′ ∥ k ≤ 2 ) . Suppose u ≥ ξ n . Then for some constant c > 0 ,

where ˜ σ = σ/ ∥ f ⋆ ∥ k .

See App. D.4 for the proof. Note that by plugging in ξ n into (126) results in a probability that depends on S in (as ξ n depends on S in ). By invoking the definition of ξ n , we may further refine the probability bound of A KT ( ξ n ) by

Lemma 8 (Controlling bad event when ∥ ̂ f KT ,λ ′ ∥ k > 2 ) . For some constants c, c ′ > 0 ,

where ˜ σ = σ/ ∥ f ⋆ ∥ k .

See App. D.5 for the proof. It is notable that ξ n in the probability bound of (128) contains a term ε n defined in (100) that is a function of S in . Invoking the definition of ξ n , we observe the probability upper bound (128) can be refined to

which does not depend on S in .

Putting the pieces together, we have the following probability bound for some constants c, c ′ > 0 ,

thereby implying P ( E c good ) ≤ c ′′ { δ + e -c ′ nδ 2 n / ( B 2 H ∧ ˜ σ 2 ) } for some constant c ′′ .

D.3 Proof of Prop. 1: Multiplicative guarantee for KT-COMPRESS++ with k RR

Fix g ∈ H . Denote ⟨ g, h ⟩ = ∫ g ( x ) h ( x ) dx as the inner product in the L 2 sense. By Mercer's theorem [29, Cor. 12.26], the k -norm of g has a basis expansion ∥ g ∥ 2 k = ∑ m i =1 ⟨ g, ϕ i ⟩ 2 /λ i so that

The assumption ∥ g ∥ 2 ≥ δ n implies that on the event E conc (110), we have

Moreover, g must be a non-zero function. Note that g 2 ∈ H ( k RR ) (see App. F.3). Thus, we may apply Lem. 14 to f 1 = f 2 = g and a = 1 , b = 0 to obtain

The LHS can be expanded as

Thus, we may rearrange (132) and combine with (130) to obtain

On event E conc, we have

Thus, we have

Using (135) to refine (133), we have on event E KT ,δ ∩ E conc :

With some algebra, this implies with probability at least 1 -δ -exp( -c ′ nδ 2 n /B 2 F ) :

uniformly over all non-zero g ∈ H such that ∥ g ∥ 2 > δ n .

D.4 Proof of Lem. 7: Controlling bad event when ∥ ̂ f KT ,λ ′ ∥ k ≤ 2

Recall E KT ,δ and E conc defined by (24) and (110). Also recall that E KT ,δ ∩ E conc combined with the assumption ∥ g ∥ 2 ≥ δ n invokes the event (124). Our aim is to show

so that we have a probability bound

The first RHS term can be bounded by δ (see (24)). The second RHS term can bounded by (125). The third term can be bounded by

where (i) follows from our assumption that u ≥ ε n and (ii) follows from applying generic concentration bounds on Z n ( u ) (see [29, Thm. 2.26, Eq. 13.66]). Putting together the pieces yields our desired probability bound (126).

Proof of claim (136). Consider the event A KT ( u ) ∩E KT ,δ ∩E conc. The norm equivalence established on the event E KT ,δ ∩ E conc in Prop. 1 is an important ingredient throughout.

Let g ∈ H be the function that satisfies three conditions: ∥ g ∥ 2 ≥ δ n , ∥ g ∥ n out ≥ u , and

Define the normalized function

so that it satisfies ∥ ˜ g ∥ n out = u and also

By triangle inequality, the LHS of (137) can be further upper bounded by

Recall the chosen g satisies ∥ g ∥ n out ≥ u . Observe that

so we may apply Lem. 14 with f 1 = f ⋆ , f 2 = g and a = -1 , b = 1 . Thus, on the event E KT ,δ ∩E conc , we have

Thus, we may rearrange (138) and combine with (137) and (139) to obtain

Note that

We tackle each term in turn. First, ∥ g ∥ k ∥ g ∥ 2 (130) ≤ √ C m . Since we assume ∥ g ∥ 2 ≥ δ n , we have ∥ g ∥ 2 ∥ g ∥ n ≤ δ n / 2+ ∥ g ∥ n ∥ g ∥ n (131) ≤ 2 on event E conc; and ∥ g ∥ n ∥ g ∥ n out (144) ≤ 2 on event E conc ∩ E KT ,δ . Taken together,

As u ≥ ξ n ≥ 4 √ C m ( ∥ f ⋆ ∥ k + 1) η n, k by assumption, we have therefore found ˜ g with norm ∥ ˜ g ∥ n out = u satisfying

We may further show that

where the last inequality follows from the fact that ∥ g ∥ 2 ≥ δ n and by applying (144). So we observe

D.5 Proof of Lem. 8: Controlling bad event when ∥ ̂ f KT ,λ ′ ∥ k > 2

Our aim is to show for any g ∈ ∂ H with ∥ g ∥ k ≥ 1 ,

Note that it is sufficient to prove our aim for g ∈ ∂ H with ∥ g ∥ k = 1 -by proving only for g with ∥ g ∥ k = 1 , then for any h ∈ ∂ H with ∥ h ∥ k ≥ 1 , we may plug g = h/ ∥ h ∥ k into

to recover the aim of interest. So without loss of generality, we show (141) for all g such that g ∈ ∂ H and ∥ g ∥ k = 1 .

Let B KT denote the event where (141) is violated, i.e. there exists g ∈ ∂ H with ∥ g ∥ k = 1 so that

We prove the following set inclusion,

where we know the RHS event of (143) has probability bounded by ce -nξ 2 n / ( c ′ ˜ σ 2 ) which is proven in [29, Lem. 13.23]. So the set inclusion (143) implies a bound over the event B KT ,

where P ( E c KT ,δ ) ≤ δ by (24) and P ( E c conc ) by (125).

Choose g so that ∥ g ∥ k = 1 and (142) holds. Condition ∥ g ∥ k = 1 as well as the condition (130) resulting from a finite rank kernel k implies δ n ≤ 1 ≤ ∥ g ∥ k ≤ √ C m ∥ g ∥ 2 . Invoke Prop. 1 for the choice of g that satisfies ∥ g ∥ 2 ≥ δ n / √ C m ≥ δ n , so that on the event E KT ,δ ∩ E conc, we have the following norm equivalence,

Then we have the following chain of inequalities, which holds on event E conc ∩ E KT ,δ

where step (i) follows from triangle inequality and step (ii) follows from our assumption (142) to bound the first term and our approximation guarantee (139) to bound the second term. By definition of ξ n , we have

Using this to refine (145), we have

which directly implies the inclusion (143) as desired.

E Proof of Thm. 3: KT-KRR guarantee for infinite-dimensional RKHS

We state a more detailed version of the theorem:

Theorem 5 (KT-KRR guarantee for infinite-dimensional RKHS, detailed) . Assume f ⋆ ∈ H ( k ) and Assum. 1 is satisfied. If k is LOGGROWTH ( α, β ) , then for some constant c (depending on d, α, β ), ̂ f KT ,λ ′ with λ ′ = O (1 /n out ) satisfies

with probability at least 1 -2 δ -2 e -nδ 2 n c 1 ( ∥ f ⋆ ∥ 2 k + σ 2 ) .

If k is POLYGROWTH ( α, β ) with α ∈ (0 , 2) , then for some constant c (depending on d, α, β ), ̂ f KT ,λ ′ with λ = O ( n -2 -α 2 out ) satisfies

with probability at least 1 -2 δ -2 e -nδ 2 n c 1 ( ∥ f ⋆ ∥ 2 k + σ 2 ) .

E.1 Generic KT-KRR guarantee

We state a generic result for infinite-dimensional RKHS that only depends on the Rademacher and Gaussian critical radii as well as the KT approximation term, all introduced in App. C.

Theorem 6 (KT-KRR) . Let f ⋆ ∈ H ( k ) and Assum. 1 is satisfied. Let δ n , ε n denote the solutions to (100) , (101) , respectively. Denote ̂ f KT ,λ ′ with regularization parameter λ ′ ≥ 2 η n, k , where η n, k is defined by (102) . Then with probability at least 1 -2 δ -2 e -nδ 2 n c ( ∥ f ⋆ ∥ 2 k + σ 2 ) -c 1 e -c 2 n ∥ f ⋆ ∥ 2 k ε 2 n σ 2 , we have

See App. F for the proof. The term U full follows from the excess risk bound of FULL-KRR ̂ f full ,λ . The term U KT follows from our KT approximation. Clearly, the best rates are achieved when we choose λ = 2 η n, k .

E.2 Proof of explicit rates

The strategy for each setting is as follows:

Bound the Gaussian critical radius (101) using [29, Cor. 13.18], which reduces to finding ε > 0 satisfying the inequality

and ˆ µ 1 ≥ ˆ µ 2 ≥ . . . ≥ ˆ µ n ≥ 0 are the eigenvalues of the normalized kernel matrix K /n , where K is defined by (5).

Bound the Rademacher critical radius (100) using [29, Cor. 14.5], which reduces to solving the inequality

where ( µ j ) ∞ j =1 are the eigenvalues of the k according to Mercer's theorem [29, Thm. 12.20] and b is the uniform bound on the function class.

Bound η n, k (102) using the covering number bound N ( B d 2 ( R in ) , ϵ ) from Assum. 3.

In the sequel, we make use of the following notation. Let

according to [16, Eq. 6], where C d is the constant that appears in Assum. 3.

E.2.1 Proof of (147)

We begin by solving (148).

Lemma 9 (Critical Gaussian radius for POLYGROWTH kernels) . Suppose Assum. 1 is satisfied and k is POLYGROWTH with α < 2 as defined by Assum. 3. Then the Gaussian critical radius satisfies

Proof. [16, Cor. B.1] implies that

Let k be the smallest integer such that

By Assum. 1, R n is a constant, so the first inequality is easily satisfied for large enough n

Then

where step (i) follows from the approximation

To solve (148), it suffices to solve

Since ε n is the smallest such solution to (148) by definition, we have (150) as desired.

We proceed to solve (149).

Lemma 10. Suppose Assum. 1 is satisfied and k is POLYGROWTH with α < 2 as defined by Assum. 3. Then the Rademacher critical radius satisfies

Proof. Thus, we can solve the following inequality

Following the same logic as in the proof of Lem. 9 but with β = 1 /b yields the desired bound. Finally, it remains to bound (102). We have

for some universal positive constant c .

In summary, there exists positive constants c 0 , c 1 , c 2 such that

Setting λ ′ = c 2 a 2 n -2 -α 2 out log n out , we have

E.2.2 Proof of (146)

We begin by solving (148).

Lemma 11 (Critical Gaussian radius for LOGGROWTH kernels) . Under Assum. 1 and LOGGROWTH version of Assum. 3, Gaussian critical radius satisfies

for some constant C ′′ α that only depends on α . where we ignore log-log factors.

Proof. [16, Cor. B.1] implies that

Let k be the smallest integer such that

By Assum. 1, R n is a constant, so the first inequality is easily satisfied for large enough n

Thus, k = ⌈ L k ( R n ) log ( 2 e ε ) α +1 ⌉ . Then

Consider the following approximation:

where c ≜ exp( -( L k ( R n ) / 2) -1 /α ) ∈ (0 , 1) . Defining m ≜ -log c > 0 and k ′ ≜ k -1 , we have

by Li et al. [16, Eq. 50], where C α > 0 is a constant satisfying ( x + y ) α ≤ C α ( x α + y α ) for any x, y > 0 and b is a known constant depending only on α . Plugging in k ′ = ⌈ L k ( R n ) log ( 2 e ε ) α ⌉ , we can bound the exponential by

Note that we can simplify the exponent by -mL k ( R n ) 1 /α = -( L k ( R n ) / 2) -1 /α L k ( R n ) 1 /α = -2 1 /α . Note that k ′ = k -1 ≥ L ( R n ) = 2 m -α . Thus, we can absorb the b α -1 m -α term in (154) into k and obtain the following bound

where C ′ α depends only on α . Plugging this bound into (153), we have

We proceed to solve (149).

Lemma 12 (Critical Gaussian radius for LOGGROWTH kernels) . Under Assum. 1 and LOGGROWTH version of Assum. 3, the Rademacher critical radius satisfies

Proof. Following the same logic as in the proof of Lem. 11 but with β = 1 /b yields the desired bound.

Finally, it remains to bound (102). We have

for some universal positive constant c .

In summary, there exists universal positive constants c 0 , c 1 , c 2 such that

Setting λ ′ = 2 c 2 a n out log( en out a ) α/ 2 , we have

F Proof of Thm. 6: KT-KRR

Our first goal is to bound the in-sample prediction error. We relate ∥ ̂ f KT ,λ ′ -f ⋆ ∥ 2 n to ∥ ̂ f full ,λ -f ⋆ ∥ 2 n , where the latter quantity has well known properties from standard analyses of the KRR estimator (refer to [29]). Note that regularization parameter λ ′ of KT based estimator ̂ f KT ,λ ′ is independently chosen from the regularization parameter λ of the estimator based on original samples ̂ f full ,λ . For ̂ f full ,λ , we choose the regularization parameter

which is known to yield optimal L 2 error rates.

for some constant C ′′ α that only depends on α and universal positive constant c . To solve (148), it suffices to solve

which is implied by the looser bound

The solution to (148) (up to log-log factors) is

Define the main event of interest,

Our goal is to show E occurs with high probability. For that end, we introduce several additional events that are used throughout this proof.

For some constant c > , define the event of an appealing in-sample prediction error of ̂ f KT ,λ ′ ,

where η n, k is defined in (102). Recall E KT ,δ is the event where KT-COMPRESS++ succeeds as defined by (24).

Further as f ⋆ and ̂ f KT ,λ ′ are both in { f ∈ H : ∥ f ∥ k ≤ R } , we may deduce that all the functions under consideration satisfies ∥ f ∥ ∞ ≤ ∥ k ∥ ∞ ∥ f ∥ k ≤ ∥ k ∥ ∞ R where ∥ k ∥ ∞ < ∞ . Accordingly, we define a uniform concentration event,

Event (156) is analogous to the event E conc previously defined in (110) when dealing with finite rank kernels.

We first show that

Notice that almost surely we have thereby implying

Next invoking the event E KT ,n ( ε n ∨ δ n ) along with (158), we have

which recovers the event of E .

The remaining task is to show E is of high-probability, which amounts to showing events E KT ,n ( t ) and E ′ conc are of high-probability by reflecting on (157). From [29, Thm. 14.1], we may immediately derive

for some constants c 1 , c 2 > 0 .

We further claim that

for some constants c 0 , c 1 , c 2 > 0 . Proof of claim (159) is deferred to App. F.1. Plugging in t = ε n ∨ δ n into (159), and invoking inequality ε n ∨ δ n ≥ δ n so as to decouple the dependence on S in , we have

which further implies

Putting the pieces together, for some constants c 0 , c 1 > 0 , we have

Overall, (157) and (160) collectively yields the desired result.

F.1 Proof of claim (159)

To prove claim (159), we introduce two new intermediary and technical events. For some positive constant c 0 , define the event 6 when in-sample prediction error of ̂ f full ,λ is appealing

The second intermediary event, denoted as E ̂ ∆ KT ( t ) , is the intersection of (169) and (170), which we do not elaborate here due to its technical nature-event E ̂ ∆ KT ( t ) plays an analogous role to A c KT ∩ B c KT defined in (123) and (136) respectively.

Our goal here is two-folds: first is to show

and second is to prove the following bound

from which (159) follows. Note that Wainwright [29, Thm. 13.17] show

for some constants c 1 , c 2 > 0 and that P ( E KT ,δ | S in ) ≥ 1 -δ . So it remains to bound the probability of event E ̂ ∆ KT ( t ) , which we show below.

Given f , define the following quantities

In the sequel, we repeatedly make use of the following fact: on event E KT ,δ defined in (24), we have

The claim of (162) is deferred to the end of this section. Given f , we can show with some algebra that

where Z ≜ ( f ( x 1 ) -f ⋆ ( x 1 ) , . . . , f ( x n ) -f ⋆ ( x n )) and ξ ≜ ( ξ 1 , . . . , ξ n ) are vectors in R n . Define the shorthands

In the sequel, we use the following shorthands:

Now for the main argument to bound ∥ ̂ f KT ,λ ′ -f ⋆ ∥ 2 n . When ∥ ̂ ∆ KT ∥ n < t , we immediately have ∥ ̂ ∆ KT ∥ 2 n < t 2 , which implies (159). Thus, we may assume that ∥ ̂ ∆ KT ∥ n ≥ t . Note that

Given the optimality of ̂ f full ,λ on the objective (3), we have

where the last inequality follows trivially from dropping the -∥ ̂ f full ,λ ∥ 2 k term. Thus,

Using standard arguments to bound the term 2 n ⟨ Z KT , ξ ⟩ , we claim that on the event E ̂ ∆ KT , we have

for some positive constants c, c ′ , and that P ( E ̂ ∆ KT | S in ) ≥ 1 -e -nt 2 2 σ 2 . We defer the proof of claim (165) to the end of this section.

Now we bound the stochastic term [ L n ( ̂ f KT ,λ ′ ) -L n ( ̂ f full ,λ ) ] in (165)-first observe the following decomposition:

On the event E KT ,δ (24), the first term in the display can be bounded by

Note that ̂ f KT ,λ ′ is the solution to the following optimization problem,

so the second term in the display can be bounded by the following basic inequality

Thus, on event E KT ,δ , we have

where steps (i) and (ii) both follow from the fact that λ ′ ≥ 2 η n, k (see assumptions in Thm. 6). To bound ∥ ̂ f full ,λ ∥ 2 k , we use the following lemma:

Lemma 13 (RKHS norm of ̂ f full ,λ ) . On event E full ,n (161) , we have the following bound

for some constant c 0 > 0 .

See App. F.2 for the proof. Putting things together, we have

for some constant c -substituting this bound into (165) yields

for some constants c, c ′ , which directly implies (159), i.e. implying

Proof of claim (162). Given f , define the function

and note that

We first prove a generic technical lemma:

Lemma 14 (KT-COMPRESS++ approximation bound using k RR ) . Suppose f 1 , f 2 ∈ H ( k ) and a, b ∈ R . Then the function

lies in the RKHS H ( k RR ) . Moreover, on event E KT ,δ , we have

uniformly for all non-zero f 1 , f 2 ∈ H ( k ) .

See App. F.3 for the proof. Applying the lemma with f 1 ≜ f, f 2 ≜ g and a = 1 , b = -2 , we have

which combined with the observation (168) yields the desired claim.

Proof of claim (165). Case I: First suppose that ∥ ̂ ∆ KT ∥ k ≤ 1 . Recall that ∥ ̂ ∆ KT ∥ n ≥ t ≥ ε n by assumption. Thus, we may apply [29, Lem. 13.12] to obtain

Plugging the above bound into (164), we have with probability at least 1 -e -nt 2 2 σ 2 :

We can solve for ∥ ̂ ∆ KT ∥ n using the quadratic formula. Specifically, if a, b ≥ 0 and x 2 -ax -b ≤ 0 , then x ≤ a + √ b . Thus, we have with probability at least 1 -e -nε 2 n 2 σ 2 :

Using the fact that ( a + √ b ) 2 ≤ 2 a 2 +2 b , we have with probability at least 1 -e -nt 2 2 σ 2 :

Case II: Otherwise, we may assume that ∥ ̂ ∆ KT ∥ k > 1 . Now we apply [29, Thm. 13.23] to obtain

for some universal positive constants c 1 , c 2 . Plugging the above bound into (164) and collecting terms, we have with probability at least 1 -c 1 e -nt 2 c 2 σ 2 :

Solving for ∥ ̂ ∆ KT ∥ n using the quadratic formula, we have with probability at least 1 -c 1 e -nt 2 c 2 σ 2 :

Using the fact that ( a + √ b ) 2 ≤ 2 a 2 +2 b , we have with probability at least 1 -c 1 e -nt 2 c 2 σ 2 :

for some positive constants c 3 , c 4 , where step (i) follows from that fact that λ = 2 ε 2 n by (155).

F.2 Proof of Lem. 13: RKHS norm of ̂ f full ,λ

Given the optimality of ̂ f full ,λ on the objective (3), we have the following basic inequality

Since ∥ ̂ f full ,λ -f ⋆ ∥ 2 n ≥ 0 , we also have the trivial lower bound

Thus,

and it remains to bound 2 n ⟨ Z full , ξ ⟩ .

Case I: First, suppose that ∥ ̂ ∆ full ∥ k > 1 . Then we may apply [29, Lem. 13.23] to obtain

Combining this bound with (171), we have with probability at least 1 -c 1 e -nε 2 n c 2 σ 2 :

where step (i) follows from triangle inequality. Solving for ∥ ̂ f full ,λ ∥ k using the quadratic formula, we have

On the event E full ,n (161), we have ∥ ̂ ∆ full ∥ n ≤ c ∥ f ⋆ ∥ k ε n for some positive constant c , which implies the claimed bound (166) after some algebra.

Case II(a): Otherwise, assume ∥ ̂ ∆ full ∥ k ≤ 1 and ∥ ̂ ∆ full ∥ n ≤ ε n . Applying [29, Thm. 2.26] to the function sup ∥ g ∥ k ≤ 1 ∥ g ∥ n ≤ ε n ∣ ∣ 1 n ∑ n i =1 ξ i g ( x i ) ∣ ∣ , we have

Combining this bound with (171), we obtain

which immediately implies the claimed bound (166).

Case II(b): Finally, assume ∥ ̂ ∆ full ∥ k ≤ 1 and ∥ ̂ ∆ full ∥ n > ε n . Applying [29, Lem. 13.12] with u = ε n , we have

Combining this bound with (171), we obtain

which immediately implies the claimed bound (166).

F.3 Proof of Lem. 14: KT-COMPRESS++ approximation bound using k RR

By Grünewälder [12, Lem. 4], ℓ f 1 ,f 2 lies in the RKHS H ( k RR ) , which is a direct sum of two RKHS:

where k 1 , k 2 : Z × Z → R are the kernels defined by

Applying Lem. 2 with

yields the following bound on event E KT ,δ (24):

We claim that

for some positive constant c , where N † k RR is the cardinality of the cover of B † k RR ≜ { ℓ ′ f / ∥ ℓ ′ f ∥ k RR : f ∈ H ( k ) } for ℓ ′ f defined by (167). Proof of the claims (172) and (173) are deferred to the end of this section. By definition of W k RR , we have

On event E KT ,δ , we have

where step (i) follows from the fact that ( ∥ k ∥ 1 / 2 ∞ , in + Y max ) 2 ≤ 2( ∥ k ∥ ∞ , in + Y 2 max ) .

Since f 1 , f 2 are non-zero, we have ∥ ℓ f 1 ,f 2 ∥ k > 0 . Thus, the function h ≜ ℓ f / ∥ ℓ f ∥ k RR ∈ H ( k RR ) is well-defined and satisfies ∥ h ∥ k RR = 1 . Applying (174), we obtain

Multiplying both sides by ∥ ℓ f ∥ k RR and noting that

we have on event E KT ,δ ,

which directly implies the bound (162) after applying the shorthand (102).

Proof of (172) Define Y max ≜ sup y ∈ ( S in ) y y . We have

Proof of (173) Since H ( k RR ) is a direct sum, we have

where N † k 1 and log N † k 2 are the covering numbers of B † k 1 ≜ { f 2 / ∥ f 2 ∥ k 1 : f ∈ H ( k ) } and B † k 2 ≜ { f ⊗⟨· , y ⟩ R / ∥ f ⊗⟨· , y ⟩ R ∥ k 2 : f ⊗⟨· , y ⟩ R ∈ H ( k 2 ) } , respectively. Note that

Define a kernel on R by k R ( y 1 , y 2 ) ≜ y 1 y 2 . When sup y ∈ ( S in ) y | y | ≤ Y max , we have

Similarly, note that

Substituting the above two log-covering number expressions into (175) yields

for some universal positive constant c .

G Non-compact kernels satisfying Assum. 2

The boundedness, Lipschitz assumption, square-integrability, and (15) follow from [10, App. O] and [10, Rmk. 8]. Thus, it only remains to verify (16) and (17) for each kernel.

G.1 Gaussian

For the kernel by k ( x 1 , x 2 ) ≜ exp( -∥ x 1 -x 2 ∥ 2 2 h 2 ) , we have κ ( u ) ≜ exp( -u 2 / 2) .

Verifying (16). For any j ≥ 1 and x ∈ X , we have

By [15, Lem. 1], we have

Define t ≜ 2 d and R ≜ 2 √ t . We can directly verify that R 2 -d ≥ 2 √ dt +2 t . Thus, we may further upper bound (176) by

whenever R ≥ 2 √ 2 d , or equivalently, j ≥ log 2 (1 + 4 √ 2 d ) . Under this regime, we have

for some positive constant C that does not depend on j or n as desired.

Verifying (17). Fixing j ≥ 1 , we have

Note that (log 2)( j ( d + β ) + d ) -(2 j -1 -1) 2 is concave over j > 1 and maximized at j = ⌊ log(1 + √ 1 + 2( β + d )) ⌋ . Treating ∫ 1 2 0 κ ( u ) u d -1 du as a constant, there exists c 2 such that (17) is satisfied for all positive integers j .

G.2 Matérn

We consider the kernel k ( x 1 , x 2 ) ≜ c ν -d 2 ( ∥ x 1 -x 2 ∥ 2 h ) ν -d 2 K ν -d 2 ( ∥ x 1 -x 2 ∥ 2 h ) with ν > d , where K a denotes the modified Bessel function of the third kind [30, Def. 5.10], and c b ≜ 2 1 -b Γ( b ) . We have k ( x 1 , x 2 ) = κ ( ∥ x 1 -x 2 ∥ 2 ) ≜ κ ν -d ( ∥ x 1 -x 2 ∥ 2 ) , where

Verifying (16). Fix j ∈ { 0 , 1 , . . . , T } . Applying Jensen's inequality, we have

where τ k ( · ) is the kernel tail decay defined by [10, Assum. 3]. We may use the same logic as in [10, App. O.3.6]-but ignoring the A ν,γ,d term and making the substitutions, a ← ν -d 2 , γ ← 1 h , and Γ( ν -1) ← Γ(2 ν -1) -to obtain

Hence, we have

whenever (2 j -1 -1 2 ) h ≥ 2 ν -d √ 2 h ⇔ j ≥ 1 + log( 2 ν -d √ 2 + 1 2 ) for some constant C h,d,ν that does not depend on j .

Verifying (17). Following similar logic as in [10, App. O.3.1], we have

Fixing j ≥ 1 , we have

Note that when ν -d 2 -1 < 1 , the RHS can be rewritten as C β,d (2 j -1 ) d + β exp( 1 2 2 j -1 ) for some constant C β,d that doesn't depend on j . When ν -d 2 -1 ≥ 1 , the RHS can be upper-bounded by C ′ β,d (2 j -1 ) j ( ν + β + d 2 -1) exp( -1 2 2 j -1 ) for some constant C ′ β,d that doesn't depend on j . Observe that the function t b e -t/ 2 attains its maximum at t = 2 b . Hence, the RHS can be bounded by a constant that does not depend on n as desired.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and introduction give clear outlines on our contributions, and we present our contributions in the main text accordingly. We have also included pointers in the introduction that would link to the referred main text containing specific contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Sec. 6, we discuss limitations as well as future work to address these limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have clarified the assumptions and models required for all the theorems and corollaries provided in the main text and appendix. Also we provide a complete proof in the appendix for all the stated results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe details of the experiments in Sec. 5 and provide links to all code and data.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to our GitHub repository containing all code in Sec. 5.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide train-test splits and hyperparameters in the main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In all figures, we plot error bars representing standard deviation across 100 trials. In all tables, we report mean +/- standard error across 100 trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We indicate the computer resources for running all experiments in Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and our paper conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work reduces the computational costs of classical methods and is applied to standard datasets. Thus, it has no outsize societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release models or data as part of this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include URL and licenses for baseline code and datasets used in Sec. 5.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release our code with documentation at https://github.com/ag2435/ npr under a BSD-3 Clause license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not have any studies or results regarding crowdsourcing experiments and human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have any studies or results including study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies the kernel thinning approach to non-parametric regression, with two kinds of estimators (NW and KRR) considered. This approach speeds up the computational efficiency by using carefully chosen coresets for approximation. Theoretical results on the approximation error are established. Numerical experiments include synthetic data and the well-known California housing dataset.\n\nSTRENGTHS:\n1. The paper is well organized and written, with enough background and clear motivation\n2. Solid theoretical results are developed for the proposed method\n3. Certain experimental results are given to justify the proposed method\n\nWEAKNESSES:\n1. Despite the well developed theory, I am not fully convinced yet by the significance/contribution of this method - I am willing to stand corrected if more comparisons can be made with other existing methods for efficient non-parametric regression. \n2. The application are only to toy examples or standard small datasets - e.g. for the Californian housing dataset with n = 20,640, doing non-parametric regression without thinning also seems fine.\n3. In the experiments, the baseline is chosen to be NW / KRR with naive thinning - some more competitive, state-of-the-art baselines could have been chosen.\n\nQUESTIONS:\n1. In the beginning of Sec 3, the NW and KRR estimators are restricted to kernels taking the form of (4) and (6). What are the specific reasons to make these restrictions? And to see how restrictive/general this is, could you give some examples of commonly used kernels that fall under this category?\n2. The theory makes the sub-Gaussian tail decay assumption on the kernel, which would exclude many commonly used kernels such as the Matern kernel. How would the theory work out, and how would the rates change if this assumption is generalized to sub-Weibull tail decay, or say a bound on the Orlicz-norm?\n3. The method (or at least the theory) is developed for one-dimensional response data. How could this be generalized to multi-dimensional response data, where issues with computational efficiency becomes more significant?\n4. In the discussion section, it is mentioned that the rates on the excess risk could be sub-optimal. Do you have a sense of what rate could be minimax optimal?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors speed up two non-parametric regression estimators, Nadaraya-Watson (NW) and Kernel Ridge Regression (KRR), by using a kernel thinning (KT) technique to compress the input data in a way that preserves important statistical properties. They include a theoretical analysis proving that that KT-based regression estimators are computational more efficient and also exhibit improved statistical efficiency over i.i.d. subsampling of the training data. They also include empirical validation of their method on simulated and real-world data.

STRENGTHS:
Good presentation with a solid analytical analysis providing theoretical guarantees. Overall well-organized paper.
The proposed method of combining KT with the estimators is novel (to my knowledge) and shows potential.

WEAKNESSES:
The empirical analysis is limited. The shortcomings on the real-world data are not adequately addressed (see also limitations).

QUESTIONS:
I have no particular questions.

LIMITATIONS:
The comparison with other large-scale kernel methods is limited to RPCholesky; a more comprehensive comparison and overview would be nice.

Performance/utility of the proposed method on real-world data remains unclear. The results obtained on real-world dataset in section 4.2 are not convincing, and the authors acknowledge the problem of kernel mis-specification but do not elaborate or provide any resolution. Btw, there is an incomplete sentence in lines 283-284.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAfter the rebuttal, I have updated my score from 3 to 5. \n\n----\nThis paper provides a meta-algorithm based on Kernel Thinning for non-parametric regression, in particular, the Nadaraya-Watson (NW) regression, and the Kernel Ridge Regression (KRR). \n\nThe idea is to run NW or KRR on a thinned coreset by KT. The core of the method is to choose a suitable kernel for which the KT assumptions are satisfied. The paper introduces to meta-kernels that takes a base kernel of features x and produces the kernel over (x,y) suitable for regression tasks. \n\nThe paper provides the theoretical guarantees for kernel thinned NW and kernel thinned KRR. \n\nIn empirical study, the paper includes simulation results and real data results which demonstrate the proposed method's strength over full computation and standard thnnning in accuracy and time; however, the SOTA RPCholesky method seems to have a better empirical performance than kernel thinned KRR.\n\nSTRENGTHS:\n- The paper in general is easy to read\n\n- The idea to introduce kernel thinning to non-parametric regression methods is novel. \n\n- Provides theoretical guarantees of the method under suitable assumptions.\n\nWEAKNESSES:\n- The empirical advantage of the proposed kernel thinned KRR over RPCholesky is not clear. In fact, from figure 5, RPCholesky has slighly higher training time compared to kernel thinned KRR, however, a much lower test MSE. \n\n- Some technical background and discussion can be made more clear. See questions. \n\n- Writing can be improved. See questions.\n\nQUESTIONS:\n- There is not an algorithmic description for Kernel Thinning algorithm. \n\n- Eq 4 and 6: why do you choose the kernel in this way? In particular, can you comment on the part $k(x_1,x_2) y_1, y_2$. Does the performance of the kernel rely on the relationship of $y$ and $x$, and if so, how? Also, in eq 6, is this a typo $k^2(x_1,x_2)^2$? \n\n- Notation suggestion: instead of $S_{KT}$ for KT-NW, could use $S_{NW}$ to match the use of $K_RR$ in KT-KRR part. \n\n- Sec 4.1: are the kernels listed in 1.2.3 used as the base kernel to construct the kernel for thinning, or they are the kernel for thinning?\n\n- Fig 2: Right panel: why not include RPCholesky?\n\n- Fig 2: why si the full not performing the best?\n\n- Fig 3 and Fig 4: why not use the function in (11)? \n\n- Fig 3 and Fig 4: what is the unit of y axis? Are they all in the same unit?\n\n- Sec 4.2: what is the conclusion from Fig 5?\n\n- Sec 4.3: line 283-284: \"In this example (and ....)\" incomplete sentence. \n\n- It'd be good to include Fig 5 in the main text instead of appendix.\n\nLIMITATIONS:\n- KT-KRR does not demonstrate an empirical advantage over RPCholesky. Missing a discussion on this point. \n\n- Missing a discussion on potential failure cases of the method, though there is  a discussion on kernel mis-specification and potential way to deal with it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-29"}