Target alignment in truncated kernel ridge regression
Authors: Arash A Amini
richard baumgartner
Dai Feng
Venue: NeurIPS 2022 Accept (2022)
Keywords: kernel methods
target alignment
minimax rates
double descent

Abstract: Kernel ridge regression (KRR) has recently attracted renewed interest due to its potential for explaining the transient effects, such as double descent, that emerge during neural network training. In this work, we study how the alignment between the target function and the kernel affects the performance of the KRR. We focus on the truncated KRR (TKRR) which utilizes an additional parameter that controls the spectral truncation of the kernel matrix. We show that for polynomial alignment, there is an over-aligned regime, in which TKRR can achieve a faster rate than what is achievable by full KRR. The rate of TKRR can improve all the way to the parametric rate, while that of full KRR is capped at a sub-optimal value. This shows that target alignemnt can be better leveraged by utilizing spectral truncation in kernel methods. We also consider the bandlimited alignment setting and show that the regularization surface of TKRR can exhibit transient effects including multiple descent and non-monotonic behavior. Our results show that there is a strong and quantifable relation between the shape of the alignment spectrum and the generalization performance of kernel methods, both in terms of rates and in finite samples.


Target alignment in truncated kernel ridge regression
Arash A. Amini', Richard Baumgartner", Dai Feng²
University of California, Los Angeles Merck & Co., Inc., Rahway, New Jersey, USA
Data and Statistical Sciences, AbbVie Inc.
aaamini comin.@uula.ccm edu, richard banngaaaeeaarrrrrrrree. dai www.@abbvie..c.mmmmm com
Abstract
Kernel ridge regression (KRR) has recently attracted renewed interest due to its potential for explaining the transient effects, such double descent, that emerge during neural network training. In this work, we study how the alignment between the target funetion and the kernel affects the performance of the KRR. We focus the truncated KRR (TKRR) which utilizes an additional parameter that controls the spectral truncation of the kernel matrix. We show that for polynomial alignment, there is an over-aligned regime, in which TKRR can achieve a faster rate than what is achievable by full KRR. The rate of TKRR the parametric rate, while that of full KRR is capped at sub-optimal value. This shows that target alignment can better leveraged by utilizing spectral truncation in kernel methods. We also consider the bandlimited alignment setting and show that the regularization surface of TKRR can exhibit transient effects including multiple descent and non-monotonic behavior. Our results show that there is a strong and quantifiable relation between the shape of the alignment spectrum and the generalization performance of kernel methods, both in terms of rates and in finite samples.
improve all the way
1 Introduction
Kernel methods have become time-proven popular mainstay in machine learning [23, 27, 22]. Implicit transformation of learning problem via suitable kernel with subsequent development of regularized linear models in the associated reproducing kernel Hilbert space (RKHS) is amenable applications and allow for investigation of theoretical properties of the kernel methods [12]. Recently, there has been a revived interest and increased research into the kernel methods [14, 17, 12, 8].
These efforts have been motivated by the connections between neural network and kernel learning [14]. In particular. the discovery transient effects such as double descent that emerge in neural networks training [6. 20] has attracted a lot of attention.
In this paper, we study the effect of target-model alignment the performance of kernel ridge regression (KRR) estimators. KRR is a well-known nonparametric regression estimator with desirable theoretical performance guarantees. We focus a generalization of the KRR, which we refer to as the Truncated Kernel Ridge Regression (TKRR), that introduces an additional parameter, for truncating the eigenvalues of the underlying kernel matrix. TKRR often used as approximation to the full KRR to scale the computations to large sample sizes. TKRR itself is often approximated in practice by various sampling and sketching schemes [30, 33, 18, 19, 24, 1, 32]. One of our main contributions in this paper is that TKRR can, in fact, improve the statistical rate of convergence in cases where the target function is well-aligned with the underlying RKHS. Roughly speaking, target
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

aligmment refers to the decay rate of the coefficients of the target function in an orthonormal basis of the eigenfunctions of the kernel operator.
In addition to improved rates for TKRR, we also show that target alignment generally predicts the generalization performance of the full KRR well TKRR. We also explore the regularization surface of the KRR estimators in both the regularization and truncation parameters, and show some surprising behavior like non-monotonicity, multiple descent and phase transition phenomena.
Our main contributions
We show that TKRR itself can be viewed as a full KRR estimator in a smaller RKHS embedded in the original RKHS (Propostion 1). This shows that the ad-hoc truncation that is often done for computational reasons can itself be viewed as a proper KRR estimator over a different RKHS.
(ii) We introduce target alignment (TA) scores in Section 4 and derive an exact expression for the Mean Sqaured Error (MSE) of TKRR based on the TA scores and eigenvalues of the kernel matrix (Theorem 1).
(iii) For the case of aolymomially-dccayii TA scores, we first derive a simplified expression for the MSE (Eqn. (14)) which is within constant factors of the exact expression. We then identify four target alignment regimes: Under-aligned, just-aligned, weakly-aligned and over-aligned. We show that TKRR can achieve fast rate of convergence in the over-aligned regime. In contrast, the rate of the full KRR is capped to a sub-optimal level, that of the best achievable in the weakly-aliggne
regime (Theorem 2).
(iv) For bandlimited TA scores, we provide more refined initle analysis. In particular, we show that the regularization curve function of the truncation level is in general non-monotonic. We also show that TA spectra that are narrower and shifted towards lower indices correspond to lower generalization errors (Proposition 2).
We provide experiments verifying the multiple-descent and non-monotonic behavior of the regular- ization curves as well as the improved rate of Theorem 2 (Section 4.2). The MSE in our results, is equivalent to the excess (empirical) generalization error. So all the conclusions, and in particular the rates, are valid for the generalization error. Overall, our results provide strong theoretical and experimental support for the idea that alignment of the target function with the eigenfunctions of the kernel is the key predictor of generalization performance of KRR estimators. is worth noting that our main theoretical results are mnnaptyptoiic and at times exact.
Related work. Kernel ridge regression (KRR) has been widely studied in the literature in the non-parametric statistical framework [29, 26). Various aspects of the KRR were elucidated. For example, the error rates of the KRR were studied in [9] and more recently benign overfitting [5] and related double descent effects in [13].
The notion of kernel-target aligmment was first introduced in [11] for classification. In [8], the generalization error of the KRR was investigated using the tools of statistical physics theory which allows for derivation of the bias-variama decomposition of the generalization error and elucidation of non-monotonic transitional effects reflected by the learning curves. In this work the authors have also demonstrated that the interplay between spectral bias (expressed as kernel eigenvalue decay) and kernel-target alignment or task-model alignment are useful for the characterization of the kernel compatibility with the problem. Building on the [8], the concept of kernel-target alignment and spectral bias was referred to as source and capacity, respectively in [12]. Similarly to [8), the transitions that emerge during crossover from noiseless noisy regime corresponding to fast slow decay of the learning curve, respectively were studied in [12]. Moreover, in [2] it has been shown that the truncated KRR outperforms full KRR over the unit ball of the RKHS Another related work is that of [25] where overparametrized KRR is studied. The work [12] is perhaps closest to ours followed by [2] and [25]. We make more detailed comparisons with these after Theorem 2. In particular, we show that the rate obtained in [12] can be improved by using TKRR. In addition, in Appendix D, we provide a more unified view of the settings of [9, 12] which can help clarify how different target alignment assumptions relate to each other. It is worth noting that approximate forms of TKRR via sampling and sketching have been analyzed extensively in the literature [21, 32, 10, 31, 15, 1, 4], but this line of work at best proves that TKRR achieves the same minimax rate achievable with full KRR, over the RKHS ball. What we show in this paper that TKRR can achieve much faster rates under proper target alignment.

---

2 Kernel Ridge Regression
The general nonparametric regression problem can be stated as
y/s = (x,) 6,, i = ,,....7, n, B( = 0, cov(e) = 0² 1
where = (Ex) E R" is a noise vector and f X' - R is the function of interest to be approximated (yi). Here, is the space to which the covariates æ = (æi) belong.
from the noisy observations We mainly focus on the fixed design regression where the covariates are assumed to be deterministic. natural estimator is the kernel ridge regression (KRR), defined as the solution of the following optimization problem:
fA,A == argmin
1 Σ f(x)) A1fIi.
feH
where >> O is regularization parameter and TH is the underlying RKHS. By the representer theorem [16], this problem can be reduced to a inite-dimeesional one:
min 1 -Iy vnKw||² + dw Kw; where WER" n
K (KKK,,,, (z))))))) E Raxx
(3)
the (normalized empirical) kernel matrix. Let us define the following operators
1 Solf) J f(z).
a Vn Σ k1,)),
(1)w
(4)
for f E H and R". We refer to Se the sampling operator. As the notation suggests, S is the adjoint of Sw as an operator from to RR. We have fn,x = St (@) where the solution of (3). We note that S² (S) = Kw, i.e., K is the matrix representation of S. IR RR.
Notation. We write (f.g)n = f(r))(z)) and [I = ((f. for the empirical inner product and norm, respectively. Note that Il (f)()) where || ll2 the vector l norm. For two functions, f and we write f S if there is constant C' > 0 such that f S Cg. and we write f= g if f and g S
3 Truncated Kernel Ridge Regression
As mentioned earlier, spectral truncation of the kernel matrix has been mainly considered as a computational device in approximating the full KRR estimator. Here, first argue that it can be reformulated as a KRR estimator over simpler RKHS. To facilitate future developments, we define this RKHS based on the given training data æ (I1,..., In).
Let K = Ei= nn be the eigendecomppsition of the kernel matrix K where μ1 2 #2 2 .... Nn 0 are the cigenvalues of K and { C _ are the corresponding cigenvectors. We assume for simplicity that H4, O, that is:
Assumption 1. The kernel matrix K K(x) is invertible.
This a very mild assumption for any would like to find functions { H such that ( = //n usa for all k, i E [n]. If dim(H!) oc, there are in general many such functions for each k. We pick the one that minimizes the H-norm:
H. Let uki be the ith coordinate of ug. We
V = argmin [[lI|E
H. Sa(v) = uu
Under Assumption 1, the above problem has a unique solution (i.e., the interpolation problem Sy (v)) has unique minimum-norm solution for any u E R² .). Since {ux} is orthonormal basis of gn, by construction, (t', ve)n = Lík f}. Consider the set of functions
Σ a ai,..., Or E ER
3

---

equipped with the inner product defined via (2hk, ve)g 1 1[k { and extended to the whole of H by bilinearity. Equivalently, if = Σk=1 ag and = Σx= Su.R,. define (f.9)a = En ouBs/Pk. Then, H an r-dimensional RKHS, with reproducing kernel (Appendix A)
K(x, y)
Σ (cdx(z)
(5)
and by construction H C Hi, although H and H. are equipped with different inner products.
3.1 TKRR is itself a valid KRR
We define the truncated kernel ridge regression (TKRR) estimator as the usual KRR estimator over HI. that is,
JAA- argmin sel
Σ Ui - (x:))
(6)
We have the following characterization of Fr.A: Proposition 1. Let K- Ek=1 M (x),,,)) be the truncated kernel matrix and
(w) = Vn w.)
which is the adjoint of Sn as a map from H. to RR. Let Q be the solution set of (3) with K replaced with K. Then, (6) has unique solution which is given by fr,x (w) for any Ω
Although the above proposition is not the main contribution of this work, it provides the interesting observation that TKRR an exact KRR estimator over a smaller RKHS. See also (21) in Appendix B for closed-form expression for fr,x-
Target alignment
We start with the definition of target alignment:
Definition 1. The alignment spectrum target function U" Tk" where U the orthogonal matrix of the eigenvectors of K, with columns corresponding to eigenvalues of K in descending order. The elements of ç* are referred as target alignment (TA) scores.
The following theorem gives an exact expression for the expected empirical MSE of TKRR estimate, in terms of the TA scores 5* and the eigenvalues (H,) of the kernel matrix:
Theorem (Exact MSE). Let Tx be an diagonal matrix with diagonal elements:
(Tx)
1(1 sisr).
(8)
14
For any TKRR solution Fr,x we have
ElI,FAA 1- II - -XD
tr([
n
2
(E:)
Σ (Et) 2
(9)
(Mi
X)
(14 + X)2
1| Σ
(μ + -w x)a a(())))))))
(10)
where (AA)- (H, X)2 X2 and the expectation is w.r.t the randomness the noise vector w.
an orthogonal matrix, this is equivalent to ||_" 1. Since the coefficient of (ET)² in (10) is negative, the more the alignment spectrum peaks near the lower indices, the smaller the expected MSE This observation made more precise in Proposition 2 below.
Assume that target function is normalized so that || 1. Since

---

4.1 Bandlimited alignment
To see the implications of Theorem 1, let us first consider a bandlimited model follows: f is randomly generated with alignment scores či satisfying
1 B(G))² - iile+liise 1itliiittt 6
(11)
for L 2 0 and b 21 1 (both integers). Then, we write f Bo, and note that R = 1 for such f". We write MSE F1lfa - Ilă where the expectation over the randomness in both f and whe noise vector (1). One can think of this setting as Bayesian model for f and of MSE as the Bayes risk w.r.t to the aforementioned prior on f". We leave the specification of the prior open subject to the condition (11) since this the only assumption we need.
Proposition 2. With Bo,t, we have
(B+b)AT Σ
(A) (p4 + X)2
o²
MSE
(12)
(1u + X)2
n
Assume for simplicity that {15} are distinct. Then, the following hold:
(a) For fixed é, b and X, let j = maxfi E 1,+ + 1 21/ps S obb/n] if the set is nonempty, otherwise j = f. Then, the MSE as a function of is (i) increasing in [1,j*), (ii) decreasing in [j", b), and (iii) increasing in [E b, n].
(b) For fixed r, b and X, the MSE as function of is increasing in & E [0, b).
(c) For fixed r, i and X, the MSE function of is increasing in b [L. 2].
Part (a) of Proposition 2 shows that MSE as function of can be non non-monotonic: go up. down and back up. This happens whenever j" + b. Similar observations as Proposition 2(a) can be made about bandlimited signals consisting of multiple nonzero bands, e.g., f +
where f:" Er Bb,,, with non-overlapping bands. In this case, MSE can show multiple descents and ascents (cf. Figure 2a). Part (b) shows that alignment spectra that are concentrated near lower indices are better. Part (c) shows that concentrated alignment spectra are better than diffuse ones.
Effect of noise level on optimal rank r Let explore the implications of Proposition 2(a) for the optimal truncation level (optimal rank). Fix À > 0. Then, for small enough noise levels (i.c., small o²/n), we have + 2A/Me-1 d2b/n. Since + μ4; is (strictly) decreasing, it follows that j and r + MSE is decreasing over the entire interval [e, l b). That is, for increasing, but the moment we enter the signal band (r keeps decreasing till we get to the end of the band (r = + b), at which point it starts to increase again. For small enough noise levels, the descent within the band is big enough to bring the MSE below its level
+ 1, the MSE is +- 1), the MSE starts decrease and
1; hence the optimal truncation level, in this case, is the end of the band: r r= + b. If we increase the noise level, at some point, j starts to move to the middle of the band [e, b). In this case, depending on where in the band j lies, the optimal truncation level could still be = L + b. But as we increase the noise level, there comes a point where the noise level is so high that j is very close to the end of the hand, in which case, the dip in the MSE is not big enough to bring it below its r 1 level. Then, the best truncation is just at r 1 level. This includes the case where j + b in which case the MSE is monotonically increasing (very high noise levels). Similar ideas play out in the multi-band setting as shown in Figure 2a in Section 5.
4.2 Polynomial alignment
We now consider the case where the TA scores decay polynomially with a rate potentially faster or slower than what is required by merely belonging to the RKHS. In particular, assume that
14 x i (
x 27a-1
(13)
for some 7 > and a 2 1. Let us justify the choice of decay for (27)2. Asymptotically, both (2 and μr should decay similar to the population level TA scores and eigenvalues. Then, f E Hi if and
5

---

only if (5)) 2 1 (see Appendix D). Assuming a polynomial decay for (E])?, it follows that that (ET)² should decay faster than i i-0-1 for the sum to converge, that is S ;-a-1-6 for some ő >> 0. Without loss of generality, we are assuming å = (2> - 1)a. This parameterization is chosen to be consistent with the existing literature. Thus, for function f E IH, 5 satisfies (13) with 7> > 1/2. The case 7 S 1/2 then corresponds to a target that does not belong to the RKHS.
The following result characterizes the performance of full KRR and TKRR under model (13):
Theorem 2. Let min(r, X 1/a)). Under the polynomial decay assumption (13),
x² -
ElJfAA 1 x X² max(1, (1, 2(7-1)a) (1)))))1 + +r 2 1rr < n)
(14)
(a) Taking A x (o² (o2/n) /n)ra/(2aaat)) and (n/o2)1/(220)) TKRR achieves the following rate
022) 27a/(270+1) for y > 1.
BI 1
(15)
12
(b) Assume n." 20 o n, and let ō == min(1, 7). Then the best rute achievable by the full KRR obtained for regularization choice X x (o² /n)a/a///// (260/12)++) and is given by
228/(2))+])
F - 1&
for > O.
(16)
Note that the exponent 8 saturates to 5 = 1 for all y > 1 giving the best rate (a²/n) 2200222222221) for the full KRR in those cases. In contrast, the rate in (15) achievable by TKRR improves for y > 1 without bound, and in the limit of + oc, approaches o² o²/n, the best parametric rate.
It is also well-known that the minimax rate over the unit ball of the RKHS with eigendecay 14 i-a is (o" /n) (a))))))) This can also be seen by letting 7 - 1/2 in either (15) or (16), removing any target alignment assumption beyond what is provided by the RKHS itself.
To summarize, let us define the rate exponent function
s s1>) := 2/a/ + 1).
(17)
There are four regimes of target alignment, implied by Theorem 2:
1. Under-aligned regime, 7 E (0, }): The target is not in the RKHS. The best achievable rate is (o² (o²/n)e(>) which is slower than the minimax rate over the ball of the RKHS, (o²/n) /n))(t)
2. Just-aligned regime, 7 (to be precise 7 4 3): The target is only assumed to be in the RKHS. The best rate achievable the minimax rate over the ball of the RKHS, (a" ((2/n)e()),
3. Weakly-aligned regime, 7 E (+) 1]: The target in the RKHS and more aligned with the kernel than what is implied by being in the RKHS. The best achievable rate (o? (2/n)(()) which is faster than the minimax rate over the ball of the RKHS, (o² (2²/n)=(2), The rate is achievable by the full KRR and hence TKRR.
4. Over-aligned regime, 7 > 1: The target is in the RKHS and is strongly aligned with the kernel. The best achievable rate is (o² //je(t)) which is achieved by TKRR. The full KRR can only achieve the rate (d²/n)o(1) in this case, which is the best achievable in the weakly-aligned regime.
Note that the best achievable rate in the first three regimes is attainable by full KRR (hence TKRR) while for the fourth regime, only TKRR can achieve the given rate. It was shown in [9] (cf. Ap- pendix D.2 for details) that the rate given in the weakly- class of targets (13) with y E (1/2, 1]. We conjecture that the minimax optimality of this rate extends to the over-aligned regime, that is, TKRR is minimax optimal for all y > 1/2.
regime is minimax optimal over the
The capped rate (16) for the full KRR is the same as the one obtained [12, Eqn. (12)). The rate (15) for the TKRR in the over-aligned regime is new to the best of our knowledge. The work of [2] shows some improvement in TKRR over full KRR, but since the minimax rate was considered only over the unit ball, no difference in rates was shown between TKRR and full KRR. In contrast, by

---

considering the smaller alignment class (13), we can show the rate differences. We also note that the cross-over effects noted in [12] for different regularization regimes follow easily from the simplified formula (14), can be seen by inspecting the various cases in the proof of part (b) of Theorem 2. To simplify the statement of Theorem 2, we have focused only on the optimal regularization regime.
There are interesting parallels but also notable differences between our work and that of [25]. They consider the usual ridge regression while we work with the kernel ridge, although the results can be translated back and forth after some transformation. They use a spectral truncation parameter k, as a proof-device to obtain the tightest upper bound, whereas the truncation level r in our case is a regularization parameter in TKRR that can be tuned in practice. The bound in [25] essentially corresponds that of the Full KRR in our paper. In fact, we also use a similar proof-device & in the proof of Theorem 2, separate from the r-truncation level of TKRR. As an important corollary. our results about TKRR cannot be deduced from those in [25]. Another point of difference the focus in [25] which is on how the eigendecay of the kernel (or equivalently the covariance matrix) affects the MSE bound, while our focus on how the interaction of the target alignment decay and the kernel eigendecay together affect the bound. There are indications of the effect of the target alignment in [25], in the form of the tail energy of their 0* parameter, but the implications this decay seems to have not been explored in detail compared to our work.
Generalization error We have considered fixed design so far, that is, fixed (z;). To see the connections with the generalization error, we need to consider a random design. Let (z, y) be random test point, and let (11.8),..... (Fn, Un.) ) be i.i.d. training data, all from the same joint distribution on (I. y). Let Fx be the marginal distribution of x under P. and for function f. write || f f JFF x (x) for the corresponding (population) squared L norm, Recalling that y = / (x) + e and E(e] = 0, the generalization error of a deterministic f is
Bly f()))² (()))) +o² =1 - f +
where the first expectation is taken w.r.t. the randomness in both and y (that is, and E). The noise variance o² is the unimprovable part of the generalization error, i.e., the minimum Bayes risk. So, the excess generalization error is |I f" For large n, since the L norm is an integral, it can be well-approximated by an average based on the training x = (z:)i- that is, which is the empirical norm that we have considered in this paper. In fact, letting B, be the expression in (10) and taking the expectation of both sides of (10) w.r.t. æ, by Fubini theorem, we have fr, ,' E, where Ex denotes the expectation w.r.t. training x. This controls the expected excess generalization error of Fr.x.
1 (f(x)) - fxx)))
More precise higb-probability bounds on I| 1' could be possible, by techniques similar to those in [28, Theorem 14.12], establishing a high probability uniform control of the form || < I?? over the unit ball of an RKHS generated by a Mereer kernel. There are, however, technical difficulties in controlling the so-called critical radius in these arguments at the level needed for the fast rates we present here. Combined with sub-Gaussian assumption on the noise vector these arguments could lead to high probability bounds on the population excess generation error similar to the results we present here. Whether such arguments are feasible is left for future work.
5 Simulations
We present various simulation results demonstrate the multiple-descent and phase transition behavior of the regularization curves, and corroborate the theoretical results. First, consider the bandlimited alignment (11) where the nonzero entries of are i.i.d. draws from N(00 1) and {" is normalized to have unit norm. We use the Gaussian kernel e -|===///1//2 in d = dimensions with bandwidth h = dd2, applied to 200 samples generated from a uniform distribution on [0, 1Jd, and let r = 1 b.
The plots in Figure la show the multiple-descent and phase transition behavior of the X-regularization curves (that is, expected MSE versus (A)) for different values of the noise level a. The corresponding contour plot is shown in Figure 1b. The plots show a transition from monotonically decreasing at a = to monotonically increasing for large a, with non-monotonic behavior for moderate values of o.
7

---

MSE vs. (o)(()
MSE for different lambda and sigma values
[m0, b=30, m30
|=0, b=30, r=30
0- 2.8
35-
1.6
033
305
050
0.8
MRE
3.434
0-4
155
001
0-
385
122
-ogglambda)
1og2.>
(a) Panel plot
(b) Contour plot
Figure Multiple-descent phase transition for A-regularization curve: (a) Expected MSE as function (A) for different values of a, and (b) overall contour plot of expected MSE for a vs. - -A)((A).
1--00 log(3))
MSE for different and sigma values 11=0, 12=30, b=10, +og(lambba)=11155
a/-7i
a/-ii 0.08
e/ - 0.12
03
34
Ogmaasqrttn
M8E
6/ 0.18
at.h 0.24
0/-7 0.3
0.5
20
30
(a) Panel plot
(b) Contour plot
Figure 2: Double-descent and phase transition for r-regularization curves: (a) Expected MSE a function of the truncation parameter r. Plots indexed by different values of a /n, and the r-axis starts 1. (b) Overall contour plot of expected MSE for / /n vs. r.
Next, we consider bandlimited alignment with two non-overlapping bands of length starting at indices & + 1 and l, + 1. The nonzero entries of g" are generated as before, followed by unit norm normalization. We fix À and consider the r-regularization curve, that is, the plot of MSE versus truncation parameter
The results are shown in Figure 2a for different values of a/ /n. Figure 2b shows the corresponding contour plot. The plots show the double-descent and phase transition behavior of the r-regularization curves. The double-decent behavior is consistent with Proposition 2(a) that predicts the MSE goes up as r varies between bands. Simulations corroborating Theorem are provided in Appendix C. The code for reproducing the simulations is available at [3].

---

6 Conclusion
We presented an analysis of the TKRR in the light of recent advances that were made for the KRR. We showed the TKRR equivalence to a KRR on (smaller) RKHS embedded in the original RKHS. We believe that this insight provides different perspective on the TKRR and unifies it with the KRR development. We derived an exact MSE decomposition for the TKRR via TA scores and eigenvalues of the kernel matrix. When TA scores exhibit polynomial decay we identified an over-aligned regime, where the TKRR can outperform ordinary KRR. When the TA scores are bandlimited, we demonstrated the non-monotonicity of the regularization curve with respect to the level of truncation and a correspondence of narrower TA spectra that are positioned towards lower indices with lower generalization error.
Although we defined the TA spectrum based on the true f*, there is a very good plugin estimator for in practice: Replacing S (f) with the noisy observations y/ v in the definition of the spectrum gives a very good estimate, that is, U //v V is very good estimate of L (f()). This is because each uf //v/ will effectively average over the noise in y.
Our results can be easily extended to the non non-additive and heteroscedastic noise setting, where (x; y) drawn from a general distribution, (z) == Ely x and = y f*(x)) The extension is possible if we assume var(s] x)) S a², in which case, Theorem 1, for example, holds as an upper bound on the MSE. However, extensions to more general loss functions, such logistic loss for classification, requires additional work. Whether the improved rate (15) for TKRR is minimax optimal for y > 1. appears to be open, although we conjecture it be the case. Due the theoretical nature of this contribution, no negative societal impact of our work is anticipated.
7 Proofs
Here, we only prove Theorem 2. The remaining proof' are given in Appendix B.
Proof Theorem 2. To simplify the argument, we assume that X-1/a is an integer, without loss of generality. Recall that for decreasing function J. we have Ei= f(i) S f(x)dr. Let us first consider the sum involved in the variance term. We will use the following inequalities, min b) < ab I min(a, b) which holds for a, b 2 0. We have a+h
h
x
min .4²)
( X)²
2 1=1
+ X)2
X2
i=1
Since μn = i- the minimum above is X for i E 11,A-1/a) and 2 for i i> 1/0, If < 1/a only the first case happens and the overall bound is Hr = r. Ifr X-1/a 1/a =; k. we have
1 [x22 1 x X2
Σ
1-20) = -
x 20 dx S k x 2/4 2a+1 z x 1/a
Jk
i 1
To summarize, 11 S min(r, x -1/a) and we note that the variance term in (9) is (o² h. For the middle term in (9), we have, assuming n,
n
227a-1 < 1 -21a- dx 22-2
(E))
Finally, for the first term in (9), we can write
2
X²42
{ 2
In 11
min (22, ,n,)((
(4:
2 14!
If r < X 1/a 1/a, we have T/ < X2
Σ
< 2
(11)
-i=1 i 2(7 1)a-1 Letting B = 2(7 1)o,
1
1 9--1 <1 /
1 3 (1-
1 1+
x x-8-1 dx = 1+
x2
2=
9

---

y 2 1, the sum above is 2 1. If < 1, then 3 < 0 and the sum is S rl The two cases can be
compactly written as S max(1, (-) P). Ifr X 1/a == k, we have
12 S
X max k0) 2
using the bounding techniques have seen before. Plugging in A: A A-1/a, we obtain
12 S X max(1, (2(7-11) + A max(>², ,²7)
To summarize, for A-1/a, we have x X (1, while for > X-1/a, have 12 X2 X2 Note that these two expressions can be combined into one follows:
1 S X² max(1, 9),
where 17 min(r; A X-1/a). Putting the pieces together gives the desired bound. The above upper bounds are sharp up to constants, since in each step, there is corresponding matching lower bound.
Proof of part (a). With the given choice for À and r. we have x -1/aa x (n//2)//((aatt)) z (n/o2))/(27a+2) x for y 2 1. Thus v = min(r, x -1/a) x r, hence 1/ -2(-11 x 1 for 7 2 since 2 1. It follows that the
BIJ Il x +r 27a σ²
Plugging in the choices for À and r, each of the three terms is = (o2 2naaa/(((aat as desired.
Proof of part (b). Here, we have n. the middle term in (14) vanishes. Assume first that 2 1. consider different regularization regimes.
Case --1/a E ni that [n-a, and n main(n,A-110 X-1/a, We have 17 2(r-1)a 12 < 1. It follows that
o² -/1a/- F - X
n
The optimal choice of À is obtained by setting X x (0² /n) x 1/a that is, x= (d2/n)//(20+) and MSE rate x (a²/n) (2a+)), Note that, assuming assumed range [n up to constants.
< o² S this choice of À is within the
Case 2: 1/1/a 1. Then, 7 = and the MSE X2 + 0² /n. Since 0² 2 by assumption and
A > 1, we have MSE X X² > which is always worse than the rate in Case 1.
Case 3: 1/a > n. Then, 77 = n and MSE x X2 + o² x o² since o² ž n 2a by assumption and
X² < 2a This rate is clearly worst than Case 1. Putting the pieces together the optimal rate is achieved by Case I.
Next, consider the case 7 < 1. We again consider the three regularization regimes:
Case -11/a E [1, n] so that [n-9, 1] and r = min(n, X-1/a) = A A-1/a, We have 17 2(1-1)a 12(--1) 1. It follows from (14) that
d² x 11/a FrA 1= x 2y +
n
The optimal choice of À is obtained by setting 2 x (a² ///n)) 1 1/a that is, X (002/n))/((220)) and MSE rate x 220//20+000 Note that, long as n -nr S o² S (which holds by the assumption on x² since 7 < 1), this choice of À is within the assumed range [n 1] up to constants.
Case 2: A 1/o 1. This similar to Case when 7 2 1.
Case 3: 1-1/a > Then, 1 and y/ -2(--1)a > 1, hence MSE =
~ 02 o2
which is clearly worst than the rate in Case 1. Putting the pieces together the optimal rate achieved by Case 1.
10

---

References
[1] Ahmed Alaoui and Michael W. Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In Advances in Neural Information Processing Systems, pages 775-783, 2013.
[2] A. A. Amini. Speccrally-truncateed kernel ridge regression and its free lunch. Electronic Journal of Statistics, 15:1935-7524, 2021.
[3] Arash A. Amini, Richard Baumgartner, and Dai Feng. Target alignment in truncated kernel ridge regression. htttp:://githhi com/
wwt.:/cacaaaaaaaa//aaaaaaa aat//aaamminiii/n waweiiiiiirrrraaettt--e align, 2022.
[4] Francis Bach. Sharp analysis of low-rank kernel matrix approximations. In Conference Learning Theory, pages 185-209, 2013.
[5] P. Bartlett, P. Long, G. Lugosi, and A. Tsigler. Benign overfitting in linear regression. Proceed- ings of the National Academy of Sciences, 117(45):30633-3007)) 2020.
[6] D. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical varianee trade-off. Proceedings of the National Academy of Sciences,
1 116632):1584--15554, 2019.
[7] B. Bordelon, A. Canatar, and C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference Machine Learning, pages 1024-1034, 2020.
[8] A. Canatar, B. Bordelon, and C. Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications, 14)2914), 2021.
[9] Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):311-366 2007.
[10] Corinna Cortes, Mehryar Mohri, and Ameet Talwalkar. On the impact of kernel approximation on learning accuracy. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 133-120, 2010.
[11] Nello Cristianini, John Shawe- Taylor, André Elisseeff, and Jaz Kandola. On kernel-target alignment. In T. Dietterich, S. Becker, and Ghahramani, editors, Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001.
[12] Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34, 2021.
[13] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-986, 2022.
[14] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Resa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[15] Rong Jin, Tianbao Yang. Mehrdad Mahdavi, Yu-Feng Li, and Zhi-Hua Zhou. Improved bounds for the Nyström method with application to kernel classification. IEEE Transactions on Information Theory, (9(1))6693966668 6949, 2013.
[16] George Kimeldorf and Grace Wahba. Some results on Tehebycheffian spline functions. Journal of Mathematical Analysis and Applications, 33(1): 1971.
[17] Dmitry Kobak, Jonathan Lomond, and Benoit Sanches. The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization. The Journal of Machine Learning Research, 21(1):1-16, 2020.

---

[18] Sanjiv Kumar, Mehryar Mohri, and Ameet Talwalkar, Ensemble Nyström method. In Advances in Neural Information Processing Systems, pages 1060-1068, 2009.
[19] Mu Li, James T. Kwok, and Bao-Liang Lu. Making large-seale Nyström approximation possible. In Proceedings of the 27th International Conference Machine Learning, pages 631-638, 2010.
[20] Marco Loog, Tom Viering, Alexander Mey, and David Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 117220):0625-10622, 2020.
[21] Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computa- tional regularization. In Advances in Neural Information Processing Systems, pages 1657-1665, 2015.
[22] B. Schoelkopf and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2001.
[23] Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science Business Media, 2008.
[24] Ameet Talwalkar and Afshin Rostamizadeh. Matrix coherence and the nyström method. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, pages 572-579, 2010.
[25] Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. arXiv preprint WrXiv:2009.14286 2020.
[26] Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer, 2009.
[27] Grace Wahba. Spline Models for Observational Data. SIAM, 1990.
[28] Martin J. Wainwright. High-dimensional Statistics: A Non-asymptotic Viewpoint. Cambridge University Press, 2019.
[29] Larry Wasserman. All of Nonparametric Statistics. Springer Science & Business Media, 2006.
[30] Christopher K. I. Williams and Matthias Seeger. Using the Nyström method speed up kernel machines. In Advances in Neural Information Processing Systems, pages 682-688, 2001.
[31] Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random Fourier features: A theoretical and empirical comparison. In Advances in Neural Information Processing Systems, pages 476-484, 2012.
[32] Yun Yang, Mert Pilanci, and Martin J. Wainwright. Randomized sketches for kernels: Fast and optimal nonparametric regression. The Annals of Statisties, 45(3):991-1023, 2017.
[33] Kai Zhang, Ivor W. Tsang, and James T. Kwok. Improved Nyström low-rank approximation and error analysis. Proceedings of the 25th International Conference on Machine Learning, pages 1232-1239. ACM, 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We adhered to the ethics review guidelines.
2. If you including theoretical results...
12

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes]
See Sec-
tions 2, 3, 4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 7.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A] Our experiments were run to support the theoretical results. See Section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] Our simulation were done on regular laptop.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Hl0l0xPwHaN", "reviewer": "Reviewer_Qioj", "summary": "this submission studies how the alignment between target function and the kernel in logistic kernel regression (KRR) can improve the regression fidelity. Kernel spectral truncation is used to adjust the alignment. The results indicate that: 1) for polynomial alignment, there is a regime that kernel truncation (TKRR) archives the parametric rate O(\\sigma^2/n) which is faster than full KRR; 2) for bandlimited alignment, TKRR exhibits multiple descents. \n", "questions": "The connection to generalization is unclear. It’s mentioned in lines 59-61 that the results can be extended to generalization error. The setting however seems to be a pure inference task, with no training involved and no training/test data defined separately. This needs to be clarified.\nThe relation between \\lambda and r is not clear. It seems that both of them are playing as regularization parameters and not clear if in certain regimes they overlap. A counter plot similar to Fig. 1 & 2 would be useful to clarify this issue for the readership.\nIt would be interesting to discuss connections to to singular value thresholding in the context of  (e.g., nuclear norm based regularization) which truncates certain singular value directions that are less aligned with the signal of interest. \nIn Fig. 2 (left) for noise intensity 0.18 and higher, it seems that the best choice for truncation is r=0. What does that mean?\n", "limitations": "They are properly discussed. \n", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\ntheoretical novelty: this is the first work to study the impact of target alignment for KRR\n\nWeakness\npractical significance: it’s not clear how useful the findings about the alignment and concentration would be in practice. It heavily relies on the spectrum of f^* which is unknown, and no side information is known about it in this setting.  \nwriting needs improvement: while the overall story is clear, this paper involves a lot of notations, where some details can be deferred to the appendix. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B2swk8lUO8Z", "reviewer": "Reviewer_UAiV", "summary": "The manuscript provides an analysis of truncated kernel ridge regression. In particular, the authors show how alignment of the target function influences convergence rates, and identify scenarios in which TKRR exhibits stronger guarantees than ridge regression. In addition, they show examples in which generalisation performance of TKRR demonstrates non-monotonicity and phase transition behaviour in regularisation and truncation parameters.", "questions": "Please, see weaknesses section above and correct/comments on any misunderstandings from my side", "limitations": "Yes, they have", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe main result of the paper is the exact expression for mean squared error of TKRR from which the authors derive two interesting conclusions:\n1. They demonstrate how a non-monotonous behaviour of the generalisation performance with respect to truncation parameter takes place, which provides a new insight into double-descent phenomena  \n2. They derive stronger convergence results for TKRR (compared to canonical KRR) in case of strong alignment between the target and kernel functions, which are new to the best of my knowledge\n\nWeaknesses:\n\n1. All analysis in the paper is performed in the fixed design setting. In the conclusion the authors state that it can be extended to the random design model. However it’s outside of my area of expertise, so I cannot judge how straightforward/possible that extension is, taking into account also that TKRR itself is data-dependent\n2. Theoretical results presented in the paper indicate importance of the noise level for selection of a suitable truncation parameter. This is also supported by simulation results, where figure 2a demonstrates that whether choosing a larger r (already being in decreasing error regime) is a beneficial approach depends on the noise level. With this in mind, I believe the manuscript would benefit from a more thorough discussion of this aspect of the problem, in particular what can be deducted from the theoretical results and what only from the simulations/experiments\n3. I found reading of the manuscript rather strenuous. While in parts it can be attributed to its level of technicality, I would suggest the authors to consider some re-organisation of the material. In particular, I would recommend at least partially incorporating discussion currently presented in Appendix D2 and D3 into the main text, potentially at the expense of the proofs in Section 7.\n4. In line 240, I believe the reference should be to Proposition 2a instead of Theorem 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4V9UQLrP4IX", "reviewer": "Reviewer_6bFj", "summary": "The paper considers kernel ridge regression (KRR)  and its truncated version for estimating a target function contaminated with the i.i.d. noise. Two situations are considered depending on the target alignment (TA) scores, which are the coefficients of projection of the underlying target vector on the eigenvalues of normalized empirical kernel matrix K. 1)TA scores are non-zero in a few consecutive components, in which they sampled as i.i.d. with non-zero mean, 2) TA scores are decaying polynomially as well as eigenvalues of $K$. In the first example, the authors demonstrate possible non-monotonous behavior of the MSE depending on truncation parameter r, and other parameters fixed. In the second one, four performance regimes are considered for truncated and full KRR based on the strength of alignment of the target function with the eigenvectors of $K$. If the target alignment coefficients decay fast, then with the proper truncation and smoothing parameter the MSE achieves a better rate than the full KRR with the parametric rate as the limit in the decay parameter.\n", "questions": "Is the statement about the solution of (6) being not unique correct? \n(6) is a ridge regression problem with a design matrix with $r$ orthogonal columns, $r<n$  (see line 128)?  \nIn the supplementary material (17) has a non-unique solution, at the same time \n$$\\tilde{S}^*_{\\mathbf{x}}(w)(x) = \\frac{1}{\\sqrt{n}}\\sum_{j=1}^n w_j \\tilde{\\mathbb{K}}(x,x_i) = \\frac{1}{\\sqrt{n}} \\sum_{k=1}^r \\mu_k \\phi_k(x) \\sum_{j=1}^n \\phi_k(x_j)  w_j  =  \\sum_{k=1}^r \\mu_k \\phi_k(x)   u_k^{\\top} w.$$\n Multiple solutions in terms of $w$  in (17) come from the fact that $w$ contains an arbitrary vector, that lies in the subspace orthogonal to the one spanning $\\{u_1,\\dots,u_r\\}$, but taking $u_k^{\\top} w$ leaves only one solutions?\n\nIt would be worth commenting the results of the submission in comparison with Tsigler, A., & Bartlett, P. L. (2020). Benign overfitting in ridge regression. arXiv preprint arXiv:2009.14286.\n\n----------------\nI increased the rating after the authors commented on the questions.", "limitations": "The limitations and well addressed in the work. No negative societal impact is anticipated.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is easy to follow and well written. The examples considered serve as a good illustration of the double-descent phenomenon in RKHS and the dependence of generalization performance of the (T)KRR  on properties of RKHS and on how well the underlying function is aligned with the eigenvectors of $K$.\n  \nWeaknesses:\n-  I am not sure I understand section 3.1, see the questions below. \n- The evaluation is illustrative enough, but only uses the artificial dataset. \n\nMinor details:\n- w is used for both noise  in (1) and coefficient vector in (3)\n- is $\\sqrt{N}$ needed in the formula between the lines 117-118?  \n- misprint:  $-1$ in the power of $\\sigma/n$ in 192 \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "12-wvZ6iVLH", "reviewer": "Reviewer_x1n8", "summary": "The paper studies a truncated KRR method and shows how the alignment of the target function with the kernel improves the performance of the TKRR. The paper also sheds light on a situation when the TKRR can achieve parametric rates of convergence - something which is not possible for KRR. Both a theoretical analysis and a simulation study is included in the paper.", "questions": "None.", "limitations": "No potential negative societal impact.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is sufficiently novel and studies a very interesting variant of KRR, namely, TKRR and demonstrates both theoretically and empirically the strength of TKRR, in particular, when the target function is aligned with the kernel function. The paper is well-written and the ideas and the results are presented clearly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept