SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tin Sum Cheng Aurelien Lucchi Ivan Dokmanić
{tinsum.cheng, aurelien.lucchi, ivan.dokmanic}@unibas.ch
Anastasis Kratsios
Department of Mathematics and Statistics,  
McMaster University and Vector Institute,  
kratsioa@mcmaster.ca
David Belius
Faculty of Mathematics and Computer Science  
UniDistance Suisse  
david.belius@cantab.ch
Existing statistical learning guarantees for general kernel regressors often yield loose bounds when used with finite-rank kernels. Yet, finite-rank kernels naturally appear in several machine learning problems, e.g. when fine-tuning a pre-trained deep neural network's last layer to adapt it to a novel task when performing transfer learning. We address this gap for finite-rank kernel ridge regression (KRR) by deriving sharp non-asymptotic upper and lower bounds for the KRR test error of any finite-rank KRR. Our bounds are tighter than previously derived bounds on finite-rank KRR, and unlike comparable results, they also remain valid for any regularization parameters.
Generalization is a central theme in statistical learning theory. The recent renewed interest in kernel methods, especially in Kernel Ridge Regression (KRR), is largely due to the fact that deep neural network (DNN) training can be approximated using kernels under appropriate conditions [19, 3, 10], in which the test error is more tractable analytically and thus enjoys stronger theoretical guarantees. However, many prior results have been derived under conditions incompatible with practical settings. For instance [24, 26, 27, 29] give asymptotic bounds on the KRR test error, which requires the input dimension  $d$  to tend to infinity. In reality, the input dimension of the data set and the target function is typically finite. A technical difficulty to provide a sharp non-asymptotic bound on the KRR test error comes from the infinite dimensionality of the kernel<sup>1</sup>. While this curse of dimensionality may be unavoidable for infinite-rank kernels (at least without additional restrictions), one may likely derive tighter bounds in a finite-rank setting. Therefore, other works [6, 2] focused on a setting where the kernel is of finite-rank, where non-asymptotic bounds and exact formula of test error can be derived.
Since different generalization behaviours are observed depending on whether the rank of the kernel rank  $M$  is smaller than the sample size  $N$ , one typically differentiates between the underparameterized  $(M < N)$  and over-parameterized regime  $(M > N)$ . We focus on the former due to its relevance to several applications in the field of machine learning, including random feature
For a fixed input dimension  $d$  and sample size  $N$ , there exists an (infinite-dimensional) subspace in the feature space in which we cannot control the feature vector.
models [32, 35, 25, 14, 17] such as reservoir computers [14, 15, 16, 11] where all the hidden layers of a deep learning model are randomly generated, and only the final layer is trainable, or when fine-tuning the final layers of pre-trained deep neural networks for transfer learning [4] or in few-shot learning [38]. The practice of only re-training pre-trained deep neural networks final layer [40, 23] is justified by the fact that earlier layers encode general features which are common to similar tasks thus fine-tuning can support's a network's ability to generalize to new tasks [21]. In this case, the network's frozen hidden layers define a feature map into a finite-dimensional RKHS, which induces a finite-rank kernel; however, similar finite-rank kernels are often considered [13] in which features are extracted directly from several hidden layers in a deep pre-trained neural network which is then fed into a trainable linear regressor.
Contribution Our main objective is to provide sharp non-asymptotic upper and lower bounds for the finite-rank KRR test error in the under-parameterized regime. We make the following contributions:
(i) Non-vacuous bound in ridgeless case: In contrast to prior work, our bounds exhibit better accuracy when the ridge parameter  $\lambda \rightarrow 0$ , matching the intuition that a smaller ridge yields a smaller test error;
(ii) Sharp non-asymptotic lower bound: We provide a sharp lower bound of test error, which matches our derived upper bound as the sample size  $N$  increases. In this sense, our bounds are tight;
(iii) Empirical validation: We experimentally validate our results and show our improvement in bounds over [6].
As detailed in Section D, Table 1 compares our results to the available risk-bounds for finite-rank KRR.
Table 1: Comparison of risk-bounds for finite-rank kernel ridge regressors.  $(^{*})$ : The bound for the inconsistent case is implicit. See Section 5 for details.  $(^{**})$ : By decay, we mean the difference between the upper bound and its limit as  $N \to \infty$ .
Assumptions / results	Mohri et al. [30]	Amini et al.[2]	Bach [6]	This paper
Include inconsistent case	✓	X	(✓)*	✓
Bias-variance decomposition	X	✓	✓	✓
Test error high probability upper bound	✓	X	✓	✓
Test error high probability lower bound	X	X	X	✓
Bounds improve with smaller ridge?	X	-	X	✓
Upper bound decay rate**	√ log N/N	-	λ + log N/λN	(λ + 1/N)√ log N/N
Organization of Paper Our paper is organized as follows: Section 2 motivates the importance of the under-parameterized finite-rank KRR. Section 3 introduces the notation and necessary background material required in our results' formulation. Section 4 summarizes our main findings and illustrates their implications via a numerical study. All notation is tabulated in Appendix A for the reader's convenience. All proofs can be found in Appendices B and C, and numerical validation of our theoretical findings in Appendix D.
2 Applications
In this section, we motivate the importance of the under-parameterized finite-rank KRR in practical settings. For readers more interested in our main results, please start from Section 3.
Application: Fine-tuning Pre-Trained Deep Neural Networks For Transfer Learning Consider the transfer-learning problem of fine-tuning the final layer of a pre-trained deep neural network model
$f:\mathbb{R}^d\to \mathbb{R}^D$  so that it can be adapted to a task that is similar to what it was initially trained for. This procedure defines a finite-rank kernel regressor because, for any  $x\in \mathbb{R}^d$ ,  $f$  can be factored as
$$
f (x) = A \phi (x),
$$
$$
\phi (x) = \sigma \bullet W ^ {(L)} \circ \dots \sigma \bullet W ^ {(1)} (x),
$$
where  $A$  is a  $D \times d_{L}$ -matrix,  $\sigma \bullet$  denotes element-wise application of a univariate non-linear function, for  $l = 1, \ldots, L$ ,  $L$  is a positive integer, and  $W^{(l)}: \mathbb{R}^{d_{l-1}} \to \mathbb{R}^{d_{l}}$ ,  $d = d_{0}$ . In pre-training, all parameters defining the affine maps  $W^{(l)}$  in the hidden layers  $\sigma \bullet W^{(L)}, \ldots, \sigma \bullet W^{(1)}$  are simultaneously optimized, while in fine-tuning, only the parameters in the final  $A$  are trained, and the others are frozen. This reduces  $f$  to a finite-rank kernel regressor with finite-rank kernel given for  $x, \tilde{x} \in \mathbb{R}^{d}$  by
$$
K (x, \tilde {x}) \stackrel {\text {d e f .}} {=} \phi (x) ^ {\top} \phi (\tilde {x}).
$$
Stably optimizing  $f$  to the new task requires strict convexity of the KRR problem
$$
\min  _ {A} \frac {1}{N} \sum_ {n = 1} ^ {N} \left(A \phi \left(x _ {n}\right) - y _ {n}\right) ^ {2} + \lambda \| A \| _ {F} ^ {2}, \tag {1}
$$
where the hyperparameter  $\lambda > 0$  ensures strong convexity of the KRR's loss function and where  $\| A \|_F$  denotes the Frobenius norm of A. The unique solution  $\hat{A}$  to (1), determines the optimally trained KRR model  $\hat{f}(x) \stackrel{\mathrm{def.}}{=} \hat{A} \phi(x)$  corresponding to the finite-rank kernel  $K$ .
Application: Random Feature Models Popularized by [32] to enable more efficient kernel computations, random feature model has recently seen a substantial spike in popularity and has been the topic of many theoretical works [20, 27]. Note that random feature models are finite-rank kernel regressors once their features are randomly sampled [32, 35, 25, 14, 11].
Application: General Use We emphasize that, though fine-tuning and random feature models provide simple typical examples of when finite-rank KRR arise in machine learning, there are several other instances where our results apply, e.g. when deriving generalization bounds for infinite-rank kernels by truncation, thereby replacing them by finite-rank kernels; e.g. [27].
We now formally define all concepts and notation used throughout this paper. A complete glossary is found in Appendix A.
3.1 The Training and Testing Data
We fix a (non-empty) input space  $\mathcal{X} \subset \mathbb{R}^d$  and a target function  $\tilde{f}: \mathcal{X} \to \mathbb{R}$  which is to be learned by the KRR from a finite number of i.i.d. (possibly noisy) samples  $\mathbf{Z} \stackrel{\mathrm{def.}}{=} (\mathbf{X}, \mathbf{y}) = ((x_i)_{i=1}^N, (y_i)_{i=1}^N) \in \mathbb{R}^{d \times N} \times \mathbb{R}^N$ . The inputs  $x_i$  are drawn from a sampling distribution  $\rho$  on  $\mathcal{X}$  and outputs are modelled as  $y_i = \tilde{f}(x_i) + \epsilon_i \in \mathbb{R}^{N \times 1}$  for some i.i.d. independent random variable  $\epsilon_i$  which is also independent of the  $\mathbf{X}$ , satisfying  $\mathbb{E}[\epsilon] = 0$  and  $\mathbb{E}[\epsilon^2] \stackrel{\mathrm{def.}}{=} \sigma^2 \geq 0$ . Our analysis is set in the space of all square-integrable "function" with respect to the sampling distribution  $L_2(\rho) \stackrel{\mathrm{def.}}{=} \{f: \mathcal{X} \to \mathbb{R}: \mathbb{E}_{x \sim \rho}[f(x)^2] < \infty\}$ .
We abbreviate  $\mathbf{y} = \tilde{f}(\mathbf{X}) + \epsilon \in \mathbb{R}^{N \times 1}$ , where  $\tilde{f}(\mathbf{X}) = [\tilde{f}(x_i)]_{i=1}^N$  and  $\epsilon = [\epsilon_i]_{i=1}^N$ .
3.2 The Assumption: The Finite-Rank Kernel Ridge Regressor
As in [2, 6], we fix a rank  $M \in \mathbb{N}_{+}$  for the kernel  $K$ .
Definition 3.1 (Finite Rank Kernel). Let  $M$  be a positive integer. A (finite) rank- $M$  kernel  $K$  is a map  $K: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  defined for any  $x, x' \in \mathcal{X}$  by
$$
K (x, x ^ {\prime}) = \sum_ {k = 1} ^ {M} \lambda_ {k} \psi_ {k} (x) \psi_ {k} \left(x ^ {\prime}\right), \tag {2}
$$
where the positive numbers  $\{\lambda_k\}_{k=1}^M$  are called the eigenvalues, and orthonormal functions  $\psi_k \in L_\rho^2$  are called the eigenfunctions.
Remark 3.2 (Eigenvalues are Ordered). Without loss of generality, we will assume that the kernel  $K$ 's eigenvalues are ordered  $\lambda_1 \geq \lambda_2 \geq \dots \geq \lambda_M > 0$ .
We denote by  $\mathcal{H} \stackrel{\mathrm{def.}}{=} \operatorname{span}\{\lambda_1^{-1/2}\psi_1, \ldots, \lambda_M^{-1/2}\psi_M\}$  the reproducing kernel Hilbert space (RKHS) associated to  $K$ . See Appendix B for additional details on kernels and RKHSs.
Together, the kernel  $K$  and a ridge  $\lambda > 0$  define an optimal regressor for the training data  $\mathbf{Z}$ .
Definition 3.3 (Kernel Ridge Regressor (KRR)). Fix a ridge  $\lambda >0$ , the regressor  $f_{\mathbf{Z},\lambda}$  of the finite-rank kernel  $K$  is the (unique) minimizer of
$$
f _ {\mathbf {Z}, \lambda} \stackrel {\text {d e f .}} {=} \min  _ {f \in \mathcal {H}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f \left(x _ {i}\right) - y _ {i}\right) ^ {2} + \lambda \| f \| _ {\mathcal {H}} ^ {2}. \tag {3}
$$
3.3 The Target Function
The only regularity assumed of the target function  $\tilde{f}$  is that it is square-integrable with respect to the sampling distribution  $\rho$ ; i.e.  $\tilde{f} \in L_{\rho}^{2}$ . We typically assume that  $\tilde{f}$  contains strictly more features than can be expressed by the finite-rank kernel  $K$ . Thus,  $\tilde{f}$  can be arbitrarily complicated even if the kernel  $K$  is not. Since  $\{\psi_k\}_{k=1}^M$  is an orthonormal set of  $L_{\rho}^{2}$ , we decompose the target function  $\tilde{f}$  as
$$
\tilde {f} = \underbrace {\sum_ {k = 1} ^ {M} \tilde {\gamma} _ {k} \psi_ {k}} _ {\tilde {f} _ {\leq M}} + \tilde {\gamma} _ {> M} \psi_ {> M}, \tag {4}
$$
for some real numbers  $\tilde{\gamma}_k$ 's and  $\tilde{\gamma}_{>M}$  and for some normal function  $\psi_{>M}$  orthogonal to  $\{\psi_k\}_{k=1}^M$ . We call  $\psi_{>M}$  the orthonormal complement and  $\tilde{\gamma}_{>M}$  the complementary coefficient. The component  $\tilde{f}_{\leq M} \stackrel{\mathrm{def.}}{=} \sum_{k=1}^{M} \tilde{\gamma}_k \psi_k$  of  $\tilde{f}$  is in  $\mathcal{H}$ . For the case  $\tilde{\gamma}_{>M} = 0$ , we call it a consistent case, as the target function  $\tilde{f}$  lies in the hypothesis set  $\mathcal{H}$ , else we call it an inconsistent case.
Alternatively, the orthonormal complement  $\psi_{>M}$  can be understood as some input-dependent noise. Assume we have chosen a suitable finite rank kernel  $K$  with corresponding RKHS  $\mathcal{H}$  such that the target function lies in  $\mathcal{H}$ . For this purpose, we can write the target function as  $\tilde{f}_{\leq M} = \sum_{k=1}^{M} \tilde{\gamma}_k \psi_k$  for some real numbers  $\tilde{\gamma}_k$ 's. Suppose that we sample in a noisy environment; then for each sample input  $x_i$ , the sample output  $y_i$  can be written as
$$
y _ {i} = \underbrace {\tilde {f} _ {\leq M} \left(x _ {i}\right)} _ {\text {t r u e l a b e l}} + \underbrace {\tilde {\gamma} _ {> M} \psi_ {> M} \left(x _ {i}\right)} _ {\text {i n p u t - d e p e n d e n t n o i s e}} + \underbrace {\epsilon_ {i}} _ {\text {i n p u t - i n d e p e n d e n t n o i s e}}. \tag {5}
$$
3.4 Test Error
Next, we introduce the subject of interest of this paper in more detail. Our statistical analysis quantifies the deviation of the learned function from the ground truth of the test error.
Definition 3.4 (KRR Test Error). Fix a sample  $\mathbf{Z} = (\mathbf{X},\tilde{f} (\mathbf{X}) + \epsilon)$  . The finite-rank KRR (3)'s test error is
$$
\mathcal {R} _ {\mathbf {Z}, \lambda} \stackrel {\text {d e f}} {=} \mathbb {E} _ {x, \epsilon} \left[ \left(f _ {\mathbf {Z}, \lambda} (x) - \tilde {f} (x)\right) ^ {2} \right] = \mathbb {E} _ {\epsilon} \left[ \int_ {\mathcal {X}} \left(f _ {\mathbf {Z}, \lambda} (x) - \tilde {f} (x)\right) ^ {2} d \rho (x) \right]. \tag {6}
$$
The analysis of this paper also follows the classical bias-variance decomposition, thus we write
$$
\mathcal {R} _ {\mathbf {Z}, \lambda} = \text {b i a s} + \text {v a r i a n c e},
$$
where bias measures the error when there is no noise in the label, that is,
$$
\operatorname {b i a s} \stackrel {\text {d e f .}} {=} \int_ {\mathcal {X}} \left(f _ {(\mathbf {X}, \tilde {f} (\mathbf {X}))}, \lambda (x) - \tilde {f} (x)\right) ^ {2} d \rho (x), \tag {7}
$$
and variance is defined to be the difference between test error and bias: variance  $\stackrel{\mathrm{def.}}{=} \mathcal{R}_{\mathbf{Z},\lambda} - \text{bias.}$
2This means that  $\int_{\mathcal{X}}\psi_k(x)\psi_l(x)d\rho (x) = \delta_{kl}$  for all  $1\leq k\leq l\leq M$
4 Main Result
We now present the informal version of our main result, which gives high-probability upper and lower bounds on the test error. This result is obtained by bounding both the bias and variance, and the probability that the bounds hold is quantified with respect to the sampling distribution  $\rho$ . Here we emphasize the condition that  $N > M$  and hence our statement is valid only in under-parametrized case.
We can assume the data-generating distribution  $\rho$  and eigenfunctions  $\psi_{k}$  are well-behaved in the sense that:
Assumption 4.1 (Sub-Gaussian-ness). We assume that the probability distribution of the random variable  $\psi_{k}(x)$ , where  $x\in \rho$ , has sub-Gaussian norm bounded by a positive constant  $G > 0$ , for all  $k\in \{1,\dots,M\} \cup \{>M\}^3$ .
In particular, if the random variable  $\psi_k(x)$  is bounded, the assumption 4.1 is fulfilled.
Theorem 4.2 (High Probability Bounds on Bias and Variance). Suppose Assumption 4.1 holds, for  $N > M$  sufficient large, there exists some constants  $C_1, C_2$  independent to  $N$  and  $\lambda$  such that, with a probability of at least  $1 - 2/N$  w.r.t. random sampling, we have the following results simultaneously:
(i) Upper-Bound on Bias: The bias is upper bounded by:
$$
\operatorname {b i a s} \leq \tilde {\gamma} _ {> M} ^ {2} + \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + 2 \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} + C _ {1} \frac {\log N}{N}, \tag {8}
$$
where we denote  $\tilde{f}_{\leq M} \stackrel{\mathrm{def.}}{=} \sum_{k=1}^{M} \tilde{\gamma}_k \psi_k = \tilde{f} - \tilde{\gamma}_{>M} \psi_{>M}$ .
(ii) Lower-Bound on Bias: The bias is lower bounded by an analogous result:
$$
\text {b i a s} \geq \tilde {\gamma} _ {> M} ^ {2} + \frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} - \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + \frac {2 \lambda^ {2}}{\lambda_ {1} + \lambda} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} - C _ {1} \frac {\log N}{N}. \tag {9}
$$
(iii) Upper-Bound on Variance: The variance is upper bounded by:
$$
\text {v a r i a n c e} \leq \sigma^ {2} \frac {M}{N} \left(1 + \sqrt {\frac {\log N}{N}} + C _ {2} \frac {\log N}{N}\right); \tag {10}
$$
(iv) Lower-Bound on Variance: The variance is lower bounded by an analogous result:
$$
\text {v a r i a n c e} \geq \frac {\lambda_ {M} ^ {2}}{(\lambda_ {M} + \lambda) ^ {2}} \sigma^ {2} \frac {M}{N} \left(1 - \sqrt {\frac {\log N}{N}}\right) - C _ {2} \sigma^ {2} \frac {M}{N} \frac {\log N}{N}. \tag {11}
$$
For  $\lambda \to 0$ , we have a simpler bound on the bias: with a probability of at least  $1 - 2 / N$ , we have
$$
\lim  _ {\lambda \rightarrow 0} b i a s \leq \tilde {\gamma} _ {> M} ^ {2} \left(1 + \frac {\log N}{N}\right) + 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}}; \tag {12}
$$
$$
\lim _ {\lambda \to 0} b i a s \geq \tilde {\gamma} _ {> M} ^ {2} \left(1 - \frac {\log N}{N}\right) - 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}}.
$$
If  $\tilde{\gamma}_{>M}^2 = 0$ , then we are in the consistent case, meaning that  $\tilde{f}$  belongs to  $\mathcal{H}$ . In this case, we have a simpler bound on the bias: with a probability of at least  $1 - 2/N$ , we have
$$
\text {b i a s} \leq \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2} \left(1 + 2 \sqrt {\frac {\log N}{N}}\right) + C _ {1} \frac {\log N}{N}. \tag {13}
$$
it means the orthonormal complement  $\psi_{>M}$  is also mentioned in the assumption.
Proof Sketch: The main technical tools are 1) more careful algebraic manipulations when dealing with terms involving the regularizer  $\lambda$  and 2) the use of a concentration result for a sub-Gaussian random covariance matrix in [37] followed by the Neumann series expansion of a matrix inverse. Hence, unlike previous work, our result holds for any  $\lambda$ , which can be chosen independence to  $N$ . The complete proof of this result, together with the explicit form of the lower bound, is presented in Theorems C.19 and C.20 in the Appendix C.
Remark 4.3. Note that one can also easily derive bounds on the expected value of the test error to allow comparisons with prior works, such as [6, 36, 33].
Remark 4.4. The main technical difference from prior works is that we consider the basis  $\{\psi_k\}_{k=1}^M$  in the function space  $L_{\rho}^{2}$  instead of  $\{\lambda_k^{-1/2}\psi_k\}_{k=1}^M$  in the RKHS  $\mathcal{H}$ . This way, we exploit decouple the effect of spectrum from the sampling randomness to obtain a sharper bound. For further details, see Remark C.12 in Appendix C.
Combining Theorem 4.2 and the bias-variance decomposition in Definition 3.4, we have both the upper and lower bounds of the test error on KRR.
Corollary 4.4.1. Under mild conditions on the kernel  $K$ , for  $N$  sufficiently large, there exist some constants  $C_1, C_2$  independent of  $N$  and  $\lambda$  such that, with a probability of at least  $1 - 2/N$  w.r.t. random sampling, we have the bounds on the test error  $\mathcal{R}_{\mathbf{Z},\lambda}$ :
$$
\begin{array}{l} \mathcal {R} _ {\mathbf {Z}, \lambda} \leq \tilde {\gamma} _ {> M} ^ {2} + \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + 2 \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} + C _ {1} \frac {\log N}{N} \\ + \sigma^ {2} \frac {M}{N} \left(1 + \sqrt {\frac {\log N}{N}} + C _ {2} \frac {\log N}{N}\right); \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {R} _ {\mathbf {Z}, \lambda} \geq \tilde {\gamma} _ {> M} ^ {2} + \frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {k} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} - \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + \frac {2 \lambda^ {2}}{\lambda_ {1} + \lambda} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} - C _ {1} \frac {\log N}{N} \\ + \sigma^ {2} \frac {M}{N} \left(1 - \sqrt {\frac {\log N}{N}} - C _ {2} \frac {\log N}{N}\right). \\ \end{array}
$$
In particular, we have:
$$
\lim  _ {\lambda \rightarrow 0} \mathcal {R} _ {\mathbf {Z}, \lambda} \leq \left(1 + \frac {\log N}{N}\right) \tilde {\gamma} _ {> M} ^ {2} + 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}} + \sigma^ {2} \frac {M}{N} \left(1 + \sqrt {\frac {\log N}{N}} + C _ {2} \frac {\log N}{N}\right).
$$
The corresponding lower bounds are given analogously:
$$
\lim _ {\lambda \to 0} \mathcal {R} _ {\mathbf {Z}, \lambda} \geq \left(1 - \frac {\log N}{N}\right) \tilde {\gamma} _ {> M} ^ {2} - 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}} + \sigma^ {2} \frac {M}{N} \left(1 - \sqrt {\frac {\log N}{N}} - C _ {2} \frac {\log N}{N}\right).
$$
See Section 5 for more details and validations of Theorem 4.2.
In this section, we first elaborate on the result from Theorem 4.2, which we then compare in detail to prior works, showcasing the improvements this paper makes. Finally, we discuss several future research directions.
5.1 Elaboration on Main Result
Bias From Eq. (8), we can draw the following observations: 1) The term  $\tilde{\gamma}_{>M}^2$  in the upper bound is the finite rank error due to the inconsistency between RKHS and the orthogonal complement of the target function, which cannot be improved no matter what sample size we have. We can also view this term as the sample-dependent noise variance (see Eq. (5)). Hence, unlike the sample-independent noise variance  $\sigma$  in Eq. (10), the sample-dependent noise variance does not vanish when  $N\to \infty$ . 2) Note that the third term  $\frac{1}{4}\| \tilde{f}\|_{L_\rho^2}^2\sqrt{\frac{\log N}{N}}$  is a residue term proportional to  $\| \tilde{f}\|_{L_\rho^2}^2$  and vanishes
when  $N\to \infty$  , which means we have better control of the sample-dependent noise around its expected value for large  $N$  . Also, note that the factor  $\| \tilde{f}\|_{L_\rho^2}^2 = \sum_{k = 1}^{M}\tilde{\gamma}_k^2 +\tilde{\gamma}_{>M}^2$  depends solely on the target function  $\tilde{f}$  but not on the kernel  $K$  or ridge parameter  $\lambda$  . 3) The second plus the fourth terms  $\left(1 + 2\sqrt{\frac{\log N}{N}}\right)\lambda \| \tilde{f}_{\leq M}\|_{\mathcal{H}}^2$  depends strongly on the kernel training: the sum is proportional to the ridge  $\lambda$  , and the RKHS norm square  $\| \tilde{f}_{\leq M}\|_{\mathcal{H}}^2 = \sum_{k = 1}^{M}\frac{\tilde{\gamma}_k^2}{\lambda_k}$  measures how well-aligned the target functions with the chosen kernel  $K$  is. Again, a larger  $N$  favors the control as the residue  $\sqrt{\frac{\log N}{N}}\rightarrow 0$  as  $N\rightarrow \infty$  . 4) The fifth term  $C_1\frac{\log N}{N}$  is a small residue term with fast decay rate, which the other terms will overshadow as  $N\rightarrow \infty$