SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 3.5. Theorem 3.4 shows that the generalization error of KRR in the noiseless case is monotone decreasing when  $\theta$  increases and reaches the optimal rate  $n^{-\beta}$  when  $\theta \geq \beta$  if  $s \leq 2$ . Since the case  $\theta \to \infty$  corresponds to kernel interpolation, our result implies that kernel interpolation is optimal when there is no noise. In contrast, as shown in Theorem 3.2 (or Li et al. [2023a]), kernel interpolation can not generalize in the noisy case. For the case  $s > 2$ , the KRR method suffers from saturation and the resulting convergence rate is limited to  $n^{-2\beta}$ , while the possible lower rate is  $n^{-s\beta}$ .
Our results provide a full picture of the generalization of KRR, which is in accordance with our traditional belief of the bias-variance trade-off principle: the generalization error is a U-shaped curve with respect to the regularization parameter  $\lambda$  in the noisy case and is monotone decreasing in the noiseless case. See Figure 1 on page 6 for an illustration.
Our rates coincide with the upper rates in the traditional KRR literature [Caponnetto and De Vito, 2007, Fischer and Steinwart, 2020]. Moreover, our results also recover the learning curves in Cui et al. [2021], but we do not need the strong assumption of Gaussian design eigenfunctions as in Cui et al. [2021], which may not be true in most cases. Our assumptions are mild and hold for a large class of kernels including the Sobolev kernels and the neural tangent kernels (NTK) on spheres.
Our results are based on the bias-variance decomposition and determining the rates for each term respectively. In the proof of Li et al. [2023b], they determined the rates of the variance term under the condition that  $\theta < \frac{1}{2}$  and that of the bias term when  $s \geq 2$  and  $\theta < 1$ . The subsequent work Li et al. [2023a] proved the rates of the variance term when  $\theta < \beta$  and provided a near constant lower bound for  $\theta \geq \beta$ . Considering the counterpart, our works further prove the rates of the bias term, which finally enables us to determine the complete learning curve of KRR.
The connection between KRR and Gaussian process regression also results in the connection between their learning curves. Jin et al. [2021] claimed to show learning curves for Gaussian process regression. However, regardless of the gap in their proof as pointed out in Zhang et al. [2023b], their results are more restrictive than ours. Considering a boundedness assumption of the eigenfunctions that  $\| e_i\|_{\infty}\leq Ci^{\tau}$  for some  $\tau \geq 0$ , they could only cover the regime of  $\theta < \beta /(1 + 2\tau)$ . Moreover, to approach the  $\theta = \beta$  regime for the  $\Omega (1)$  bound in the noisy case or the optimal rate in noiseless case, they have to require  $\tau = 0$ , that is, the eigenfunctions are uniformly bounded, but it is not true for some kernels such as dot-product kernels on spheres (and thus for NTK) since in general spherical harmonics are not uniformly bounded. In contrast, our embedding index assumption still holds in this case.
4 Proof sketch
We first introduce the following sample versions of the auxiliary integral operators, which are commonly used in the related literature [Caponnetto and De Vito, 2007, Fischer and Steinwart, 2020, Li et al., 2023b]. We define the sampling operator  $K_{x}:\mathbb{R}\rightarrow \mathcal{H}$  by  $K_{x}y = yk(x,\cdot)$ , whose adjoint  $K_{x}^{*}:\mathcal{H}\to \mathbb{R}$  is given by  $K_{x}^{*}f = f(x)$ . The sample covariance operator  $T_{X}:\mathcal{H}\to \mathcal{H}$  is defined by
$$
T _ {X} := \frac {1}{n} \sum_ {i = 1} ^ {n} K _ {x _ {i}} K _ {x _ {i}} ^ {*}, \tag {19}
$$
and the sample basis function is  $g_Z \coloneqq \frac{1}{n} \sum_{i=1}^{n} K_{x_i} y_i \in \mathcal{H}$ . As shown in Caponnetto and De Vito [2007], the operator form of KRR writes
$$
\hat {f} _ {\lambda} = \left(T _ {X} + \lambda\right) ^ {- 1} g _ {Z}. \tag {20}
$$
Let us further define
$$
\tilde {g} _ {Z} := \mathbb {E} \left(g _ {Z} | X\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} K _ {x _ {i}} f _ {\rho} ^ {*} \left(x _ {i}\right) \in \mathcal {H}, \tag {21}
$$
$$
\tilde {f} _ {\lambda} := \mathbb {E} \left(\hat {f} _ {\lambda} | X\right) = \left(T _ {X} + \lambda\right) ^ {- 1} \tilde {g} _ {Z} \in \mathcal {H}. \tag {22}
$$
Then, the traditional bias-variance decomposition [Li et al., 2023b, Zhang et al., 2023a] yields
$$
\mathbb {E} \left(\left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2} \mid X\right) = \operatorname {B i a s} ^ {2} (\lambda) + \operatorname {V a r} (\lambda), \tag {23}
$$
$$
\mathbf {B i a s} ^ {2} (\lambda) := \left\| \tilde {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2}, \quad \mathbf {V a r} (\lambda) := \frac {\sigma^ {2}}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \left(T _ {X} + \lambda\right) ^ {- 1} k \left(x _ {i}, \cdot\right) \right\| _ {L ^ {2}} ^ {2}. \tag {24}
$$
4.1 The noisy case
To prove the desired result, we have to establish the asymptotic orders of both  $\mathbf{Bias}^2 (\lambda)$  and  $\mathbf{Var}(\lambda)$ . We first prove the asymptotic order of  $\mathbf{Bias}^2 (\lambda)$  as one of our technical contributions. As far as we know, we are the first to provide such a lower bound in (25).
Lemma 4.1. Under Assumptions 1,2,3, suppose  $\lambda \asymp n^{-\theta}$  for  $\theta \in (0,\beta)$ . Then,
$$
\mathbf {B i a s} ^ {2} (\lambda) = \tilde {\Theta} _ {\mathbb {P}} \left(n ^ {- \min  (s, 2) \theta}\right), \tag {25}
$$
Proof sketch of Lemma 4.1. Denote  $\tilde{s} = \min (s,2)$ . We first introduce the regularized regression function  $f_{\lambda}\coloneqq T(T + \lambda)^{-1}f_{\rho}^{*}$  and triangle inequality implies
$$
\mathbf {B i a s} (\lambda) = \left\| \tilde {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} \geq \left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} - \left\| \tilde {f} _ {\lambda} - f _ {\lambda} \right\| _ {L ^ {2}}.
$$
There is no randomness in the first term and we can use the expansion (12) and (5) to show that  $\left\| f_{\lambda} - f_{\rho}^{*}\right\|_{L^2} = \tilde{\Theta}\left(n^{-\tilde{s}\theta}\right)$ . Then, we have to prove the error term  $\left\| \tilde{f}_{\lambda} - f_{\lambda}\right\|_{L^2}$  to be infinitesimal with respect to the main term, which is the main difficulty since it requires a refined analysis. Previous work only considers the case  $\theta = \frac{\beta}{\tilde{s}\beta + 1}$  (corresponding to the optimal regularization) and show an  $O(n^{-\tilde{s}\theta})$  bound rather than the  $o(n^{-\tilde{s}\theta})$  bound that we require. For the proof, we (1) apply the concentration techniques in Fischer and Steinwart [2020]; (2) consider the  $L^q$ -embedding property in Zhang et al. [2023a] for the mis-specified case when  $s$  is small; (3) sharpen the estimation by exploiting the embedding property  $\alpha_0 = 1 / \beta$  and  $\theta < \beta$ . For the detail, see Section 2.2 in the supplementary material.
The variance term has been analyzed in Li et al. [2023a]. We present the following proposition as a combination of Proposition 5.3 and Theorem 5.10 in Li et al. [2023a].
Proposition 4.2. Under Assumptions 1-5, suppose that  $\lambda \asymp n^{-\theta}$ . Then,
$$
\mathbf {V a r} (\lambda) = \left\{ \begin{array}{l l} \Theta_ {\mathbb {P}} ^ {\text {p o l y}} \left(\sigma^ {2} n ^ {- (1 - \theta / \beta)}\right), & \text {i f} \quad \theta <   \beta ; \\ \Omega_ {\mathbb {P}} ^ {\text {p o l y}} \left(\sigma^ {2}\right), & \text {i f} \quad \theta \geq \beta . \end{array} \right. \tag {26}
$$
4.2 The noiseless case
For the noiseless case, the variance term vanishes in (23), and thus we only need to consider the bias term. Since we have already established the estimation for large  $\lambda$  in Lemma 4.1, we focus on the case of small  $\lambda$ .
Lemma 4.3. Under Assumptions 1,2,3, assume further  $s > 1$ . Suppose  $\lambda \asymp n^{-\theta}$  for  $\theta \geq \beta$ . Then,
$$
\operatorname {B i a s} ^ {2} (\lambda) = O _ {\mathbb {P}} ^ {\text {p o l y}} \left(n ^ {- \min  (s, 2) \beta}\right). \tag {27}
$$
Proof sketch of Lemma A.12. Intuitively, we hope to bound  $\mathbf{Bias}^2 (\lambda)$  with  $\mathbf{Bias}^2 (\tilde{\lambda})$  for  $\tilde{\lambda} >\lambda$  such that concentration still works. However, we can not directly derive no monotone property of  $\mathbf{Bias}(\lambda)$ . Nevertheless, since  $f_{\rho}^{*}\in \mathcal{H}$  when  $s > 1$ , the bias term can be written as
$\mathbf{Bias}(\lambda) = \left\| \lambda (T_X + \lambda)^{-1}f_\rho^*\right\|_{L^2} = \left\| T^{\frac{1}{2}}\lambda (T_X + \lambda)^{-1}f_\rho^*\right\|_{\mathcal{H}}\leq \left\| T^{\frac{1}{2}}\lambda (T_X + \lambda)^{-1}\right\|_{\mathcal{B}(\mathcal{H})}\left\| f_\rho^*\right\|_{\mathcal{H}}.$
Then, by operator calculus we can show that
$$
\left\| T ^ {s} \left[ \lambda (T _ {X} + \lambda) ^ {- 1} \right] \right\| _ {\mathcal {B} (\mathcal {H})} \leq \left\| T ^ {s} \left[ \tilde {\lambda} (T _ {X} + \tilde {\lambda}) ^ {- 1} \right] \right\| _ {\mathcal {B} (\mathcal {H})}
$$
reducing  $\lambda$  to  $\tilde{\lambda}$ . Now, we can replace  $T_{X}$  with  $T$  using concentration results and derive the desired upper bound.
The following proposition shows that the upper bound in Lemma A.12 matches the information-theoretical lower bound. The proof follows the idea of the minimax principle [Micchelli and Wahba, 1979] and is deferred to the supplementary material.
Proposition 4.4. Suppose Assumption 1 holds and  $s \geq 1$ . For any  $X = (x_{1}, \ldots, x_{n})$ , we have
$$
\sup  _ {\left\| f _ {\rho} ^ {*} \right\| _ {[ \mathcal {H} ] ^ {s}} \leq R} \mathbf {B i a s} ^ {2} (\lambda) = \Omega (n ^ {- s \beta}), \tag {28}
$$
where we note that here  $\mathbf{Bias}(\lambda)$  is viewed as a function depending also on  $f_{\rho}^{*}$  and  $X$ .
Lots of numerical experiments on both synthetic data and real data are done to study to learning curves of KRR [Li et al., 2023b, Cui et al., 2021]. In this section, we consider numerical experiments on a toy model to verify our theory.
Let us consider the kernel  $k(x,y) = \min (x,y)$  and  $x\sim \mathcal{U}[0,1]$ . Then, the corresponding RKHS is [Wainwright, 2019]
$$
\mathcal {H} = \left\{f: [ 0, 1 ] \rightarrow \mathbb {R} \mid f \text {i s a b s o l u t e l y c o n t i n u o u s}, f (0) = 0, \int_ {0} ^ {1} \left(f ^ {\prime} (x)\right) ^ {2} d x <   \infty \right\}
$$
and the eigenvalue decay rate  $\beta = 2$ . Moreover, the eigensystem of  $k$  is known to be  $\lambda_{i} = \left(\frac{2i - 1}{2}\pi\right)^{-2}$  and  $e_i(x) = \sqrt{2}\sin \left(\frac{2i - 1}{2}\pi x\right)$ , which allows us to directly compute the smoothness of certain functions. For some  $f^{*}$ , we generate data from the model  $y = f^{*}(x) + \varepsilon$  where  $\varepsilon \sim \mathcal{N}(0,0.05)$  and perform KRR with  $\lambda = cn^{-\theta}$  for different  $\theta$ 's with some fixed constant  $c$ . Then, we numerically compute the variance, bias and excess risk by Simpson's formula with  $N \gg n$  nodes. Repeating the experiment for  $n$  ranged in 1000 to 5000, we can estimate the convergence rate  $r$  by a logarithmic least-squares log err  $= r\log n + b$  on the values (variance, bias and excess risk). The results are collected in Table 1 on page 10. It can be seen that the resulting values basically match the theoretical values and we conclude that our theory is supported by the experiments. For more experiments and more details, we refer to the supplementary material.
f*(x) =	cos 2πx (s = 1/2)	sin 2πx (s = 1.5)	sin 3/2πx (s = ∞)
θ	Variance	Bias	Risk	Bias	Risk	Bias	Risk
0.2	0.90 (0.90)	0.13 (0.10)	0.13 (0.10)	0.34 (0.30)	0.34 (0.30)	0.40 (0.40)	0.42 (0.40)
0.4	0.80 (0.80)	0.22 (0.20)	0.22 (0.20)	0.68 (0.60)	0.69 (0.60)	0.82 (0.80)	0.81 (0.80)
0.5	0.75 (0.75)	0.26 (0.25)	0.26 (0.25)	0.84 (0.75)	0.79 (0.75)	1.04 (1.00)	0.77 (0.75)
1.0	0.49 (0.50)	0.54 (0.50)	0.52 (0.50)	1.69 (1.50)	0.49 (0.50)	2.21 (2.00)	0.49 (0.50)
2.0	0.00 (0.00)	1.05 (1.00)	0.09 (0.00)	3.26 (3.00)	0.00 (0.00)	3.99 (4.00)	0.00 (0.00)
3.0	0.00 (0.00)	1.05 (1.00)	0.09 (0.00)	3.26 (3.00)	0.00 (0.00)	3.98 (4.00)	0.00 (0.00)
Table 1: Asymptotic rates of bias, variance and excess risk under three regressions and different choices of  $\theta$ . The numbers in parenthesis are the theoretical values. The bolded cells correspond to the best rate over the choices of  $\theta$ 's.
In this paper, we prove rigorously the learning curves of KRR, showing the interplay of the eigenvalue decay of the kernel, the relative smoothness of the regression function, the noise and the choice of the regularization parameter. The results justify our traditional bias-variance trade-off principle and provide a full picture of the generalization performance of KRR. These results will help us better understand the generalization mystery of neural networks.
As for future works, we notice that for the nearly interpolating regime when  $\theta \geq \beta$ , there are still some missing parts due to technical limitations. We expect that further analysis will prove the exact orders of the variance term like that given in Mallinar et al. [2022] under the Gaussian design assumption. We also hypothesize that Lemma A.12 still holds in the mis-specified case ( $s < 1$ ).
This work is supported in part by the Beijing Natural Science Foundation (Grant Z190001) and National Natural Science Foundation of China (Grant 11971257).
Ingo Steinwart (auth.) Andreas Christmann. Support Vector Machines. Information Science and Statistics. Springer-Verlag New York, New York, NY, 1 edition, 2008. ISBN 0-387-77242-1 0-387-77241-3 978-0-387-77241-7 978-0-387-77242-4. doi: 10.1007/978-0-387-77242-4.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063-30070, December 2020. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1907378117.
Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit. Kernel ridgeless regression is inconsistent in low dimensions. (arXiv:2205.13525), June 2022. doi: 10.48550/arXiv.2205.13525.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.
Alberto Bietti and Francis Bach. Deep equals shallow for ReLU networks in kernel regimes. arXiv preprint arXiv:2009.14397, 2020.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems, volume 32, 2019.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, pages 1024-1034. PMLR, November 2020.
Barry Simon. *Operator Theory*. American Mathematical Society, Providence, Rhode Island, November 2015. ISBN 978-1-4704-1103-9 978-1-4704-2763-4. doi: 10.1090/simon/004.
Ingo Steinwart and C. Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation, 35(3):363-417, 2012. doi: 10.1007/S00365-012-9153-3.
Ingo Steinwart, D. Hush, and C. Scovel. Optimal rates for regularized least squares regression. In  $\text{COLT}$ , pages 79-93, 2009.
Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer science & business media, 1999.
Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627771.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. (arXiv:1611.03530), February 2017.
Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms. (arXiv:2303.14942), March 2023a. doi: 10.48550/arXiv.2303.14942.
Haobo Zhang, Yicheng Li, Weihao Lu, and Qian Lin. On the optimality of misspecified kernel ridge regression. In International Conference on Machine Learning, 2023b.
The first step of the proof is the traditional bias-variance decomposition. Let us further define
$$
\tilde {g} _ {Z} := \mathbb {E} \left(g _ {Z} | X\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} K _ {x _ {i}} f _ {\rho} ^ {*} \left(x _ {i}\right) \in \mathcal {H}, \tag {29}
$$
$$
\tilde {f} _ {\lambda} := \mathbb {E} \left(\hat {f} _ {\lambda} | X\right) = \left(T _ {X} + \lambda\right) ^ {- 1} \tilde {g} _ {Z} \in \mathcal {H}. \tag {30}
$$
Recalling (20), we have
$$
\begin{array}{l} \hat {f} _ {\lambda} = \frac {1}{n} (T _ {X} + \lambda) ^ {- 1} \sum_ {i = 1} ^ {n} K _ {x _ {i}} y _ {i} = \frac {1}{n} (T _ {X} + \lambda) ^ {- 1} \sum_ {i = 1} ^ {n} K _ {x _ {i}} \left(f _ {\rho} ^ {*} \left(x _ {i}\right) + \epsilon_ {i}\right) \\ = (T _ {X} + \lambda) ^ {- 1} \tilde {g} _ {Z} + \frac {1}{n} \sum_ {i = 1} ^ {n} (T _ {X} + \lambda) ^ {- 1} K _ {x _ {i}} \epsilon_ {i}, \\ \end{array}
$$
$$
\hat {f} _ {\lambda} - f _ {\rho} ^ {*} = \left(\tilde {f} _ {\lambda} - f _ {\rho} ^ {*}\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} (T _ {X} + \lambda) ^ {- 1} K _ {x _ {i}} \epsilon_ {i}.
$$
Taking expectation over the noise  $\epsilon$  conditioned on  $X$ , since  $\varepsilon |x$  are independent noise with mean 0 and variance  $\sigma^2$ , we have
$$
\mathbb {E} \left(\left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2} \mid X\right) = \operatorname {B i a s} ^ {2} (\lambda) + \operatorname {V a r} (\lambda), \tag {31}
$$
$$
\mathbf {B i a s} ^ {2} (\lambda) := \left\| \tilde {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2}, \quad \mathbf {V a r} (\lambda) := \frac {\sigma^ {2}}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \left(T _ {X} + \lambda\right) ^ {- 1} k \left(x _ {i}, \cdot\right) \right\| _ {L ^ {2}} ^ {2}. \tag {32}
$$
A.1 The variance term
Theorem A.1. Under Assumptions 1-5, suppose that  $\lambda \asymp n^{-\theta}$ . Then,
$$
\mathbf {V a r} (\lambda) = \left\{ \begin{array}{l l} \Theta_ {\mathbb {P}} ^ {\text {p o l y}} \left(\sigma^ {2} n ^ {- (1 - \theta / \beta)}\right), & i f \quad \theta <   \beta ; \\ \Omega_ {\mathbb {P}} ^ {\text {p o l y}} \left(\sigma^ {2}\right), & i f \quad \theta \geq \beta . \end{array} \right. \tag {33}
$$
The computation in Li et al. [2023b] shows that
$$
\mathbf {V a r} (\lambda) = \frac {\sigma^ {2}}{n ^ {2}} \int_ {\mathcal {X}} \mathbb {K} (x, X) (K + \lambda) ^ {- 2} \mathbb {K} (X, x) \mathrm {d} \mu (x).
$$
Then, Theorem A.1 directly follows from Proposition 5.3 and Theorem 5.10 in Li et al. [2023a].
A.2 The bias term
Theorem A.2. Under Assumptions 1,2,3, suppose  $\lambda \asymp n^{-\theta}$  for  $\theta \in (0,\beta)$ . Then,
$$
\mathbf {B i a s} ^ {2} (\lambda) = \tilde {\Theta} _ {\mathbb {P}} \left(n ^ {- \min  (s, 2) \theta}\right), \tag {34}
$$
Let us define the regularized version of the regression function
$$
f _ {\lambda} := (T + \lambda) ^ {- 1} T f _ {\rho} ^ {*}. \tag {35}
$$
Then, the triangle inequality implies that
$$
\mathbf {B i a s} (\lambda) = \left\| \tilde {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} \geq \left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} - \left\| \tilde {f} _ {\lambda} - f _ {\lambda} \right\| _ {L ^ {2}} \tag {36}
$$
Then, the proof of Theorem A.2 is the combination of the following Lemma A.3 (with  $\gamma = 0$ ) and Lemma A.4, showing that the main term  $\left\| f_{\lambda} - f_{\rho}^{*}\right\|_{L^2} = \tilde{\Theta}_{\mathbb{P}}\left(n^{-\min (s,2)\theta /2}\right)$  and the error term  $\left\| \tilde{f}_\lambda -f_\lambda \right\|_{L^2} = o_{\mathbb{P}}\left(n^{-\min (s,2)\theta /2}\right)$ .
Lemma A.3. Under Assumptions 1 and 3, for any  $0 \leq \gamma < s$ , we have
$$
\left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {[ \mathcal {H} ] ^ {\gamma}} ^ {2} \asymp \left\{ \begin{array}{l l} \lambda^ {s - \gamma}, & s - \gamma <   2; \\ \lambda^ {2} \ln \frac {1}{\lambda}, & s - \gamma = 2; \\ \lambda^ {2}, & s - \gamma > 2. \end{array} \right. \tag {37}
$$
Proof. From the definition of interpolating norms, letting  $p = (s - \gamma) / 2$ , we have
$$
\left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {[ \mathcal {H} ] ^ {\gamma}} ^ {2} = \sum_ {i = 1} ^ {\infty} a _ {i} ^ {2} \frac {\lambda^ {2}}{(\lambda_ {i} + \lambda) ^ {2}} \left(\lambda_ {i} ^ {s} i ^ {- 1}\right) \lambda_ {i} ^ {- \gamma} \asymp \lambda^ {2} \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1}. \tag {38}
$$
Then result then follows by applying Proposition B.2 for the last series.
The following lemma shows the error term in (36) is infinitesimal, whose proof relies on fine-grained concentration results established in Section A.3.
Lemma A.4. Under Assumptions 1-3. Suppose  $\lambda \asymp n^{-\theta}$  with  $\theta \in (0,\beta)$ , then
$$
\left\| \tilde {f} _ {\lambda} - f _ {\lambda} \right\| _ {L ^ {2}} = o _ {\mathbb {P}} \left(n ^ {- \min  (s, 2) \theta / 2}\right) \tag {39}
$$
Proof. We begin with
$$
\begin{array}{l} \left\| \tilde {f} _ {\lambda} - f _ {\lambda} \right\| _ {L ^ {2}} = \left\| T ^ {\frac {1}{2}} \left(\tilde {f} _ {\lambda} - f _ {\lambda}\right) \right\| _ {\mathcal {H}} \\ \leq \left\| T ^ {\frac {1}{2}} T _ {\lambda} ^ {- \frac {1}{2}} \right\| \cdot \left\| T _ {\lambda} ^ {\frac {1}{2}} T _ {X \lambda} ^ {- 1} T _ {\lambda} ^ {\frac {1}{2}} \right\| \cdot \left\| T _ {\lambda} ^ {- \frac {1}{2}} \left(\tilde {g} _ {Z} - T _ {X \lambda} f _ {\lambda}\right) \right\| _ {\mathcal {H}}. \tag {40} \\ \end{array}
$$
From operator calculus we know  $\left\| T^{\frac{1}{2}}T_{\lambda}^{-\frac{1}{2}}\right\| \leq 1$ . Moreover, since  $\theta < \beta$  and the embedding index  $\alpha_0 = 1 / \beta$ , by Lemma B.5 we get  $\left\| T_{\lambda}^{\frac{1}{2}}T_{X\lambda}^{-1}T_{\lambda}^{\frac{1}{2}}\right\| \leq 3$  with high probability as long as  $n$  is sufficiently large. For the last term in (40), we have
$$
\begin{array}{l} T _ {\lambda} ^ {- \frac {1}{2}} \left(\tilde {g} _ {Z} - T _ {X \lambda} f _ {\lambda}\right) = T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - \left(T _ {X} + \lambda + T - T\right) f _ {\lambda}\right) \right] \\ = T _ {\lambda} ^ {- \frac {1}{2}} \left[ (\tilde {g} _ {Z} - T _ {X} f _ {\lambda}) - (T + \lambda) f _ {\lambda} + T f _ {\lambda} \right] \\ = T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - T _ {X} f _ {\lambda}\right) - \left(g - T f _ {\lambda}\right) \right]. \\ \end{array}
$$
Therefore, Lemma A.5 and Lemma A.10 show that
$$
\left\| T _ {\lambda} ^ {- \frac {1}{2}} \left(\tilde {g} _ {Z} - T _ {X \lambda} f _ {\lambda}\right) \right\| _ {\mathcal {H}} = \left\| T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - T _ {X} f _ {\lambda}\right) - (g - T f _ {\lambda}) \right] \right\| _ {\mathcal {H}} = o _ {\mathbb {P}} \left(n ^ {- \min  (s, 2) \theta / 2}\right)
$$
for both  $s > \alpha_0$  and  $s \leq \alpha_0$  cases.
A.3 Approximation results
Let us further denote
$$
\xi (x) = T _ {\lambda} ^ {- \frac {1}{2}} \left(K _ {x} f _ {\rho} ^ {*} (x) - T _ {x} f _ {\lambda}\right). \tag {41}
$$
Then, it is easy to check that
$$
T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - T _ {X} f _ {\lambda}\right) - (g - T f _ {\lambda}) \right] = \frac {1}{n} \sum_ {i = 1} ^ {n} \xi \left(x _ {i}\right) - \mathbb {E} _ {x \sim \mu} \xi (x).
$$
The following lemma deals with the easy case when  $s > \alpha_0$ .
Lemma A.5. Suppose Assumptions 1-3 hold and  $s > \alpha_0$ . Let  $\lambda \asymp n^{-\theta}$  with  $\theta \in (0, \beta)$  and  $\delta \in (0, 1)$ . Then, for  $\alpha > \alpha_0 = \beta^{-1}$  being sufficiently close, it holds with probability at least  $1 - \delta$  that
$$
\left\| T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - T _ {X} f _ {\lambda}\right) - (g - T f _ {\lambda}) \right] \right\| _ {\mathcal {H}} \leq C \ln \frac {2}{\delta} \cdot \left(M _ {\alpha} ^ {2} \frac {\lambda^ {- \alpha}}{n} + M _ {\alpha} \sqrt {\frac {\lambda^ {- \alpha} \ln n}{n}}\right) \lambda^ {\tilde {s} / 2}, \tag {42}
$$
where  $\tilde{s} = \min (s,2)$ . Consequently,