SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The following concentration result has been shown in Fischer and Steinwart [2020], Zhang et al. [2023a]. We use the form in Li et al. [2023a, Proposition 5.8] for convenience, see also Zhang et al. [2023a, Lemma 12].
Lemma B.5. Suppose  $\mathcal{H}$  has embedding index  $\alpha_0$  and Assumption 1 holds. Let  $\lambda = \lambda(n) \to 0$  satisfy  $\lambda = \Omega\left(n^{-1 / \alpha_0 + p}\right)$  for some  $p > 0$  and fix arbitrary  $\alpha \in (\alpha_0, \alpha_0 + p)$ . Then, for all  $\delta \in (0, 1)$ , when  $n$  is sufficiently large, with probability at least  $1 - \delta$ ,
$$
\left\| T _ {\lambda} ^ {- \frac {1}{2}} \left(T - T _ {X}\right) T _ {\lambda} ^ {- \frac {1}{2}} \right\| _ {\mathcal {H}} \leq C \left(\frac {\lambda^ {- \alpha}}{n} \ln n\right) ^ {1 / 2}, \tag {61}
$$
where  $C > 0$  is a constant no depending on  $\delta, n, \alpha$ , and we also have
$$
\left\| T _ {\lambda} ^ {1 / 2} T _ {X \lambda} ^ {- 1 / 2} \right\| _ {\mathcal {B} (\mathcal {H})}, \left\| T _ {\lambda} ^ {- 1 / 2} T _ {X \lambda} ^ {1 / 2} \right\| _ {\mathcal {B} (\mathcal {H})} \leq \sqrt {3}. \tag {62}
$$
The following operator inequality[Fujii et al., 1993] will be used in our proofs.
Lemma B.6 (Cordes' Inequality). Let  $A, B$  be two positive semi-definite bounded linear operators on separable Hilbert space  $H$ . Then
$$
\left\| A ^ {s} B ^ {s} \right\| _ {\mathcal {B} (H)} \leq \left\| A B \right\| _ {\mathcal {B} (H)} ^ {s}, \quad \forall s \in [ 0, 1 ]. \tag {63}
$$
The following lemma is a consequence of the fact that  $x^r$  is operator monotone when  $r \in (0,1]$  and is Lipschitz when  $r > 1$ , see Zhang et al. [2023a, Lemma 35] or Lin et al. [2018, Lemma 5.8].
Lemma B.7. Suppose that  $A$  and  $B$  are two positive self-adjoint operators on some Hilbert space, then
for  $r\in (0,1]$  , we have
$$
\left\| A ^ {r} - B ^ {r} \right\| \leq \left\| A - B \right\| ^ {r}.
$$
for  $r\geq 1$  denote  $c = \max (\| A\| ,\| B\|)$  , we have
$$
\left\| A ^ {r} - B ^ {r} \right\| \leq r c ^ {r - 1} \| A - B \|.
$$
C.1 Details of experiments in the main text
Recall that in the experiments section of the main text, we considered the kernel  $k(x,y) = \min (x,y)$  and  $x\sim \mathcal{U}[0,1]$ . We know the eigensystem of  $k$  that  $\lambda_{i} = \left(\frac{2i - 1}{2}\pi\right)^{-2}$  and  $e_i(x) = \sqrt{2}\sin \left(\frac{2i - 1}{2}\pi x\right)$ . For the three target functions used in the experiments, simple calculation shows that the relative smoothness (source condition) of  $\cos (2\pi x),\sin (2\pi x),\sin \left(\frac{3}{2}\pi x\right)$  are 0.5, 1.5,  $\infty$  respectively.
For some  $f^{*}$ , we generate data from the model  $y = f^{*}(x) + \varepsilon$  where  $\varepsilon \sim \mathcal{N}(0, 0.05)$  and perform KRR with  $\lambda = cn^{-\theta}$  for different  $\theta$ 's with some fixed constant  $c$ . Then, we numerically compute the variance, bias and excess risk by Simpson's formula with  $N \gg n$  nodes. Repeating the experiment for  $n$  ranged in 1000 to 5000 with an increment of 100, we can estimate the convergence rate  $r$  by a logarithmic least-squares log err  $= r \log n + b$  on the resulting values (variance, bias and excess risk). Figure 2 on page 22 shows the corresponding curves of the results in Table 1 in the main text. Note that for each setting, we tried different  $c$ 's in the regularization parameter  $\lambda = cn^{-\theta}$  and show the curves under the best choice of  $c$  ( $c = 0.005$ ).
Figure 2: Decay curves of the variance; the bias and excess risk of three target functions. Both axes are logarithmic. The curves show the average bias over 100 trials; and the regions within one standard deviation are shown in the corresponding colors.
C.2 Learning curves with different noises
Cui et al. [2021] discussed the 'crossover from the noiseless to noisy regime' and shown the interaction between the magnitude of noise and the sample size. As discussed in Remark 3.2 in the main text, our theory can also reflect this interaction. In Figure 3 on page 23, we exhibit the learning curves with different magnitudes of noises and visualize this interaction. Note that in the following the sample size is chosen as  $10, 20, \dots, 100, 120, \dots, 1000, 1100, \dots, 5000$ , and we use the same kernel and data generation process as before. We repeat the experiments for 100 times for each sample size and present the average excess risk.
Figure 3: Learning curves of three target functions with different noises when choosing  $\lambda = cn^{-\theta}$ ,  $\theta = 1.0, 2.0$ . Both axes are logarithmic. The black dashed lines represent the theoretical slopes under each choice of  $\theta$ .
In the above settings, the bias decays faster than variance. Figure 3 on page 23 shows that the excess risk decays fast when  $n$  is relatively small and coincides the theoretical asymptotic rate in Theorem 3.2 when  $n$  is large. The crossover happens for smaller  $n$  when the magnitude of noise is larger. Similar phenomenon has also been reported by Cui et al. [2021, FIG.2, FIG.3]. In addition, comparing the sample size when crossover happens for three target functions, our results show that the crossover happens for smaller  $n$  when the function is smoother, which is also consistent with Theorem 3.2.
Theorem 3.2 shows that when  $\theta \geq \beta$ , the excess risk is a constant asymptotically. Figure 4 on page 24 shows the curves of kernel interpolation  $(\lambda = 0)$ . It can be seen that they are similar to the curves in the second column of Figure 3 on page 23, where we choose  $\theta = \beta = 2$ .
Figure 4: Learning curves of three target functions with different noises when choosing  $\lambda = 0$ . Both axes are logarithmic. The black dashed lines represent the theoretical slopes.