SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and Carl's inequality, see e.g. (Carl and Stephani, 1990, Theorem 3.1.1) then gives
$$
\varepsilon_ {i} (I _ {k, \nu}) \leq \sqrt {c} \cdot \tilde {K} _ {q} \cdot i ^ {- q / 2}, \quad i \geq 1,
$$
where  $\tilde{K}_q$  is a constant only depending on  $q$ . By (Carl and Stephani, 1990, Inequality (3.0.9)) and (F.9) we then obtain
$$
a _ {i} \left(I _ {\tilde {k}, \nu}\right) \leq 2 \varepsilon_ {i} \left(I _ {\tilde {k}, \nu}\right) \leq 2 C \varepsilon_ {i} \left(I _ {k, \nu}\right) \leq 2 C \sqrt {c} \cdot \tilde {K} _ {q} \cdot i ^ {- q / 2}, \quad i \geq 1.
$$
Another application of (F.8) then yields the assertion for  $K_{q} \coloneqq 4\tilde{K}_{q}^{2}$ .
F.3 Kernel matrix eigenvalue bounds
For upper bounds on the eigenvalues of kernel matrices, we use the following result:
Proposition F.11 (Kernel matrix eigenvalue upper bound in expectation). For  $m \geq 1$ , we have
$$
\mathbb {E} _ {\boldsymbol {X}} \sum_ {i = m} ^ {n} \lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \leq \sum_ {i = m} ^ {\infty} \lambda_ {i} \left(T _ {k}\right). \tag {F.10}
$$
Proof. Theorem 7.29 in Steinwart and Christmann (2008) shows that
$$
\mathbb {E} _ {D \sim \mu^ {n}} \sum_ {i = m} ^ {\infty} \lambda_ {i} \left(T _ {k, D}\right) \leq \sum_ {i = m} ^ {\infty} \lambda_ {i} \left(T _ {k, \mu}\right), \tag {F.11}
$$
where  $T_{k,\mu}: L_2(\mu) \to L_2(\mu), f \mapsto \int k(x, \cdot) f(x) \, \mathrm{d}\mu(x)$  is the integral operator corresponding to the measure  $\mu$  and  $T_{k,D}$  is the corresponding discrete version thereof. We set  $\mu := P_X$  and need to show that  $k(\mathbf{X}, \mathbf{X}) / n$  has the same eigenvalues as  $T_{k,D}$  if  $D$  and  $\mathcal{X}$  contain the same data points  $\mathbf{x}_1, \ldots, \mathbf{x}_n$ . Consider a fixed  $D$ . Then, we can write  $T_{k,D}(f) = n^{-1} AB f$ , where
$$
A: \mathbb {R} ^ {n} \rightarrow L _ {2} (D), \boldsymbol {v} \mapsto \sum_ {i = 1} ^ {n} v _ {i} k (\boldsymbol {x} _ {i}, \cdot)
$$
$$
B: L _ {2} (D) \to \mathbb {R} ^ {n}, f \mapsto \left(f (\boldsymbol {x} _ {1}), \dots , f (\boldsymbol {x} _ {n})\right) ^ {\top}.
$$
Then,  $k(\mathbf{X}, \mathbf{X}) / n$  is the matrix representation of  $n^{-1}BA$  with respect to the standard basis of  $\mathbb{R}^n$ . But  $AB$  and  $BA$  have the same non-zero eigenvalues, which means that
$$
\sum_ {i = m} ^ {n} \lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) = \sum_ {i = m} ^ {\infty} \lambda_ {i} (T _ {k, D}),
$$
Usually, dyadic entropy numbers are denoted by  $e_i(\cdot)$ , but since this symbol is already used for eigenfunctions, we use  $\varepsilon_i(\cdot)$  instead.
To obtain a lower bound, we want to leverage the lower bound by Buchholz (2022) for a certain radial basis function kernel with data generated from an open subset of  $\mathbb{R}^d$ . However, we want to consider different kernels and distributions on the whole sphere. The following theorem bridges the gap by going to subsets of the data on a sphere cap, projecting them to  $\mathbb{R}^d$ , and using the kernel equivalence results from Appendix F.2:
Theorem F.12 (Kernel matrix eigenvalue lower bound for Sobolev kernels on the sphere). Let  $k$  be a kernel on  $\mathbb{S}^d$  such that its RKHS  $\mathcal{H}_k$  is equivalent to a Sobolev space  $H^{s}(\mathbb{S}^{d})$  with smoothness  $s > d / 2$ . Moreover, let  $P_{X}$  be a probability distribution on  $\mathbb{S}^d$  with lower and upper bounded density. Let the rows of  $X \in \mathbb{R}^{n\times d}$  are drawn independently from  $P_{X}$ . Then, for  $\varepsilon \in (0,1 / 20)$ , there exists a constant  $c > 0$  and  $n_0 \in \mathbb{N}$  such that for all  $n \geq n_0$ ,
$$
\lambda_ {m} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \geq c n ^ {- 2 s / d}
$$
holds with probability  $\geq 4 / 5$  for all  $m\in \mathbb{N}$  with  $1\leq m\leq (1 - 11\varepsilon)n$
Proof. We can choose a suitably large sphere cap  $T$  such that  $P_X(T) \geq 1 - \varepsilon$ . Define the conditional distribution  $P_T(\cdot) \coloneqq P_X(\cdot |T)$ . Out of the points  $\mathbf{X} = (\pmb{x}_1, \dots, \pmb{x}_n)$ , we can consider the submatrix  $\mathbf{X}_T = (\pmb{x}_{i_1}, \dots, \pmb{x}_{i_N})^\top$  of the points lying in  $T$ . Conditioned on  $N$ , these points are i.i.d. samples from  $P_T$ . Moreover, by applying Markov's inequality to a Bernoulli distribution, we obtain  $N \geq (1 - 10\varepsilon)n$  with probability  $\geq 9/10$ . We fix a value of  $N \geq (1 - 10\varepsilon)n$  in the following and condition on it.
We denote the centered unit ball in  $\mathbb{R}^d$  by  $B_{1}(\mathbb{R}^{d})$ . Using a construction as in Lemma E.1, we can transport  $k$  and  $P_T$  from  $T$  to the unit ball  $B_{1}(\mathbb{R}^{d})$  using a rescaled stereographic projection feature map  $\phi$ , such that we obtain a kernel  $k_{\phi}$  and a distribution  $P_{\phi} = (P_{T})_{\phi}$  on  $B_{1}(\mathbb{R}^{d})$  that generate the same distribution of kernel matrices as  $k$  with  $P_T$ , and such that  $\mathcal{H}_{k_{\phi}} \cong H^{s}(B_{1}(\mathbb{R}^{d}))$ . The rows of  $\pmb{X}_{\phi} \coloneqq \phi(\pmb{X}_T)$  are i.i.d. samples from  $P_{\phi}$ . Moreover, we know that  $P_{\phi}$  has an lower and upper bounded density w.r.t. the Lebesgue measure on  $B_{1}(\mathbb{R}^{d})$ .
In order to apply the results from Buchholz (2022), we define a translation-invariant reference kernel on  $\mathbb{R}^d$  through the Fourier transform
$$
\hat {k} _ {\mathrm {r e f}} (\xi) = \left(1 + | \xi | ^ {2}\right) ^ {- 2 s},
$$
see Eq. (3) in Buchholz (2022). The RKHS of  $k_{\mathrm{ref}}$  on  $\mathbb{R}^d$  is equivalent to the Sobolev space  $H^{s}(\mathbb{R}^{d})$ . Therefore, the RKHS of  $k_{\mathrm{ref}}|_{B_1(\mathbb{R}^d), B_1(\mathbb{R}^d)}$  is  $H^{s}(B_{1}(\mathbb{R}^{d}))$ , cf. the remarks in Appendix B.1 and Lemma F.7.
Now, let  $1 \leq m \leq (1 - 11\varepsilon)n$ , which implies
$$
1 \leq m \leq (1 - 1 1 \varepsilon) n \leq (1 - \varepsilon) (1 - 1 0 \varepsilon) n \leq (1 - \varepsilon) N.
$$
We apply Theorem 12 by Buchholz (2022) with bandwidth  $\gamma = 1$  and  $\alpha = 2s$  to  $\lambda_{m}$  and obtain with probability at least  $1 - 2 / N$ :
$$
\begin{array}{l} \lambda_ {m} \left(k _ {\text {r e f}} \left(\boldsymbol {X} _ {\phi}, \boldsymbol {X} _ {\phi}\right)\right) ^ {- 1} \leq c _ {3} \left(\frac {N ^ {2 (\alpha - d) / d}}{(N - m) ^ {(\alpha - d) / d}} + 1\right) \leq c _ {3} \left(\frac {N ^ {2 (\alpha - d) / d}}{(\varepsilon N) ^ {(\alpha - d) / d}} + 1\right) \\ \leq c _ {4} \left(n ^ {\alpha / d - 1} + 1\right) \\ \end{array}
$$
as long as  $N$  is large enough such that  $(1 - \varepsilon)N < N - 32\ln (N)$ , which is the case if  $n$  is large enough. Here, the constant  $c_{3}$  from Buchholz (2022) does not depend on  $N$  or  $m$ , but only on  $\alpha$ ,  $d$ , and the upper and lower bounds on the density, which in our case depend on  $\varepsilon$  through the choice of  $T$ . Since  $\alpha = 2s > d$ , we have  $n^{\alpha /d - 1} > 1$  and therefore
$$
\lambda_ {m} \left(k _ {\mathrm {r e f}} \left(\boldsymbol {X} _ {\phi}, \boldsymbol {X} _ {\phi}\right) / n\right) \geq c _ {5} n ^ {- \alpha / d} = c _ {5} n ^ {- 2 s / d}.
$$
Now, we want to translate this to the kernel  $k$ . Since the RKHSs of  $k_{\phi}$  and  $k_{\mathrm{ref}}$  on  $B_{1}(\mathbb{R}^{d})$  are both equivalent to  $H^{s}(B_{1}(\mathbb{R}^{d}))$ , the kernels themselves are  $C$ -equivalent for some constant  $C \geq 1$  as defined in Definition F.4. Therefore, Proposition F.8 shows that the corresponding kernel matrices are  $C^2$ -equivalent, which implies
$$
\lambda_ {m} \left(k _ {*}, \phi \left(\boldsymbol {X} _ {\phi}, \boldsymbol {X} _ {\phi}\right) / n\right) \geq c _ {5} C ^ {- 2} n ^ {- 2 s / d}.
$$
By Cauchy's interlacing theorem, we therefore have
$$
\lambda_ {m} (k _ {*} (\boldsymbol {X}, \boldsymbol {X}) / n) \geq \lambda_ {m} (k _ {*} (\boldsymbol {X} _ {T}, \boldsymbol {X} _ {T}) / n) = \lambda_ {m} (k _ {*, \phi} (\boldsymbol {X} _ {\phi}, \boldsymbol {X} _ {\phi}) / n) \geq c _ {5} C ^ {- 2} n ^ {- 2 s / d}.
$$
Denoting the event where  $\lambda_{m}(k_{*}(\pmb {X},\pmb {X}) / n)\geq c_{5}C^{-2}n^{-2s / d}$  by  $A$  , we thus have
$$
\begin{array}{l} P (A) = P (A | N \geq (1 - 1 0 \varepsilon) n) P (N \geq (1 - 1 0 \varepsilon) n) \geq \frac {9}{1 0} P (A | N \geq (1 - 1 0 \varepsilon) n) \\ = \frac {9}{1 0} \sum_ {\hat {N} = \lceil (1 - 1 0 \varepsilon) n \rceil} ^ {n} P (N = \hat {N} | N \geq (1 - 1 0 \varepsilon) n) P (A | N = \hat {N}) \\ \geq \frac {9}{1 0} \sum_ {\hat {N} = \lceil (1 - 1 0 \varepsilon) n \rceil} ^ {n} P (N = \hat {N} | N \geq (1 - 1 0 \varepsilon) n) (1 - 2 / N) \\ \geq \frac {9}{1 0} \left(1 - \frac {2}{(1 - 1 0 \varepsilon) n}\right) \sum_ {\hat {N} = \lceil (1 - 1 0 \varepsilon) n \rceil} ^ {n} P (N = \hat {N} | N \geq (1 - 1 0 \varepsilon) n) \\ = \frac {9}{1 0} \left(1 - \frac {2}{(1 - 1 0 \varepsilon) n}\right) \geq \frac {4}{5}, \\ \end{array}
$$
where the last step holds for sufficiently large  $n$ .
F.4 Spectral lower bound for dot-product kernels on the sphere
An application of the spectral generalization bound in Proposition 5 requires a lower bound on eigenvalues of the kernel matrix  $k_{*}(\mathbf{X},\mathbf{X})$ . To achieve this, we need to understand the properties of the convolution kernel  $k_{*}$ . Since the eigenvalues of  $T_{k_{*},P_{X}}$  are the squared eigenvalues of  $T_{k,P_X}$ , one might hope that if  $\mathcal{H}_k$  is equivalent to a Sobolev space  $H^{s}$ , then  $\mathcal{H}_{k_*}$  is equivalent to a Sobolev space  $H^{\tilde{2}s}$ . Unfortunately, this is not the case in general, as  $\mathcal{H}_{k_*}$  might be a smaller space that involves additional boundary conditions (Schaback, 2018). However, perhaps since the sphere is a manifold without boundary, the desired characterization of  $\mathcal{H}_{k_*}$  holds for dot-product kernels on the sphere:
Lemma F.13 (RKHS of convolution kernels). Let  $k$  be a dot-product kernel on  $\mathbb{S}^d$  such that its RKHS  $\mathcal{H}_k$  is equivalent to a Sobolev space  $H^{s}(\mathbb{S}^{d})$  with smoothness  $s > d / 2$ , and let  $P_{X}$  be a distribution on  $\mathbb{S}^d$  with lower and upper bounded density. Then, the RKHS  $\mathcal{H}_{k_*}$  of the kernel
$$
k _ {*}: \mathbb {S} ^ {d} \times \mathbb {S} ^ {d} \rightarrow \mathbb {R}, k _ {*} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) := \int k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime \prime}) k (\boldsymbol {x} ^ {\prime \prime}, \boldsymbol {x} ^ {\prime}) \mathrm {d} P _ {X} (\boldsymbol {x} ^ {\prime \prime})
$$
is equivalent to the Sobolev space  $H^{2s}(\mathbb{S}^d)$ .
$$
k _ {*, \text {u n i f}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \int k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime \prime}) k (\boldsymbol {x} ^ {\prime \prime}, \boldsymbol {x} ^ {\prime}) \mathrm {d} \mathcal {U} (\mathbb {S} ^ {d}) (\boldsymbol {x} ^ {\prime \prime}).
$$
For the corresponding integral operator, we have
$$
T _ {k _ {*}, \text {u n i f}}, \mathcal {U} (\mathbb {S} ^ {d}) = T _ {k, \mathcal {U} (\mathbb {S} ^ {d})} ^ {2}.
$$
This means that the corresponding eigenvalues are the squares of the eigenvalues of the corresponding integral operator of  $k$ . Especially, we obtain the Mercer representations
$$
k \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = \sum_ {l = 0} ^ {\infty} \mu_ {l} \sum_ {i = 1} ^ {N _ {l, d}} Y _ {l, i} (\boldsymbol {x}) Y _ {l, i} \left(\boldsymbol {x} ^ {\prime}\right),
$$
$$
k _ {*, \text {u n i f}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sum_ {l = 0} ^ {\infty} \mu_ {l} ^ {2} \sum_ {i = 1} ^ {N _ {l, d}} Y _ {l, i} (\boldsymbol {x}) Y _ {l, i} (\boldsymbol {x} ^ {\prime}),
$$
where Lemma B.1 yields  $\mu_l = \Theta((l + 1)^{-2s})$ , hence  $\mu_l^2 = \Theta((l + 1)^{-4s})$  and hence  $\mathcal{H}_{k_{*,\mathrm{unif}}}\cong H^{2s}(\mathbb{S}^d)$ .
Next, we show the equality of the ranges of the integral operators:
$$
R \left(T _ {k, \mathcal {U} (\mathbb {S} ^ {d})}\right) = R \left(T _ {k, P _ {X}}\right).
$$
Let  $p_X$  be a density of  $P_X$  w.r.t. the uniform distribution  $\mathcal{U}(\mathbb{S}^d)$ . If  $f \in R(T_{k,\mathcal{U}(\mathbb{S}^d)})$ , there exists  $g \in L_2(\mathcal{U}(\mathbb{S}^d))$  with  $f = T_{k,\mathcal{U}(\mathbb{S}^d)}g$ . But then, since  $p_X$  is lower bounded, we have  $g / p_X \in L_2(P_X)$  and therefore
$$
f = T _ {k, P _ {X}} \left(g / p _ {X}\right) \in R \left(T _ {k, P _ {X}}\right).
$$
An analogous argument shows that  $R(T_{k,P_X}) \subseteq R(T_{k,\mathcal{U}(\mathbb{S}^d)})$  since  $p_X$  is upper bounded.
The equality of the ranges yields for the RKHSs (as sets)
$$
\mathcal {H} _ {k _ {*}, \text {u n i f}} = R \left(T _ {k, \mathcal {U} (\mathbb {S} ^ {d})}\right) = R \left(T _ {k, P _ {X}}\right) = \mathcal {H} _ {k _ {*}},
$$
Applying Lemma F.7 twice then shows  $\mathcal{H}_{k_*} \cong H^{2s}(\mathbb{S}^d)$ .
Proof. Step 0: Preparation. Since the Sobolev space  $H^{2s}(\mathbb{S}^d)$  is dense in the space of continuous functions  $\mathbb{S}^d \to \mathbb{R}$ , the kernel  $k$  is universal. Applying (Steinwart and Christmann, 2008, Corollary 5.29 and Corollary 5.34) for the least squares loss thus shows that  $k$  is strictly positive definite. If we have mutually distinct  $x_1, \ldots, x_n$ , the corresponding Gram matrix  $k((x_i, x_j))_{i,j=1}^n$  is therefore invertible. Now, our assumptions on  $P$  guarantee that  $X$  consists almost surely of mutually distinct observations, and therefore  $k(X, X)$  is almost surely invertible.
By Proposition 5, we know that
$$
\begin{array}{l} \mathbb {E} _ {D} \mathcal {R} _ {P} (f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}) - \mathcal {R} _ {P} ^ {*} \geq \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {X}} \frac {\lambda_ {i} \left(k _ {*} (\boldsymbol {X} , \boldsymbol {X}) / n\right) \left(1 - e ^ {- 2 t \left(\lambda_ {i} \left(k (\boldsymbol {X} , \boldsymbol {X}) / n\right) + \rho\right)\right) ^ {2}}{\left(\lambda_ {i} \left(k (\boldsymbol {X} , \boldsymbol {X}) / n\right) + \rho\right) ^ {2}} \\ \geq \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {X}} \frac {\lambda_ {i} \left(k _ {*} (\boldsymbol {X} , \boldsymbol {X}) / n\right) \left(1 - e ^ {- 2 C ^ {- 1} n ^ {2 s / d} \left(\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + 0\right)}\right) ^ {2}}{\left(\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + C n ^ {- 2 s / d}\right) ^ {2}} \\ \geq c _ {n} \sigma^ {2} \\ \end{array}
$$
for a suitable constant  $c_{n} > 0$  depending on  $n$  but not on  $\sigma^2, t, \rho$ , since the kernel matrix eigenvalues are nonzero almost surely. It is therefore sufficient to show the desired statement (with  $c$  independent of  $n, \sigma^2, t, \rho$ ) for sufficiently large  $n$ .
In the following, we assume  $n \geq 40$  and set  $\varepsilon \coloneqq 1 / 100$ .
Step 1: Eigenvalue decay for the integral operator. From Lemma B.1, we know that
$$
\lambda_ {i} \left(T _ {k, \mathcal {U} (\mathbb {S} ^ {d})}\right) = \Theta \left(i ^ {- 2 s / d}\right).
$$
Therefore, by Lemma F.9, we know that
$$
\lambda_ {i} \left(T _ {k, P _ {X}}\right) = \Theta \left(i ^ {- 2 s / d}\right).
$$
Step 2: Eigenvalue upper bound. Next, we want to upper-bound suitable eigenvalues of the form  $\lambda_{i}(k(\mathbf{X},\mathbf{X}) / n)$  using Proposition F.11. Using Step 1, we derive
$$
\sum_ {i = m} ^ {\infty} \lambda_ {i} \left(T _ {k, P _ {X}}\right) \leq C _ {1} \sum_ {i = m} ^ {\infty} i ^ {- 2 s / d} \leq C _ {2} \int_ {m} ^ {\infty} x ^ {- 2 s / d} \mathrm {d} x = C _ {3} m ^ {1 - 2 s / d}
$$
with constants independent of  $m \geq 1$ . For sufficiently large  $n$ , we can choose  $m \in \mathbb{N}_{\geq 1}$  such that  $\varepsilon n \leq m \leq 2\varepsilon n$ . Then, Proposition F.11 yields
$$
\mathbb {E} _ {\boldsymbol {X}} \sum_ {i = m} ^ {n} \lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \leq \sum_ {i = m} ^ {\infty} \lambda_ {i} (T _ {k}) \leq C _ {3} m ^ {1 - 2 s / d} \leq C _ {4} n ^ {1 - 2 s / d}.
$$
Since  $\mathbb{E}_{\pmb{X}}\lambda_{i}(k(\pmb {X},\pmb {X}) / n)$  is decreasing with  $i$  , we have for  $i\geq 4\varepsilon n\geq 2m$  ..
$$
\mathbb {E} _ {\boldsymbol {X}} \lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \leq C _ {4} n ^ {1 - 2 s / d} / m \leq C _ {5} n ^ {- 2 s / d} \leq C _ {6} \lambda_ {i} \left(T _ {k, P _ {X}}\right).
$$
Step 3: Eigenvalue lower bounds. From Lemma F.13, we know that  $\mathcal{H}_{k_*} \cong H^{2s}(\mathbb{S}^d)$ . Therefore, we can apply Lemma F.13 to both  $k$  and  $k_*$  and obtain for sufficiently large  $n$  and suitable constants  $c_1, c_2 > 0$  that
$$
\lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \geq c _ {1} n ^ {- 2 s / d}
$$
$$
\lambda_ {i} \left(k _ {*} (\boldsymbol {X}, \boldsymbol {X}) / n\right) \geq c _ {2} n ^ {- 4 s / d}
$$
individually hold with probability  $\geq 4/5$  for all  $i \in \mathbb{N}$  with  $1 \leq i \leq (1 - 11\varepsilon)n$ . By the union bound, both bounds hold at the same time with probability  $\geq 3/5$ .
Step 4: Final result. Now, using the value of  $m$  from Step 2, consider an index  $i$  with  $2m \leq i \leq (1 - 11\varepsilon)n$ . Since  $2m \leq 4\varepsilon n$  and  $\varepsilon = 1/100$ , there are at least  $n/2$  such indices. By combining Steps 3 and Step 1, we have
$$
\lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \geq c _ {3} \lambda_ {i} \left(T _ {k, P _ {X}}\right)
$$
$$
\lambda_ {i} (k _ {*} (\boldsymbol {X}, \boldsymbol {X}) / n) \geq c _ {4} \lambda_ {i} (T _ {k, P _ {X}}) ^ {2}
$$
with probability  $\geq 3/5$ . By applying Markov's inequality to Step 2, we obtain
$$
\lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \leq 1 0 C _ {6} \lambda_ {i} \left(T _ {k, P _ {X}}\right)
$$
with probability  $\geq 9/10$ . Therefore, by the union bound, all three inequalities hold simultaneously with probability  $\geq 1/2$ . Moreover, for  $q = 2s/d$ , we have  $\lambda_i(T_{k,P_X}) \geq c_5i^{-q}$  by Step 1. We can thus apply the first lower bound from Theorem F.2 to obtain
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} (f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}) - R _ {P} ^ {*} \geq c ^ {\prime} \sigma^ {2} \frac {1}{1 + (\rho + t ^ {- 1}) n ^ {2 s / d}} \cdot \frac {| \mathcal {I} (n) |}{n} \\ \geq c ^ {\prime} \sigma^ {2} \frac {1}{1 + (C n ^ {- 2 s / d} + C n ^ {- 2 s / d}) n ^ {2 s / d}} \cdot \frac {n / 2}{n} \\ = \frac {c ^ {\prime}}{2 + 2 C} \sigma^ {2}. \\ \end{array}
$$
G Proof of Theorem 8
Here we denote the solution of kernel ridge regression on  $D$  with the kernel function  $k$  and regularization parameter  $\rho > 0$  as
$$
\hat {f} _ {\rho} ^ {k} (\boldsymbol {x}) = k (\boldsymbol {x}, \boldsymbol {X}) (k (\boldsymbol {X}, \boldsymbol {X}) + \rho \boldsymbol {I}) ^ {- 1} \boldsymbol {y},
$$
and write  $\hat{f}_0^k (\pmb {x}) = k(\pmb {x},\pmb {X})k(\pmb {X},\pmb {X})^+ \pmb{y}$  for the minimum-norm interpolant in the RKHS of  $k$
While Theorem 1 states that overfitting kernel ridge regression using Sobolev kernels is always inconsistent as long as the derivatives remain bounded by the derivatives of the minimum-norm interpolant of the fixed kernel (Assumption (N)), here we show that consistency over a large class of distributions is achievable by designing a kernel sequence, which can have Sobolev RKHS, that consists of a smooth component for generalization and a spiky component for interpolation.
Recall that  $\tilde{k}$  denotes any universal kernel function for the smooth component, and  $\tilde{k}_{\gamma}$  denotes the kernel function of the spiky component with bandwidth  $\gamma$ . Then we define the  $\rho$ -regularized spiky-smooth kernel with spike bandwidth  $\gamma$  as
$$
k _ {\rho , \gamma} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \tilde {k} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) + \rho \cdot \check {k} _ {\gamma} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}).
$$
Let  $B_{t}(\pmb {x})\coloneqq \{\pmb {y}\in \mathbb{R}^{d}\mid |\pmb {x} - \pmb {y}|\leq t\}$  denote the Euclidean ball of radius  $t\geq 0$  around  $\pmb {x}\in \mathbb{R}^d$
(D2) There exists a constant  $\beta_{X} > 0$  and a continuous function  $\phi :[0,\infty)\to [0,1]$  with  $\phi (0) = 0$  such that  $P_{X}(B_{t}(\pmb {x}))\leq \phi (t) = O(t^{\beta_{X}})$  for all  $\pmb {x}\in \Omega$  and all  $t\geq 0$
The kernel  $\tilde{k}_{\gamma}$  of the spiky component should fulfill the following weak assumption on its decay behaviour. For example, Laplace, Matérn, and Gaussian kernels all fulfill Assumption (SK).
(SK) There exists a function  $\varepsilon : (0, \infty) \times [0, \infty) \to [0, 1]$  such that for any bandwidth  $\gamma > 0$  and any  $\delta > 0$  it holds that
(i)  $\varepsilon (\gamma ,0) = 1$
(ii)  $\varepsilon (\gamma ,\delta)$  is monotonically increasing in  $\gamma$
(iii) For all  $\pmb{x}, \pmb{y} \in \Omega$ , if  $|\pmb{x} - \pmb{y}| \geq \delta$  then  $|\tilde{k}_{\gamma}(\pmb{x}, \pmb{y})| \leq \varepsilon (\gamma, \delta)$ ,
(iv) For any rates  $\beta_{X},\beta_{k} > 0$  there exists a rate  $\beta_{\gamma} > 0$  such that, if  $\delta_n = \Omega (n^{-\beta_X})$  and  $\gamma_{n} = O(n^{-\beta_{\gamma}})$ , then  $\varepsilon (\gamma_n,\delta_n) = O(n^{-\beta_k})$ .
Theorem G.1 (Consistency of spiky-smooth ridgeless kernel regression). Assume that the training set  $D$  consists of  $n$  i.i.d. pairs  $(x,y)\sim P$  such that the marginal  $P_{X}$  fulfills (D2) and  $\mathbb{E}y^2 < \infty$ . Let the kernel components satisfy:
-  $\tilde{k}$  denotes an arbitrary universal kernel, and  $\rho_n\to 0$  and  $n\rho_n^4\rightarrow \infty$
-  $\tilde{k}_{\gamma_n}$  denotes a kernel function that fulfills Assumption (SK) with a sequence of positive bandwidths  $(\gamma_{n})$  fulfilling  $\gamma_{n} = O(\exp (-\beta n))$  for some arbitrary  $\beta >0$
Remark G.2 (Spike bandwidth scaling). Under stronger assumptions on  $\phi$  and  $\varepsilon$  in assumptions (D2) and (SK), the spike bandwidths  $\gamma_{n}$  can be chosen to converge to 0 at a much slower rate. For example, if we choose  $\check{k}_{\gamma}$  to be the Laplace kernel, choosing bandwidths  $0 < \gamma_{n} \leq \frac{\delta}{\beta \ln n}$  yields, for separated points  $|\pmb{x} - \pmb{y}| \geq \delta$ ,
$$
\check {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {y}) \leq \exp \left(- \frac {\delta}{\gamma_ {n}}\right) \leq n ^ {- \beta}.
$$
For probability measures with upper bounded Lebesgue density, we can choose  $\delta_n = n^{-\frac{2 + \alpha}{d}}$  and  $\beta = \frac{9}{4} + \frac{\alpha}{2}$  for consistency or  $\beta = \frac{11}{4} + \frac{\alpha}{2}$  for optimal convergence rates, for any fixed  $\alpha > 0$ , in the proof of Theorem 8. Hence the Laplace kernel only requires a slow bandwidth decay rate of  $\gamma_n = \Omega \left( \frac{n - \frac{2 + \alpha}{d}}{\alpha \ln(n)} \right)$ , where  $\alpha > 0$  arbitrary. For the Gaussian kernel an analogous argument yields  $\gamma_n = \Omega \left( \frac{n - \frac{4 + 2\alpha}{d}}{\alpha \ln(n)} \right)$ . The larger the dimension  $d$ , the slower the required bandwidth decay.
Remark G.3 (Generalizations). If one does not care about continuous kernels, one could simply take a Dirac kernel as the spike and then obtain consistency for all atom-free  $P_{X}$ . However, we need a continuous kernel to be able to translate it to an activation function for the NTK. Beyond kernel regression, the spike component  $\tilde{k}_{\gamma}$  does not even need to be a kernel, it just needs to fulfill Assumption (SK) or a similar decay criterion. Then one could still use the 'quasi minimum-norm estimator'  $\boldsymbol{x} \mapsto (\tilde{k} + \rho_n \check{k}_{\gamma_n})(\boldsymbol{x}, \boldsymbol{X}) \cdot (\tilde{\boldsymbol{K}} + \rho_n \check{\boldsymbol{K}}_{\gamma_n})^+ \boldsymbol{y}$ .
Remark G.4 (Consistency with a single kernel function). Without resorting to kernel sequences as we do, there seems to be no rigorous proof showing that ridgeless kernel regression can be consistent in fixed dimension. In future work, can an analytical expression of such a kernel be found? According to the semi-rigorous results in Mallinar et al. (2022) a spectral decay like  $\lambda_{k} = \Theta (k^{-1}\cdot \log^{\alpha}(k))$ ,  $\alpha > 1$  could lead to such a kernel.