SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem G.1. Given any universal kernel, (Steinwart, 2001, Theorem 3.11 or Example 4.6) implies universal consistency of kernel ridge regression if  $\rho_{n}\to 0$  and  $n\rho_n^4\to \infty$ . Hence, for any  $\varepsilon >0$  it holds that
$$
\lim  _ {n \rightarrow \infty} P ^ {n} \left(D \in (\mathbb {R} ^ {d} \times \mathbb {R}) ^ {n} \mid R _ {P} (\hat {f} _ {\rho_ {n}} ^ {\tilde {k}}) - R _ {P} (f _ {P} ^ {*}) = \mathbb {E} _ {\boldsymbol {x}} (\hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x}) - f _ {P} ^ {*} (\boldsymbol {x})) ^ {2} \geq (\varepsilon / 2) ^ {2}\right) = 0.
$$
Due to the triangle inequality in  $L_{2}(P_{X})$ , we know
$$
R _ {P} (\hat {f} _ {0} ^ {k _ {n}}) - R _ {P} (f _ {P} ^ {*}) = \mathbb {E} _ {\boldsymbol {x}} (\hat {f} _ {0} ^ {k _ {n}} (\boldsymbol {x}) - f _ {P} ^ {*} (\boldsymbol {x})) ^ {2}
$$
$$
\leq \left(\left(\mathbb {E} _ {\boldsymbol {x}} (\hat {f} _ {0} ^ {k _ {n}} (\boldsymbol {x}) - \hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x})) ^ {2}\right) ^ {1 / 2} + \left(\mathbb {E} _ {\boldsymbol {x}} (\hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x}) - f _ {P} ^ {*} (\boldsymbol {x})) ^ {2}\right) ^ {1 / 2}\right) ^ {2}.
$$
It is left to show that  $k_{n}$  fulfills
$$
\lim  _ {n \rightarrow \infty} P ^ {n} \left(D \in (\mathbb {R} ^ {d} \times \mathbb {R}) ^ {n} \mid \mathbb {E} _ {\boldsymbol {x}} \left(\hat {f} _ {0} ^ {k _ {n}} (\boldsymbol {x}) - \hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x})\right) ^ {2} \geq (\varepsilon / 2) ^ {2}\right) = 0.
$$
For this purpose we decompose the above difference into the difference of  $\pmb{K}_{\gamma_n} \coloneqq \check{k}_{\gamma_n}(\pmb{X}, \pmb{X})$  and  $\pmb{I}_n$  and a remainder term depending on  $\check{k}_{\gamma_n}$ . We denote the 2-operator norm by  $\| \cdot \|$  and the Euclidean norm in  $\mathbb{R}^n$  by  $|\cdot|$ . For any  $\pmb{x} \in \mathbb{R}^d$  it holds that
$$
\begin{array}{l} \left. \left| \hat {f} _ {0} ^ {k} (\boldsymbol {x}) - \hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x}) \right| \leq \left| (\tilde {k} + \rho_ {n} \check {k} _ {\gamma_ {n}}) (\boldsymbol {x}, \boldsymbol {X}) \cdot \left(\tilde {\boldsymbol {K}} + \rho_ {n} \check {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} \boldsymbol {y} - \tilde {k} (\boldsymbol {x}, \boldsymbol {X}) \cdot \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {y} \right| \right. \\ \leq \left| \tilde {k} (\boldsymbol {x}, \boldsymbol {X}) \left(\left(\tilde {\boldsymbol {K}} + \rho_ {n} \breve {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} - \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1}\right) \boldsymbol {y} \right| \\ + \rho_ {n} \cdot \left| \check {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {X}) \left(\tilde {\boldsymbol {K}} + \rho_ {n} \check {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} \boldsymbol {y} \right| \\ \leq \| \tilde {k} (\boldsymbol {x}, \boldsymbol {X}) \| \cdot \left\| \left(\tilde {\boldsymbol {K}} + \rho_ {n} \check {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} - \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \right\| \cdot | \boldsymbol {y} | \\ + \rho_ {n} \cdot \| \tilde {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {X}) \| \cdot \| (\tilde {\boldsymbol {K}} + \rho_ {n} \tilde {\boldsymbol {K}} _ {\gamma_ {n}}) ^ {- 1} \| \cdot | \boldsymbol {y} |. \\ \end{array}
$$
Consequently we get
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \left(\hat {f} _ {0} ^ {k} (\boldsymbol {x}) - \hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x})\right) ^ {2} \leq 2 \mathbb {E} _ {\boldsymbol {x}} \| \tilde {k} (\boldsymbol {x}, \boldsymbol {X}) \| ^ {2} \cdot \left\| \left(\tilde {\boldsymbol {K}} + \rho_ {n} \tilde {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} - \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \right\| ^ {2} \cdot | \boldsymbol {y} | ^ {2} (G.1) \\ + 2 \rho_ {n} ^ {2} \cdot \mathbb {E} _ {\boldsymbol {x}} \| \tilde {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {X}) \| ^ {2} \cdot \| \left(\tilde {\boldsymbol {K}} + \rho_ {n} \tilde {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} \| ^ {2} \cdot | \boldsymbol {y} | ^ {2}. (G.2) \\ \end{array}
$$
We now bound the individual terms in Eq. (G.1) and (G.2). To this end, fix any  $\alpha > 0$ .
Bounding Eq. (G.1):
Since we assumed  $y_{i}$  i.i.d. and  $\mathbb{E}y_1^2 <  \infty$  , the Markov inequality implies, with  $b_{n} = \mathbb{E}y_{1}^{2}\cdot n^{\alpha}$
$$
P (| \boldsymbol {y} | ^ {2} \geq b _ {n} n) \leq \frac {\mathbb {E} y _ {1} ^ {2}}{b _ {n}} = n ^ {- \alpha}.
$$
Stated differently, with probability at least  $1 - n^{-\alpha}$  it holds that  $|\pmb{y}|^2 \leq \mathbb{E}y_1^2 \cdot n^{1 + \alpha}$ .
In order to bound the spectrum of  $\tilde{K}_{\gamma_n}$ , Lemma G.7 implies that there exists a positive sequence  $\delta_{\alpha}(n) = n^{-\frac{2 + \alpha}{\beta X}}$  such that with probability at least  $1 - O(n^{-\alpha})$  it holds that
$$
\min_{i,j\in [n]:i\neq j}|\boldsymbol{x}_{i} - \boldsymbol{x}_{j}| \geq \delta_{\alpha}(n).
$$
Since  $(\gamma_{n})$  fulfills  $\gamma_{n} = O(n^{-\beta_{\gamma}})$  for any  $\beta_{\gamma} > 0$ , by Assumption (SK) there exists a sequence  $\varepsilon_{n} = o(\rho_{n}n^{-2 - \frac{\alpha}{2}})$  such that  $\varepsilon (\gamma_n,\delta_\alpha (n))\leq \varepsilon_n$ . Assumption (SK) further implies that whenever  $\min_{i,j\in [n]:i\neq j}|\pmb {x}_i - \pmb {x}_j|\geq \delta_\alpha (n)$  it holds that  $(\bar{K}_{\gamma_n})_{ii} = 1$  and  $0\leq (\tilde{K}_{\gamma_n})_{ij}\leq \varepsilon (\gamma_n,\delta_\alpha (n))\leq \varepsilon_n$  for  $i\neq j$ . Then Gershgorin's theorem (Gerschgorin, 1931) implies that for all eigenvalues of  $\tilde{K}_{\gamma_n}$
$$
\left| \lambda_ {i} \left(\tilde {\boldsymbol {K}} _ {\gamma_ {n}}\right) - 1 \right| \leq (n - 1) \varepsilon_ {n} \text {f o r a l l} i \in [ n ].
$$
This in turn implies
$$
\| \tilde {\boldsymbol {K}} _ {\gamma_ {n}} - \boldsymbol {I} _ {n} \| \leq (n - 1) \varepsilon_ {n}, \quad \lambda_ {\max } (\tilde {\boldsymbol {K}} _ {\gamma_ {n}}) \leq 1 + (n - 1) \varepsilon_ {n}, \quad \lambda_ {\min } (\tilde {\boldsymbol {K}} _ {\gamma_ {n}}) \geq 1 - (n - 1) \varepsilon_ {n}.
$$
Using  $\| (\tilde{\pmb{K}} +\rho_n\pmb {I}_n)^{-1}\| \leq \frac{1}{\lambda_{\min}(\tilde{\pmb{K}}) + \rho_n}\leq \rho_n^{-1}$  and  $\| \tilde{\pmb{K}}_{\gamma_n} - \pmb {I}_n\| \leq (n - 1)\varepsilon_n$  , Lemma G.8 implies
$$
\begin{array}{l} \left\| \left(\tilde {\boldsymbol {K}} + \rho_ {n} \breve {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} - \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \right\| \leq \frac {\| \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \| ^ {2} \cdot \rho_ {n} \| \breve {\boldsymbol {K}} _ {\gamma_ {n}} - \boldsymbol {I} _ {n} \|}{1 - \| \left(\tilde {\boldsymbol {K}} + \rho_ {n} \boldsymbol {I} _ {n}\right) ^ {- 1} \| \cdot \rho_ {n} \| \breve {\boldsymbol {K}} _ {\gamma_ {n}} - \boldsymbol {I} _ {n} \|} \\ \leq \frac {\rho_ {n} ^ {- 1} (n - 1) \varepsilon_ {n}}{1 - (n - 1) \varepsilon_ {n}}. \\ \end{array}
$$
Using  $|\tilde{k}(\boldsymbol{x}, \boldsymbol{X}_i)| \leq 1$  for all  $i \in [n]$  yields the naive bound  $\| \tilde{k}(\boldsymbol{x}, \boldsymbol{X}) \|^2 \leq n$ .
Combining all terms in Eq. (G.1) yields its convergence to 0 as the product satisfies the rate  $O(n^{4 + \alpha}\rho_n^{-2}\varepsilon_n^2) = o(1)$  with probability at least  $1 - 2n^{-\alpha}$ .
Bounding Eq. (G.2):
The analysis below is restricted to the event of probability at least  $1 - 2n^{-\alpha}$ , on which the bound on Eq. (G.1) holds.
Since  $(n - 1)\varepsilon_{n}\rightarrow 0$ , for any  $C > 1$  it holds for  $n$  large enough,
$$
\rho_ {n} \cdot \| (\tilde {\boldsymbol {K}} + \rho_ {n} \breve {\boldsymbol {K}} _ {\gamma_ {n}}) ^ {- 1} \| \leq \frac {\rho_ {n}}{\lambda_ {\min} (\tilde {\boldsymbol {K}}) + \rho_ {n} (1 - (n - 1) \varepsilon_ {n})} \leq \frac {1}{(1 - (n - 1) \varepsilon_ {n})} \leq C.
$$
Finally we show  $\sup_{\pmb{x}^{\prime}\in \mathbb{R}^{d}}\mathbb{E}_{\pmb{x}}\check{k}_{\gamma_{n}}(\pmb {x},\pmb{x}^{\prime})^{2}\leq 2n^{-(2 + \alpha)}$  for  $n$  large enough.
Fix an arbitrary  $\pmb{x}^{\prime} \in \mathbb{R}^{d}$ . Then by construction of  $\delta_{\alpha}(n)$  and  $\varepsilon_{n}$  it holds that
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \check {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) ^ {2} \leq 1 \cdot P _ {X} \big (\{\boldsymbol {x} \in \mathbb {R} ^ {d}: \check {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) ^ {2} \geq \varepsilon_ {n} ^ {2} \} \big) + \varepsilon_ {n} ^ {2} \\ \leq P _ {X} \left(\left\{\boldsymbol {x} \in \mathbb {R} ^ {d}: | \boldsymbol {x} - \boldsymbol {x} ^ {\prime} | <   \delta_ {\alpha} (n) \right\}\right) + \varepsilon_ {n} ^ {2} \\ \leq \phi (\delta_ {\alpha} (n)) + \varepsilon_ {n} ^ {2} \leq n ^ {- (2 + \alpha)} + \varepsilon_ {n} ^ {2}. \\ \end{array}
$$
Since  $\varepsilon_n^2 = o(\rho_n^2 n^{-4 - \alpha})$ , we get  $\mathbb{E}_{\boldsymbol{x}}\check{k}_{\gamma_n}(\boldsymbol{x},\boldsymbol{x}')^2\leq 2n^{-(2 + \alpha)}$  for  $n$  large enough.
Combining all terms in Eq. (G.2) yields its convergence to 0 with the rate  $O(n^{-(2 + \alpha)}\cdot 1\cdot n^{1 + \alpha}) = O(n^{-1})$  with probability at least  $1 - 2n^{-\alpha}$ , which concludes the proof.
The following theorem shows that the minimum-norm interpolants of the spiky-smooth kernel sequence can achieve optimal convergence rates for Sobolev target functions, as long as  $\rho_{n}$  is properly chosen. We therefore introduce Assumption (D3), which resembles Assumption (D1) but allows more general target functions  $f^{*} \in H^{s^{*}}(\Omega) \backslash \{0\}$ ,  $s^{*} > 0$ , that may lie outside of the RKHS.
(D3) Let  $\Omega = \mathbb{S}^d$  or let  $\Omega \subseteq \mathbb{R}^d$  be a bounded open Lipschitz domain. Let  $P_X$  be a distribution on  $\Omega$  with lower- and upper-bounded Lebesgue density. Consider i.i.d. data sets  $D = \{(x_1, y_1), \ldots, (x_n, y_n)\} \subseteq \Omega \times \mathbb{R}$ , where  $x_i \sim P_X$ ,  $f^*(\pmb{x}) = \mathbb{E}[y|\pmb{x}] \in H^{s^*}(\Omega) \setminus \{0\}$ ,  $s^* > 0$ , with  $\|f^*\|_{L^\infty(P_X)} < B_\infty$  for some constant  $B_\infty > 0$ ,  $\mathbb{E}y^2 < \infty$  and there are constants  $\sigma, L > 0$  such that
$$
\mathbb {E} \left[ | y - f ^ {*} (\boldsymbol {x}) | ^ {m} \mid \boldsymbol {x} \right] \leq \frac {1}{2} m! \sigma^ {2} L ^ {m - 2},
$$
for  $P_{X}$ -almost all  $\pmb{x} \in \Omega$  and all  $m \geq 2$ .
The above moment condition holds for additive Gaussian noise with variance  $\sigma^2 > 0$ . Hence Assumption (D1) is strictly stronger than Assumption (D3). The spike components  $\tilde{k}_{\gamma_n}$  can also be chosen as in Theorem G.1.
Theorem G.5. Assume Assumption (D3) holds and that the kernel components satisfy:
- the RKHS  $\tilde{\mathcal{H}}$  of  $\tilde{k}$  satisfies  $\tilde{\mathcal{H}} = H^s$  as sets with  $s > \max(s^*, d/2)$ ,
-  $\check{k}_{\gamma_n}$  denotes the Laplace kernel with a sequence of positive bandwidths  $(\gamma_n)$  fulfilling  $\gamma_n \leq n^{-\frac{2 + \alpha}{d}} \left( \left( \frac{11}{4} + \frac{\alpha}{2} \right) \ln n \right)^{-1}$ , where  $\alpha > 0$  is arbitrary.
Then there exists a constant  $C > 0$  independent of  $n$  and there is a sequence  $(\rho_{n})_{n\in \mathbb{N}}$  of order  $n^{-s / (s^{*} + d / 2)}$  such that the minimum-norm interpolant  $f_{\rho_n,\gamma_n}$  of the  $\rho_{n}$ -regularized spiky-smooth kernel sequence  $k_{n}\coloneqq k_{\rho_{n},\gamma_{n}}$  fulfills, with probability at least  $1 - 6n^{-(1\wedge \alpha)}$ , for  $n$  large enough,
$$
R _ {P} \left(f _ {\rho_ {n}, \gamma_ {n}}\right) - R _ {P} \left(f ^ {*}\right) \leq C n ^ {- \frac {s ^ {*}}{\left(s ^ {*} + d / 2\right)}} \log^ {2} (n).
$$
Proof. Step 1: Kernel ridge regression  $\hat{f}_{\rho_n}^{\tilde{k}}$  with optimal regularization achieves the desired convergence rate, with high probability.
We slightly modify the proof of Theorem 8. Instead of using (Steinwart, 2001, Theorem 3.11 or Example 4.6), we use results of Fischer and Steinwart (2020). Here we first note that in the case
$\mathcal{H} = H^{s}(\Omega), \Omega \subseteq \mathbb{R}^{d}$  as RKHSs, we could directly use (Fischer and Steinwart, 2020, Corollary 5). Since we only have equivalent norms and also want to consider  $\Omega = \mathbb{S}^{d}$ , see Lemma F.7, we need to resort to the underlying more general result (Fischer and Steinwart, 2020, Theorem 1). To this end, we first need to verify its Assumptions (EMB), (EVD), (SRC), and (MOM).
Step 1.1: Verifying (MOM). The moment condition (MOM) on the noise distributions holds since we assumed it in Assumption (D3).
Step 1.2: Simpler equivalent spaces. We verify the remaining conditions by analyzing them for a nicer equivalent RKHS  $\mathcal{H}$  and with uniform distribution  $\nu$  on  $\Omega$ . For the non-spherical case  $\Omega \subseteq \mathbb{R}^d$ , we choose  $\mathcal{H} \coloneqq H^s(\Omega)$ . For the case  $\Omega = \mathbb{S}^d$ , we choose  $\mathcal{H}$  as an RKHS associated to a dot-product kernel  $k$  with  $\mu_l = \Theta((l + 1)^{-2s})$ , such that  $\mathcal{H} \cong H^s \cong \tilde{\mathcal{H}}$  by Lemma B.1. In each case,  $\mathcal{H} = \tilde{\mathcal{H}}$  with equivalent norms, and  $L_2(\nu) = L_2(P_X)$  with equivalent norms since we assumed in (D3) that  $P_X$  has an upper- and lower-bounded density.
Step 1.3: Verifying  $(\mathbf{EVD}+)$ . It suffices to verify the eigenvalue decay condition  $(\mathrm{EVD}+)$  for  $\mathcal{H}$  and  $\nu$ , since Lemma F.10 and Lemma F.9 then allow to transfer it to  $\tilde{\mathcal{H}}$  and  $P_{X}$ .
For  $\Omega \subseteq \mathbb{R}^d$ , as pointed out in front of (Fischer and Steinwart, 2020, Corollary 5), it is well-known that  $\mathcal{H}$  satisfies the polynomial eigenvalue decay assumption (EVD+) for  $p \coloneqq \frac{d}{2s}$ .
For  $\Omega = \mathbb{S}^d$ , our definition of  $\mathcal{H}$  together with Lemma B.1 directly yields (EVD+) for  $\mathcal{H}$  and  $\nu$  with  $p := \frac{d}{2s}$ .
Step 1.4: Verifying (EMB) and (SRC). The remaining two conditions of (Fischer and Steinwart, 2020, Theorem 1) are stated in terms of so-called power spaces, which in turn can be described by interpolation spaces of the real method. We therefore quickly recall these spaces. To this end, let us assume that we have two Banach spaces  $E$  and  $F$  such that  $F \subset E$  and the corresponding inclusion map is continuous. Then the so-called  $K$ -functional of an  $x \in E$  is defined by
$$
K (x, t, E, F) := \inf  _ {y \in F} \left(t \| y \| _ {F} + \| x - y \| _ {E}\right), \qquad \qquad t > 0.
$$
For  $q\in (0,1)$  and  $x\in E$  we then define
$$
\| x \| _ {q, 2} ^ {2} := \int_ {0} ^ {\infty} t ^ {- 2 q - 1} K ^ {2} (x, t, E, F) d t
$$
and  $[E,F]_{q,2} \coloneqq \{x \in E : \| x\|_{q,2} < \infty\}$ . Let us now consider the cases  $(E,F) = (L_2(\nu),\mathcal{H})$  and  $(E,F) = (L_2(P_X),\tilde{\mathcal{H}})$ . Now, for a suitable constant  $C$ ,  $\mathcal{H}$  and  $\tilde{\mathcal{H}}$  are  $C$ -equivalent, and  $L_2(\nu)$  and  $L_2(P_x)$  are also  $C$ -equivalent. We then find
$$
C ^ {- 1} K (f, t, L _ {2} (\Omega), \mathcal {H}) \leq K (f, t, L _ {2} (\Omega), \tilde {\mathcal {H}}) \leq C K (f, t, L _ {2} (\Omega), \mathcal {H})
$$
for all  $t > 0$  and  $f\in L_2(\Omega)$ , and consequently we have
$$
[ L _ {2} (\nu), \mathcal {H} ] _ {q, 2} = [ L _ {2} (P _ {X}), \tilde {\mathcal {H}} ] _ {q, 2}
$$
for all  $q \in (0,1)$  with  $C$ -equivalent norms. Now, (Steinwart and Scovel, 2012, Theorem 4.6) shows that the power space  $[\mathcal{H}]_{\nu}^{q}$  defined in (Steinwart and Scovel, 2012, Equation (36)) satisfies
$$
[ \mathcal {H} ] _ {\nu} ^ {q} = [ L _ {2} (\nu), \mathcal {H} ] _ {q, 2}
$$
with equivalent norms, and an analogous result is true for  $\tilde{\mathcal{H}}$  and  $P_{X}$ . Moreover,  $\mathcal{H}$  is dense in  $L_{2}(\nu)$ , and therefore (Steinwart and Scovel, 2012, Equations (36) and (18)) together with (Steinwart and Scovel, 2012, Lemma 2.2) show  $[\mathcal{H}]_{\nu}^{0} = L_{2}(\nu)$  as spaces. Again, we analogously find  $[\tilde{\mathcal{H}}]_{P_{X}}^{0} = L_{2}(P_{X})$  Consequently, for all  $0 \leq q < 1$  we have
$$
[ \mathcal {H} ] _ {\nu} ^ {q} = [ \tilde {\mathcal {H}} ] _ {P _ {X}} ^ {q}
$$
with equivalent norms. From this we easily deduce that the Assumptions (EMB) and (SRC) are satisfied for  $(\tilde{\mathcal{H}}, P_X)$  if and only if they are satisfied for  $(\mathcal{H}, \nu)$ .
Step 1.4.1: Non-spherical case. Now, let  $\Omega \subseteq \mathbb{R}^d$ . Then, (EMB) and (SRC) are satisfied for  $(\mathcal{H},\nu)$  for  $\beta \coloneqq s^{*} / s$  and an arbitrary but fixed  $\alpha \in (p,\min \{1,p + \beta \})$  as outlined in front of (Fischer and Steinwart, 2020, Corollary 5). Applying Part ii) of (Fischer and Steinwart, 2020, Theorem 1) for
$\gamma = 0$  then shows that there exists a sequence  $(\rho_n)_{n \in \mathbb{N}}$  of order  $n^{-s / (s^* + d / 2)}$  and a constant  $K_1 > 0$  independent of  $n$  such that, for  $n$  large enough,
$$
P ^ {n} \left(D \in \left(\mathbb {R} ^ {d} \times \mathbb {R}\right) ^ {n} \mid \mathbb {E} _ {\boldsymbol {x}} \left(\hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x}) - f ^ {*} (\boldsymbol {x})\right) ^ {2} \geq K _ {1} n ^ {- \frac {s ^ {*}}{\left(s ^ {*} + d / 2\right)}} \log^ {2} (n)\right) \leq 4 n ^ {- 1}. \tag {G.3}
$$
Step 1.4.2: Spherical case. Suppose  $\Omega = \mathbb{S}^d$ . Using the differently normalized spherical harmonics  $Y_{l,i}$  and  $\tilde{Y}_{l,i}$  from Appendix B.3, we obtain for the power spaces:
$$
\begin{array}{l} [ \mathcal {H} ] _ {\nu} ^ {q} = \left\{\sum_ {l = 0} ^ {\infty} \sum_ {i = 1} ^ {N _ {l, d}} a _ {l i} \tilde {\mu} _ {i} ^ {q / 2} [ \tilde {Y} _ {l, i} ] _ {\sim} \mid (a _ {l i}) \in \ell_ {2} \right\} \\ = \left\{\sum_ {l = 0} ^ {\infty} \sum_ {i = 1} ^ {N _ {l, d}} a _ {l, i} (l + 1) ^ {- q s} \left[ Y _ {l, i} \right] _ {\sim} \mid \left(a _ {l i}\right) \in \ell_ {2} \right\} \\ = \left\{\sum_ {l = 0} ^ {\infty} \sum_ {i = 1} ^ {N _ {l, d}} b _ {l, i} \left[ Y _ {l, i} \right] \sim \mid \sum_ {l = 0} ^ {\infty} \sum_ {i = 1} ^ {N _ {l, d}} b _ {l, i} ^ {2} (l + 1) ^ {2 q s} <   \infty \right\} \\ = H ^ {q s} (\mathbb {S} ^ {d}), \\ \end{array}
$$
where the first equation follows from (Steinwart and Scovel, 2012, Eq. (36)), the second one from our definition of the  $\mu_l$  in Step 1.2, and the last one from (Hubbert et al., 2023, Section 3). Again, we can choose an arbitrary but fixed  $\alpha \in (p, \min\{1, p + \beta\})$ . Then,  $\alpha s > ps = d/2$ , which means that  $[\mathcal{H}]_{\nu}^{\alpha} = H^{\alpha s}$  is an RKHS with bounded kernel (De Vito et al., 2021, Theorem 8), hence the embedding condition (EMB) holds. Similarly, (SRC) holds for  $\beta := s^{*}/s$  and the result follows as above.
Step 2:  $\hat{f}_0^{k_n}$  and  $\hat{f}_{\rho_n}^{\tilde{k}}$  are close in  $L_{2}(P_{X})$ , with high probability.
Since  $\frac{s^*}{(s^* + d / 2)} < 1$ , it suffices to show that  $k_{n}$  fulfills, for some constant  $K_{2} > 0$
$$
P ^ {n} \left(D \in \left(\mathbb {R} ^ {d} \times \mathbb {R}\right) ^ {n} \mid \mathbb {E} _ {\boldsymbol {x}} \left(\hat {f} _ {0} ^ {k _ {n}} (\boldsymbol {x}) - \hat {f} _ {\rho_ {n}} ^ {\tilde {k}} (\boldsymbol {x})\right) ^ {2} \geq K _ {2} n ^ {- 1}\right) \leq 2 n ^ {- \alpha}. \tag {G.4}
$$
Since  $\gamma_{n} \leq n^{-\frac{2 + \alpha}{d}}\left(\left(\frac{11}{4} + \frac{\alpha}{2}\right) \ln n\right)^{-1}$ , it holds that  $|\check{k}_{\gamma_n}(\pmb{x}, \pmb{y})| \leq \varepsilon_n \coloneqq \rho_n n^{-\frac{5 + \alpha}{2}}$  (cf. Remark G.2). Then the product of all terms in Eq. (G.1) satisfies  $O(n^{4 + \alpha} \rho_n^{-2} \varepsilon_n^2) = o(n^{-1})$  with probability at least  $1 - 2n^{-\alpha}$ . On the same event, the bound on Eq. (G.2) remains of order  $O(n^{-1})$ , which shows Eq. (G.4). Combining (G.3) and (G.4) with the triangle inequality in  $L_2(P_X)$  concludes the proof.
Remark G.6 (Optimality of the rates). In the setting of Theorem G.5, we can apply (Fischer and Steinwart, 2020, Theorem 2) in order to obtain lower bounds on the achievable rates. We have already verified the conditions (MOM), (EVD+), (EMB), and (SRC) in the proof of Theorem G.5. In the case  $s^* > d/2$ , we have  $\beta = s^* / s > d / (2s) = p$  and can therefore choose  $\alpha \in (p, \beta)$  such that  $\beta > \alpha$ . Then, (Fischer and Steinwart, 2020, Theorem 2) yields a lower bound on the rate of the form (with constant probability)
$$
n ^ {- \frac {\beta}{\beta + p}} = n ^ {- \frac {s ^ {*}}{s ^ {*} + d / 2}},
$$
which matches the rates in Theorem G.5 up to log terms.
G.1 Auxiliary results for the proof of Theorem 8
The distributional Assumption (D2) immediately implies that the training points are separated with high probability.
Lemma G.7. Assume (D2) is fulfilled with  $\beta_{X} > 0$ . Then with probability at least  $1 - O(n^{-\alpha})$
$$
\min _ {i, j \in [ n ]: i \neq j} | \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} | \geq n ^ {- \frac {2 + \alpha}{\beta_ {X}}}.
$$
Proof. For any  $i \in [n]$ , the union bound implies
$$
P \left(\min  _ {j \in [ n ]: i \neq j} | \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} | \leq \delta\right) = P \left(\bigcup_ {j \in [ n ]: j \neq i} \left\{\boldsymbol {x} _ {j} \in B _ {\delta} (\boldsymbol {x} _ {i}) \right\}\right) \leq (n - 1) \phi (\delta).
$$
Another union bound yields
$$
P(\min_{i,j\in [n]:i\neq j}|\boldsymbol{x}_{i} - \boldsymbol{x}_{j}|\leq \delta)\leq n(n - 1)\phi (\delta).
$$
Choosing  $\delta_{\alpha}(n) = n^{-\frac{2 + \alpha}{\beta X}}$  yields  $\phi (\delta_{\alpha}(n)) = O(\frac{1}{n^{2 + \alpha}})$ , which concludes the proof.
The following lemma bounds  $\| \mathbf{A}^{-1} - \mathbf{B}^{-1} \|$  via  $\| \mathbf{A}^{-1} \|$  and  $\| \mathbf{A} - \mathbf{B} \|$ . Similar results can for example be found in (Horn and Johnson, 2013, Section 5.8).
Lemma G.8. Let  $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{n \times n}$  be invertible matrices and let  $\| \cdot \|$  be a submultiplicative matrix norm with  $\| I_n \| = 1$ . If  $\mathbf{A}$  and  $\mathbf{B}$  fulfill  $\| \mathbf{A}^{-1} \| \| \mathbf{A} - \mathbf{B} \| < 1$ , then it holds that
$$
\left\| \boldsymbol {B} ^ {- 1} - \boldsymbol {A} ^ {- 1} \right\| \leq \frac {\left\| \boldsymbol {A} ^ {- 1} \right\| ^ {2} \cdot \left\| \boldsymbol {A} - \boldsymbol {B} \right\|}{1 - \left\| \boldsymbol {A} ^ {- 1} \right\| \cdot \left\| \boldsymbol {A} - \boldsymbol {B} \right\|}.
$$
Proof. Because of  $\| A^{-1}(A - B) \| \leq \| A^{-1} \| \| A - B \| < 1$  we get
$$
\left\| \boldsymbol {I} - \boldsymbol {A} ^ {- 1} (\boldsymbol {A} - \boldsymbol {B}) \right\| \geq 1 - \left\| \boldsymbol {A} ^ {- 1} \right\| \left\| \boldsymbol {A} - \boldsymbol {B} \right\|.
$$
Writing  $\pmb{B} = \pmb{A}(\pmb{I} - \pmb{A}^{-1}(\pmb{A} - \pmb{B}))$  yields  $\pmb{B}^{-1} = (\pmb{I} - \pmb{A}^{-1}(\pmb{A} - \pmb{B}))^{-1}\pmb{A}^{-1}$  which implies
$$
\| B ^ {- 1} \| \leq \frac {\| A ^ {- 1} \|}{1 - \| A ^ {- 1} \| \| A - B \|}.
$$
Now write  $B^{-1} - A^{-1} = A^{-1}(A - B)B^{-1}$  to get
$$
\| \boldsymbol {B} ^ {- 1} - \boldsymbol {A} ^ {- 1} \| \leq \| \boldsymbol {A} ^ {- 1} \| \| \boldsymbol {A} - \boldsymbol {B} \| \| \boldsymbol {B} ^ {- 1} \|.
$$
Combining the last two inequalities concludes the proof.
G.2 RKHS norm bounds
Here we show that if  $\tilde{k}$  and  $\tilde{k}_{\gamma}$  have RKHS equivalent to some Sobolev space  $H^{s}$ ,  $s > d / 2$ , then the RKHS of the spiky-smooth kernel  $k_{\rho, \gamma}$  is also equivalent to  $H^{s}$ , for any fixed  $\rho, \gamma > 0$ . Hence all members of the spiky-smooth kernel sequence may have RKHS equivalent to a Sobolev space  $H^{s}$  and are individually inconsistent due to Theorem 1; yet the sequence is consistent. This shows that when arguing about generalization properties based on RKHS equivalence, the constants matter and the narrative that depth does not matter in the NTK regime as in Bietti and Bach (2021) is too simplified.