SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
S _ {l} = \sum_ {i = 0} ^ {l} \Theta ((\pi (i) + 1) ^ {d - 1}) = \sum_ {i = 0} ^ {l} \Theta ((i + 1) ^ {d - 1}) = \Theta ((l + 1) ^ {d}).
$$
Now, let  $i \geq 1$  and let  $l \in \mathbb{N}_0$  such that  $S_{l-1} < i \leq S_l$ . We have  $i \geq \Omega(l^d)$ , and  $i \leq O((l+1)^d)$  which implies  $i = \Theta((l+1)^d)$  since  $i \geq 1$ . Therefore,
$$
\lambda_ {i} = \tilde {\mu} _ {\pi (l)} = \Theta ((\pi (l) + 1) ^ {- 2 s}) = \Theta ((l + 1) ^ {- 2 s}) = \Theta \left(i ^ {- 2 s / d}\right).
$$
B.4 Neural kernels
Several NTK and NNGP kernels have RKHSs that are equivalent to Sobolev spaces on  $\mathbb{S}^d$ . In the following cases, we can deduct this from known results:
- Consider fully-connected NNs with  $L \geq 3$  layers without biases and the activation function  $\varphi(x) = \max\{0, x\}^m$ ,  $m \in \mathbb{N}_0$ . Especially, the case  $m = 1$  corresponds to the ReLU activation. Vakili et al. (2021) generalize the result by Bietti and Bach (2021) from  $m = 1$  to  $m \geq 1$ , showing that the NTK-RKHS is equivalent to  $H^s(\mathbb{S}^d)$  for  $s = (d + 2m - 1)/2$  and the NNGP-RKHS is equivalent to  $H^s(\mathbb{S}^d)$  for  $s = (d + 2m + 1)/2$ . For  $m = 0$ , Bietti and Bach (2021) essentially show that the NNGP-RKHS is equivalent to  $H^s(\mathbb{S}^d)$  for  $s = (d + 2^{2 - L})/2$ . However, all of the aforementioned results have the problem that the main theorem by Bietti and Bach (2021) allows for the possibility that finitely many  $\mu_l$  are zero, which can change the RKHS. Using our Lemma B.2 below, it follows that all  $\mu_l$  are in fact nonzero for NNGPs and NTKs since they are kernels in every dimension  $d$  using the same function  $\kappa$  independent of the dimension. Hence, the equivalences to Sobolev spaces stated before are correct.
- Chen and Xu (2021) prove that the RKHS of the NTK corresponding to fully-connected ReLU NNs with zero-initialized biases and  $L \geq 2$  (as opposed to no biases and  $L \geq 3$  above) layers is equivalent to the RKHS of the Laplace kernel on the sphere. Since the Laplace kernel is a Matérn kernel of order  $\nu = 1/2$  (see e.g. Section 4.2 in Rasmussen and Williams (2005)), we can use Proposition 5.2 of Hubbert et al. (2023) to obtain equivalence to  $H^{s}(\mathbb{S}^{d})$  with  $s = (d + 1)/2$ . Alternatively, we can obtain the RKHS of the Laplace kernel from Bietti and Bach (2021) combined with Lemma B.2.
Bietti and Bach (2021) also show that under an integrability condition on the derivatives,  $C^\infty$  activations induce NTK and NNGP kernels whose RKHSs are smaller than every Sobolev space.
Lemma B.2 (Guaranteeing non-zero eigenvalues). Let  $\kappa : [-1, 1] \to \mathbb{R}$ , let  $d \geq 1$ , and let
$$
k _ {d}: \mathbb {S} ^ {d} \times \mathbb {S} ^ {d}, k _ {d} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) := \kappa \left(\left\langle \boldsymbol {x}, \boldsymbol {x} ^ {\prime} \right\rangle\right)
$$
$$
k _ {d + 2}: \mathbb {S} ^ {d + 2} \times \mathbb {S} ^ {d + 2}, k _ {d + 2} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) := \kappa \left(\left\langle \boldsymbol {x}, \boldsymbol {x} ^ {\prime} \right\rangle\right).
$$
Suppose that  $k_{d+2}$  is a kernel. Then,  $k_d$  is a kernel. Moreover, if the corresponding eigenvalues  $\mu_l$  of  $k_d$  satisfy  $\mu_l > 0$  for infinitely many  $l$ , then they satisfy  $\mu_l > 0$  for all  $l \in \mathbb{N}_0$ .
Proof. The fact that  $k_d$  is a kernel follows directly from the inclusion  $\Phi_{d+2} \subseteq \Phi_d$  mentioned in Gneiting (2013). For  $D \in \{d, d+2\}$ , let  $\mu_{l,d}$  be the sequence of eigenvalues  $\mu_l$  associated with  $k_D$ . Then, as mentioned for example by Hubbert et al. (2023), the Schoenberg coefficients  $b_{l,d}$  satisfy
$$
b _ {l, d} = \frac {\Gamma \left(\frac {d + 1}{2}\right) N _ {m , d} \mu_ {l , d}}{2 \pi^ {(d + 1) / 2}}.
$$
Especially, the Schoenberg coefficients  $b_{l,d}$  have the same sign as the eigenvalues  $\mu_{l,d}$ . We use
$$
0 \leq b _ {l, d + 2} = \left\{ \begin{array}{l l} b _ {l, d} - \frac {1}{2} b _ {l + 2, d} & , l = 0 \text {a n d} d = 1 \\ \frac {1}{2} (l + 1) (b _ {l, d} - b _ {l + 2, d}) & , l \geq 1 \text {a n d} d = 1 \\ \frac {(l + d - 1) (l + d)}{d (2 l + d - 1)} b _ {l, d} - \frac {(l + 1) (l + 2)}{d (2 l + d + 3)} b _ {l + 2, d} & , d \geq 2  , \end{array} \right.
$$
where the inequality follows from the fact that  $k_{d+2}$  is a kernel and the equality is the statement of Corollary 3 by Gneiting (2013). In any of the three cases,  $b_{l+2,d} > 0$  implies  $b_{l,d} > 0$ . Hence, if  $b_{l,d} > 0$  for infinitely many  $l$ , then  $b_{l,d} > 0$  for all  $l$ , which implies  $\mu_{l,d} > 0$  for all  $l$ .
C Gradient flow and gradient descent with kernels
C.1 Derivation of gradient flow and gradient descent
Here, we derive expressions for gradient flow and gradient descent in the RKHS for the regularized loss
$$
L (f) := \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - f (\boldsymbol {x} _ {i})\right) ^ {2} + \rho \| f \| _ {\mathcal {H} _ {k}} ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - \langle k (\boldsymbol {x} _ {i}, \cdot), f \rangle_ {\mathcal {H} _ {k}}\right) ^ {2} + \rho \langle f, f \rangle_ {\mathcal {H} _ {k}} ^ {2}.
$$
Note that we will take derivatives in the RKHS with respect to  $f$ , which is different from taking derivatives w.r.t. the coefficients  $c$  in a model  $f(\pmb{x}) = c^\top k(\pmb{X},\pmb{x})$ .
In the RKHS-Norm, the Fréchet derivative of  $L$  is
$$
\frac {\partial L (f)}{f} = \frac {2}{n} \sum_ {i = 1} ^ {n} \left(f \left(\boldsymbol {x} _ {i}\right) - y _ {i}\right) \langle k \left(\boldsymbol {x} _ {i}, \cdot\right), \cdot \rangle_ {\mathcal {H} _ {k}} + 2 \rho \langle f, \cdot \rangle_ {\mathcal {H} _ {k}},
$$
which is represented in  $\mathcal{H}_k$  by
$$
L ^ {\prime} (f) = \frac {2}{n} \sum_ {i = 1} ^ {n} \left(f \left(\boldsymbol {x} _ {i}\right) - y _ {i}\right) k \left(\boldsymbol {x} _ {i}, \cdot\right) + 2 \rho f.
$$
Now assume that  $f = \sum_{i=1}^{n} a_{i} k(\pmb{x}_{i}, \cdot) = \pmb{a}^{\top} k(\pmb{X}, \cdot)$ . Then,
$$
\begin{array}{l} L ^ {\prime} (f) = \frac {2}{n} \sum_ {i = 1} ^ {n} \left(\boldsymbol {a} ^ {\top} k (\boldsymbol {X}, \boldsymbol {x} _ {i}) - y _ {i}\right) k \left(\boldsymbol {x} _ {i}, \cdot\right) + 2 \rho \boldsymbol {a} ^ {\top} k (\boldsymbol {X}, \cdot) \\ = \frac {2}{n} \left(\boldsymbol {a} ^ {\top} k (\boldsymbol {X}, \boldsymbol {X}) k (\boldsymbol {X}, \cdot) - \boldsymbol {y} ^ {\top} k (\boldsymbol {X}, \cdot) + \rho n \boldsymbol {a} ^ {\top} k (\boldsymbol {X}, \cdot)\right) \\ = \frac {2}{n} \left(\left(k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}\right) \boldsymbol {a} - \boldsymbol {y}\right) ^ {\top} k (\boldsymbol {X}, \cdot). \\ \end{array}
$$
Especially, under gradient flow of  $f$ , the coefficients  $a$  follow the dynamics
$$
\dot {\boldsymbol {a}} (t) = \frac {2}{n} (\boldsymbol {y} - (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}) \boldsymbol {a} (t)),
$$
which is solved for  $\pmb{a}(0) = 0$  by
$$
\boldsymbol {a} (t) = \left(\boldsymbol {I} _ {n} - e ^ {- \frac {2}{n} t (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n})}\right) \left(k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {y},
$$
which is the closed form expression (1) of  $f_{t,\rho}$ .
For gradient descent, assuming that  $f_{t,\rho}^{\mathrm{GD}} = \pmb{c}_{t,\rho}^{\top}k(\pmb {X},\cdot)$  , we have
$$
\begin{array}{l} f _ {t + 1, \rho} ^ {\mathrm {G D}} = f _ {t, \rho} ^ {\mathrm {G D}} - \eta_ {t} L ^ {\prime} \left(f _ {t, \rho} ^ {\mathrm {G D}}\right) = \boldsymbol {c} _ {t, \rho} ^ {\top} k (\boldsymbol {X}, \cdot) - \eta_ {t} \frac {2}{n} \left(\left(k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}\right) \boldsymbol {c} _ {t, \rho} - \boldsymbol {y}\right) ^ {\top} k (\boldsymbol {X}, \cdot) \\ = \left(\boldsymbol {c} _ {t, \rho} + \eta_ {t} \frac {2}{n} \left(\boldsymbol {y} - (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}) \boldsymbol {c} _ {t, \rho}\right)\right) ^ {\top} k (\boldsymbol {X}, \cdot) \\ \end{array}
$$
If  $f_{0,\rho}^{\mathrm{GD}} \equiv 0$ , the coefficients evolve as  $c_0 = 0$  and
$$
\boldsymbol {c} _ {t + 1, \rho} = \boldsymbol {c} _ {t, \rho} + \eta_ {t} \frac {2}{n} (\boldsymbol {y} - (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}) \boldsymbol {c} _ {t, \rho}).
$$
For an analysis of gradient descent for kernel regression with  $\rho = 0$ , we refer to, e.g., Yao et al. (2007).
C.2 Gradient flow and gradient descent initialized at 0 have monotonically growing  $\mathcal{H}$ -norm
In the following proposition we show that under gradient flow and gradient descent with sufficiently small learning rates initialized at 0, the RKHS norm grows monotonically with time  $t$ . This immediately implies that Assumption (N) with  $C_{\mathrm{norm}} = 1$  holds for all estimators  $f_{t,\rho}$  from (1).
(i) For any  $t\in [0,\infty ]$  and any  $\rho \geq 0$ ,  $f_{t,\rho}$  from (1) fulfills Assumption (N) with  $C_{\mathrm{norm}} = 1$ .
(ii) For any  $t \in \bar{\mathbb{N}}_0 \cup \{\infty\}$  and any  $\rho \geq 0$ , with sufficiently small fixed learning rate  $0 \leq \eta \leq \frac{1}{2(\rho + \lambda_{\max}(k(\pmb{X},\pmb{X})) / n)}$ ,  $f_{t,\rho}^{\mathrm{GD}}$  fulfills Assumption (N) with  $C_{\mathrm{norm}} = 1$ .
Proof. Proof of (i):
We write  $f_{t,\rho}(\pmb{x}) = k(\pmb{x},\pmb{X})\pmb{c}_{t,\rho}$ , where  $\pmb{c}_{t,\rho} \coloneqq A_{t,\rho}(\pmb{X})\pmb{y}$ . We now show that the RKHS-norm of  $f_{t,\rho}$  grows monotonically in  $t$ , by using the eigendecomposition  $k(\pmb{X},\pmb{X}) = \pmb{U}\Lambda \pmb{U}^{\top}$ , where  $\Lambda = \mathrm{diag}(\lambda_1, \dots, \lambda_n) \in \mathbb{R}^{n \times n}$  is diagonal and  $\pmb{U} \in \mathbb{R}^{n \times n}$  is orthonormal, and writing  $\tilde{\pmb{y}} \coloneqq \pmb{U}^{\top}\pmb{y}$ . Then it holds that
$$
\begin{array}{l} \left\| f _ {t, \rho} \right\| _ {\mathcal {H}} ^ {2} = \left(\boldsymbol {c} _ {t, \rho}\right) ^ {\top} k (\boldsymbol {X}, \boldsymbol {X}) \boldsymbol {c} _ {t, \rho} = \tilde {\boldsymbol {y}} ^ {\top} \left(\Lambda + \rho n \boldsymbol {I} _ {n}\right) ^ {- 1} \left(\boldsymbol {I} _ {n} - \exp \left(- \frac {2 t}{n} (\Lambda + \rho n \boldsymbol {I} _ {n})\right)\right) \Lambda \cdot \\ \left(\boldsymbol {I} _ {n} - \exp \left(- \frac {2 t}{n} (\Lambda + \rho n \boldsymbol {I} _ {n})\right)\right) (\Lambda + \rho n \boldsymbol {I} _ {n}) ^ {- 1} \tilde {\boldsymbol {y}} \\ = \sum_ {k = 1, \lambda_ {k} + \rho n > 0} ^ {n} \tilde {y} _ {k} ^ {2} \underbrace {\frac {\lambda_ {k}}{(\lambda_ {k} + \rho n) ^ {2}}} _ {\leq 1 / \lambda_ {k}} \underbrace {\left(1 - \exp \left(- \frac {2 t}{n} (\lambda_ {k} + \rho n)\right)\right)} _ {\leq 1} \\ \leq \sum_ {k = 1, \lambda_ {k} > 0} ^ {n} \tilde {y} _ {k} ^ {2} \frac {1}{\lambda_ {k}} = \| f _ {\infty , 0} \| _ {\mathcal {H}} ^ {2}. \\ \end{array}
$$
Expanding the iteration in the definition of  $c_{t,\rho}$  yields
$$
\boldsymbol {c} _ {t + 1, \rho} = \sum_ {i = 0} ^ {t} \prod_ {j = 0} ^ {t - i - 1} \left(\boldsymbol {I} - \frac {2 \eta_ {t - j}}{n} (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I})\right) \frac {2 \eta_ {i}}{n} \boldsymbol {y}.
$$
We again use the eigendecomposition  $k(\mathbf{X}, \mathbf{X}) = \mathbf{U} \Lambda \mathbf{U}^{\top}$ , where  $\Lambda = \mathrm{diag}(\lambda_1, \dots, \lambda_n) \in \mathbb{R}^{n \times n}$  is diagonal and  $\mathbf{U} \in \mathbb{R}^{n \times n}$  is orthonormal, and write  $\tilde{\pmb{y}} := \mathbf{U}^{\top} \pmb{y}$ . Then, using sufficiently small learning rates  $0 \leq \eta_t \leq \frac{1}{2(\rho + \lambda_{\max}(k(\mathbf{X}, \mathbf{X})) / n)}$  in all time steps  $t \in \mathbb{N}$ , it holds that
$$
\begin{array}{l} \| f _ {t, \rho} ^ {\mathrm {G D}} \| _ {\mathcal {H}} ^ {2} \\ = (\boldsymbol {c} _ {t, \rho}) ^ {\top} k (\boldsymbol {X}, \boldsymbol {X}) \boldsymbol {c} _ {t, \rho} \\ = \tilde {\boldsymbol {y}} ^ {\top} \left(\sum_ {i = 0} ^ {t} \frac {2 \eta_ {i}}{n} \prod_ {j = 0} ^ {t - i - 1} \left((1 - 2 \eta_ {t - j} \rho) \boldsymbol {I} - \frac {2 \eta_ {t - j}}{n} \Lambda\right)\right) \Lambda \left(\sum_ {i = 0} ^ {t} \frac {2 \eta_ {i}}{n} \prod_ {j = 0} ^ {t - i - 1} \left((1 - 2 \eta_ {t - j} \rho) \boldsymbol {I} - \frac {2 \eta_ {t - j}}{n} \Lambda\right)\right) \tilde {\boldsymbol {y}} \\ = \sum_ {k = 1} ^ {n} \underbrace {\tilde {y} _ {k} ^ {2} \lambda_ {k}} _ {\geq 0} \left(\sum_ {i = 0} ^ {t} \frac {2 \eta_ {i}}{n} \prod_ {j = 0} ^ {t - i - 1} \underbrace {(1 - 2 \eta_ {t - j} (\rho + \lambda_ {k} / n))} _ {\in [ 0, 1 ]}\right) ^ {2}. \tag {C.1} \\ \end{array}
$$
The last display shows that  $\| f_{t,\rho}^{\mathrm{GD}}\|_{\mathcal{H}}^{2}$  grows monotonically in  $t$ , strictly monotonically if  $\eta_t \in (0, \frac{1}{2(\rho + \lambda_{\max}(k(\pmb{X},\pmb{X})) / n)})$  holds for all  $t$ . It also shows that if  $\rho' \geq \rho$  then  $\| f_{t,\rho'}^{\mathrm{GD}}\|_{\mathcal{H}} \leq \| f_{t,\rho}^{\mathrm{GD}}\|_{\mathcal{H}}$  for any  $t \in \mathbb{N} \cup \{\infty\}$ . To see that  $\| f_{t,\rho}^{\mathrm{GD}}\|_{\mathcal{H}}^{2} \leq \| f_{\infty,0}\|_{\mathcal{H}}^{2}$  for all  $t \in \mathbb{N} \cup \{\infty\}$  and all  $\rho \geq 0$ , observe that with fixed learning rates  $\eta_t = \eta \in (0, \frac{1}{2(\rho + \lambda_{\max}(k(\pmb{X},\pmb{X})) / n)}) \subseteq (0, \frac{1}{2\lambda_{\max}(k(\pmb{X},\pmb{X})) / n})$ , for all  $t \in \mathbb{N} \cup \{\infty\}$  it holds that
$$
\sum_ {i = 0} ^ {t} \frac {2 \eta_ {i}}{n} \prod_ {j = 0} ^ {t - i - 1} \left(1 - 2 \eta_ {t - j} \lambda_ {k} / n\right) = \frac {2 \eta}{n} \sum_ {i = 0} ^ {t} \left(1 - 2 \eta \lambda_ {k} / n\right) ^ {t - i}
$$
$$
= \frac {2 \eta}{n} \sum_ {i = 0} ^ {t} (1 - 2 \eta \lambda_ {k} / n) ^ {i} = \frac {2 \eta}{n} \frac {1 - (1 - 2 \eta \lambda_ {k} / n) ^ {t + 1}}{2 \eta \lambda_ {k} / n} \leq \frac {1}{\lambda_ {k}}.
$$
Since it suffices to consider the case  $\rho \to 0$ , using the above derivation in (C.1) yields  $\| f_{t,\rho}^{\mathrm{GD}}\|_{\mathcal{H}}^{2}\leq$ $\| f_{\infty ,0}\|_{\mathcal{H}}^{2}$  for all  $t\in \mathbb{N}$ , which concludes the proof.
Our goal in this section is to prove Theorem D.1, which can be seen as a generalization of Theorem 1 to varying bandwidths. To be able to speak of bandwidths, we need to consider translation-invariant kernels. Although Theorem 1 is formulated for general kernels with Sobolev RKHS, it follows from Theorem D.1 since we can always find, for a fixed bandwidth, a translation-invariant kernel with equivalent RKHS, such that only the constant  $C_{\mathrm{norm}}$  changes in the theorem statement.
To generate the RKHS  $H^{s}$ , Buchholz (2022) uses the translation-invariant kernel  $k^{B}(\pmb{x},\pmb{y}) = u^{B}(\pmb{x} - \pmb{y})$  defined via its Fourier transform  $\hat{u}^{B}(\pmb{\xi}) = (1 + |\pmb{\xi}|^{2})^{-s}$ . Adapting the bandwidth, the kernel is then normalized in the usual  $L_{1}$ -sense,
$$
k _ {\gamma} ^ {B} (\boldsymbol {x}, \boldsymbol {y}) = \gamma^ {- d} u ^ {B} ((\boldsymbol {x} - \boldsymbol {y}) / \gamma). \tag {D.1}
$$
Theorem D.1 (Inconsistency of overfitting estimators). Let assumptions (D1) and (K) hold. Let  $c_{\mathrm{fit}} \in (0, 1]$  and  $C_{\mathrm{norm}} > 0$ . Then, there exist  $c > 0$  and  $n_0 \in \mathbb{N}$  such that the following holds for all  $n \geq n_0$  with probability  $1 - O(1/n)$  over the draw of the data set  $D$  with  $n$  samples: For every function  $f \in \mathcal{H}_k$  with
$(O)$ $\frac{1}{n}\sum_{i = 1}^{n}(f(x_i) - y_i)^2\leq (1 - c_{\mathrm{fit}})\cdot \sigma^2$  (training error below Bayes risk) and
(N)  $\| f\|_{\mathcal{H}_k}\leq C_{\mathrm{norm}}\| f_{\infty ,0}\|_{\mathcal{H}_k}$  (norm comparable to minimum-norm interpolant (1)),
the excess risk satisfies
$$
R _ {P} (f) - R _ {P} \left(f ^ {*}\right) \geq c > 0. \tag {D.2}
$$
If  $k_{\gamma}$  denotes a  $L_1$ -normalized translation-invariant kernel with bandwidth  $\gamma > 0$ , i.e. there exists a  $q: \mathbb{R}^d \to \mathbb{R}$  such that  $k_{\gamma}(x, y) = \gamma^{-d} q(\frac{x - y}{\gamma})$ , then inequality (D.2) holds with  $c$  independent of the sequence of bandwidths  $(\gamma_n)_{n \in \mathbb{N}} \subseteq (0, 1)$ , as long as  $f_D$  fulfills (N) for the sequence  $(\mathcal{H}_{\gamma_n})_{n \in \mathbb{N}}$  with constant  $C_{\mathrm{norm}} > 0$ .
Proof. By assumption, the RKHS norm  $\| \cdot \|_{\mathcal{H}_k}$  induced by the kernel  $k$  (or  $k_{\gamma}$  if we allow bandwidth adaptation) is equivalent to the RKHS norm  $\| \cdot \|_{\mathcal{H}_{\gamma}}$  induced by a kernel of the form (D.1) with an arbitrary but fixed choice of bandwidth  $\gamma \in (0,1)$ , which means that there exists a constant  $C_{\gamma} > 0$  such that  $\frac{1}{C_{\gamma}} \| f\|_{\mathcal{H}_{\gamma}} \leq \| f\|_{\mathcal{H}_k} \leq C_{\gamma}\| f\|_{\mathcal{H}_{\gamma}}$  for all  $f \in \mathcal{H}_k$ . Hence the minimum-norm interpolant  $g_{D,\gamma}$  in  $\mathcal{H}_{\gamma}$  satisfies
$$
\| f _ {D} \| _ {\mathcal {H} _ {\gamma}} \leq C _ {\gamma} \| f _ {D} \| _ {\mathcal {H} _ {k}} \leq C _ {\gamma} C _ {\text {n o r m}} \| g _ {D} \| _ {\mathcal {H} _ {k}} \leq C _ {\gamma} C _ {\text {n o r m}} \| g _ {D, \gamma} \| _ {\mathcal {H} _ {k}} \leq C _ {\gamma} ^ {2} C _ {\text {n o r m}} \| g _ {D, \gamma} \| _ {\mathcal {H} _ {\gamma}},
$$
where  $\| g_D\|_{\mathcal{H}_k}\leq \| g_{D,\gamma}\|_{\mathcal{H}_k}$  because,  $g_{D}$  is the minimum-norm interpolant in  $\mathcal{H}_k$
Now consider the RKHS norm  $\| \cdot \|_{\tilde{\mathcal{H}}_\gamma}$  of a translation-invariant kernel  $k_{\gamma}$ . Then the functions  $\{h_p(x) = e^{ip\cdot x}\}_{p\in \mathbb{R}^d}$  are eigenfunctions of the kernel's integral operator, so that the RKHS norm can be written as (Rakhlin and Zhai, 2019)
$$
\| f \| _ {\tilde {\mathcal {H}} _ {\gamma}} ^ {2} = \frac {1}{(2 \pi) ^ {d}} \int_ {\mathbb {R} ^ {d}} \frac {| \hat {f} (\omega) | ^ {2}}{\hat {q} (\omega)} d \omega ,
$$
where  $\hat{f}$  denotes the Fourier transform of  $f$ .
By assumption we know that there exists a  $C_{\gamma_0} > 0$  such that  $\frac{1}{C_{\gamma_0}} \| f \|_{\mathcal{H}_{\gamma_0}} \leq \| f \|_{\tilde{\mathcal{H}}_{\gamma_0}} \leq C_{\gamma_0} \| f \|_{\mathcal{H}_{\gamma_0}}$  holds for some fixed bandwidth  $\gamma_0 > 0$ , then substituting by  $\tilde{\omega} = \frac{\gamma}{\gamma_0} \omega$  yields
$$
\| f \| _ {\tilde {\mathcal {H}} _ {\gamma}} = \frac {1}{(2 \pi) ^ {d}} \int_ {R ^ {d}} \frac {| \hat {f} (\omega) | ^ {2}}{\hat {q} _ {1} (\gamma \omega)} d \omega = \frac {1}{(2 \pi) ^ {d}} \int_ {R ^ {d}} \frac {| \hat {f} (\frac {\gamma_ {0}}{\gamma} \tilde {\omega}) | ^ {2}}{\hat {q} _ {1} (\gamma_ {0} \tilde {\omega})} \left(\frac {\gamma_ {0}}{\gamma}\right) ^ {d} d \tilde {\omega} = \| \tilde {f} \| _ {\tilde {\mathcal {H}} _ {\gamma_ {0}}}
$$
$$
\leq C _ {\gamma_ {0}} \| \tilde {f} \| _ {\mathcal {H} _ {\gamma_ {0}}} = \frac {C _ {\gamma_ {0}}}{(2 \pi) ^ {d}} \int_ {R ^ {d}} \frac {| \hat {f} (\frac {\gamma_ {0}}{\gamma} \tilde {\omega}) | ^ {2}}{\hat {q} _ {2} (\gamma_ {0} \tilde {\omega})} \left(\frac {\gamma_ {0}}{\gamma}\right) ^ {d} d \tilde {\omega} = \frac {C _ {\gamma_ {0}}}{(2 \pi) ^ {d}} \int_ {R ^ {d}} \frac {| \hat {f} (\omega) | ^ {2}}{\hat {q} _ {2} (\gamma \omega)} d \omega = C _ {\gamma_ {0}} \| f \| _ {\mathcal {H} _ {\gamma}}.
$$
In the same way we get  $\frac{1}{C_{\gamma_0}} \| f \|_{\mathcal{H}_{\gamma}} \leq \| f \|_{\tilde{\mathcal{H}}_{\gamma}}$  for arbitrary  $\gamma \in (0, 1)$ . This shows that the constant  $C_{\gamma_0}$ , that quantifies the equivalence between  $\| \cdot \|_{\mathcal{H}_{\gamma}}$  and  $\| \cdot \|_{\tilde{\mathcal{H}}_{\gamma}}$  does not depend on the bandwidth  $\gamma$ . Finally Proposition D.4, Proposition D.2 and Remark D.3 together yield the result.
The following proposition generalizes the inconsistency result for large bandwidths, Proposition 4 in Buchholz (2022), beyond interpolating estimators to estimators that overfit at least an arbitrary constant fraction beyond the Bayes risk and whose RKHS norm is at most a constant factor larger than the RKHS norm of the minimum-norm interpolant. Compared to Rakhlin and Zhai (2019), Buchholz gets a statement in probability over the draw of a training set  $D$  and less restrictive assumptions on the domain  $\Omega$  and dimension  $d$ .
Proposition D.2 (Inconsistency for large bandwidths). Let  $c_{\mathrm{fit}} \in (0,1]$  and  $C_{\mathrm{norm}} > 0$ . Let the data set  $D = \{(x_1,y_1),\ldots ,(x_n,y_n)\}$  be drawn i.i.d. from a distribution  $P$  that fulfills Assumption (D1), let  $g_{D,\gamma}$  be the minimum-norm interpolant in  $\mathcal{H} \coloneqq \mathcal{H}_{\gamma}$  with respect to the kernel (D.1) for a bandwidth  $\gamma >0$ . Then, for every  $A > 0$ , there exist  $c > 0$  and  $n_0 \in \mathbb{N}$  such that the following holds for all  $n \geq n_0$  with probability  $1 - O(1 / n)$  over the draw of the data set  $D$  with  $n$  samples:
For every function  $f \in \mathcal{H}$  that fulfills Assumption (O) with  $c_{\mathrm{fit}}$  and Assumption (N) with  $C_{\mathrm{norm}}$  the excess risk satisfies
$$
\mathbb {E} _ {\boldsymbol {x}} \left(f (\boldsymbol {x}) - f ^ {*} (\boldsymbol {x})\right) ^ {2} \geq c > 0,
$$
where  $c$  depends neither on  $n$  nor on  $1 > \gamma > An^{-1/d} > 0$ .
Remark D.3. Proposition D.2 holds for any kernel that fulfills Assumption (K). The reason is that any kernel  $k$  that fulfills assumption (K) and the kernel defined in (D.1) have the same RKHS and equivalent norms. Therefore every function  $f \in \mathcal{H}_k = \mathcal{H}_{\gamma}$  (equality as sets) that fulfills Assumptions (O) and (N) for the kernel  $k$  also fulfills Assumptions (O) and (N) with an adapted constant  $C_{\mathrm{norm}}$  for the kernel (D.1).
Proof. Step 1: Generalizing the procedure in Buchholz (2022).
We write  $[n] = \{1, \dots, n\}$  and follow the proof of Proposition 4 in Buchholz (2022). Define  $u(\pmb{x}) = f(\pmb{x}) - f^{*}(\pmb{x})$ . We need to show that with probability at least  $1 - O(n^{-1})$  over the draw of  $D$  it holds that  $\| u \|_{L^2(P_X)} \geq c > 0$ , where  $c$  depends neither on  $n$  nor on  $\gamma$ .
For this purpose we show that with probability at least  $1 - 3n^{-1}$  over the draw of  $D$  there exist a constants  $c^{\prime \prime},\kappa^{\prime \prime} > 0$  depending only on  $c_{\mathrm{fit}}$  and a subset  $\mathcal{P}''\subseteq [n]$  with  $|\mathcal{P}''|\geq \lfloor \kappa ''\cdot n\rfloor$  such that
$$
\left| f \left(\boldsymbol {x} _ {i}\right) - f ^ {*} \left(\boldsymbol {x} _ {i}\right) \right| \geq c ^ {\prime \prime} > 0 \text {h o l d s f o r a l l} i \in \mathcal {P} ^ {\prime \prime}. \tag {D.3}
$$
Then via Lemma 7 in Buchholz (2022) as well as Lemma D.7 we can choose a large subset  $\mathcal{P}^{\prime \prime \prime} \subseteq [n]$  of the training point indices with  $|\mathcal{P}^{\prime \prime \prime}| \geq n - |\mathcal{P}^{\prime \prime}| / 2$ , such that the  $\pmb{x}_i$  for  $i \in \mathcal{P}^{\prime \prime}$  are well-separated in the sense that  $\min_{\{i,j \in \mathcal{P}^{\prime \prime \prime}, i \neq j\}} \| \pmb{x}_i - \pmb{x}_j \| \geq d_{\min}$  with  $d_{\min} := c^{\prime \prime \prime} n^{-1 / d}$ , where  $c^{\prime \prime \prime}$  depends on  $c_{\mathrm{fit}}, d$ , the upper bound on the Lebesgue density  $C_u$  and on the smoothness of the RKHS  $s$ . Then the intersection  $\mathcal{P}^{\prime \prime} \cap \mathcal{P}^{\prime \prime \prime}$  contains at least  $\frac{|\mathcal{P}^{\prime \prime}|}{2}$  points. Now we can replace  $\mathcal{P}'$  in the proof of Proposition 4 for  $s \in \mathbb{N}$  in Buchholz (2022) by the intersection  $\mathcal{P}^{\prime \prime} \cap \mathcal{P}^{\prime \prime \prime}$ . The rest of the proof applies without modification, where (42) holds by our assumption  $\| f \|_{\mathcal{H}} \leq C_{\mathcal{H}} \| g_D \|_{\mathcal{H}}$ . Our modifications do not affect Buchholz' arguments for the extension to  $s \notin \mathbb{N}$ .
Step 2: The existence of  $\mathcal{P}''$ .