SlowGuess/NeurIPS_2023_OCR huggingface dataset:
James Benjamin Simon, Sajant Anand, and Mike Deweese. Reverse engineering the neural tangent kernel. In International Conference on Machine Learning (ICML), 2022.
Ingo Steinwart. Consistency of support vector machines and other regularized kernel machines. IEEE Transactions on Information Theory, 2001.
Ingo Steinwart. A short note on the comparison of interpolation widths, entropy numbers, and Kolmogorov widths. J. Approx. Theory, 215:13-27, 2017.
Ingo Steinwart and Andreas Christmann. Support Vector Machines. Springer New York, 2008.
Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: on the interaction between measures, kernels, and RKHSs. Constr. Approx., 35:363-417, 2012.
Ingo Steinwart, Don R. Hush, and Clint Scovel. Optimal rates for regularized least squares regression. In Conference on Learning Theory (COLT), 2009.
Daniel W. Stroock et al. Essentials of integration theory for analysis. Springer, 2011.
Sattar Vakili, Michael Bromberg, Jezabel Garcia, Da-shan Shiu, and Alberto Bernacchia. Uniform generalization bounds for overparameterized neural networks. arXiv:2109.06099, 2021.
Ernesto Araya Valdivia. Relative concentration bounds for the spectrum of kernel matrices. arXiv:1812.02108, 2018.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science. Cambridge University Press, 2018.
Guorong Wang, Yimin Wei, Sanzheng Qiao, Peng Lin, and Yuzhuo Chen. Generalized Inverses: Theory and Computations. Springer, 2018.
Yutong Wang and Clayton Scott. Consistent interpolating ensembles via the manifold-Hilbert kernel. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yuting Wei, Fanny Yang, and Martin J Wainwright. Early stopping for kernel boosting algorithms: A general analysis with localized complexities. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Holger Wendland. Scattered Data Approximation. Cambridge University Press, 2005.
Xingyu Xu and Yuantao Gu. Benign overfitting of non-smooth neural networks beyond lazy training. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning (ICML), 2021.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26:289-315, 2007.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo, and Zheng Ma. A type of generalization error induced by initialization in deep neural networks. In Proceedings of The First Mathematical and Scientific Machine Learning Conference, 2020.
Appendix Contents.
A Detailed related work 17
B Kernels and Sobolev spaces on the sphere 19
B.1 Background on Sobolev spaces 19
B.2 General kernel theory and notation 19
B.3 Dot-product kernels on the sphere 20
B.4 Neural kernels 21
C Gradient flow and gradient descent with kernels 22
C.1 Derivation of gradient flow and gradient descent 22
C.2 Gradient flow and gradient descent initialized at 0 have monotonically growing  $\mathcal{H}$ -norm 23
D Proof of Theorem 1 24
D.1 Auxiliary results for the proof of Theorem 1 28
E Translating between  $\mathbb{R}^d$  and  $\mathbb{S}^d$  32
F Spectral lower bound 35
F.1 General lower bounds 35
F.2 Equivalences of norms and eigenvalues 38
F.3 Kernel matrix eigenvalue bounds 41
F.4 Spectral lower bound for dot-product kernels on the sphere 43
G Proof of Theorem 8 45
G.1 Auxiliary results for the proof of Theorem 8 50
G.2 RKHS norm bounds 51
H Spiky-smooth activation functions induced by Gaussian components 52
I Additional experimental results 56
I.1 Experimental details of Figure 1 56
I.2 Disentangling signal from noise in neural networks with spiky-smooth activation functions 56
I.3 Repeating the finite-sample experiments 58
I.4 Spiky-smooth activation functions 58
I.5 Isolated spike activation functions 60
I.6 Additive decomposition and sin-fit 62
I.7 Spiky-smooth kernel hyper-parameter selection 63
A Detailed related work
Motivated by Zhang et al. (2021) and Belkin et al. (2018), an abundance of papers have tried to grasp when and how benign overfitting occurs in different settings. Rigorous understanding is mainly restricted to linear (Bartlett et al., 2020), feature (Hastie et al., 2022) and kernel regression (Liang and Rakhlin, 2020) under restrictive distributional assumptions. In the well-specified linear setting under additional assumptions, the minimum-norm interpolant is consistent if and only if  $k \ll n \ll d$ , the top- $k$  eigendirections of the covariate covariance matrix align with the signal, followed by sufficiently many 'quasi-isotropic' directions with eigenvalues of similar magnitude (Bartlett et al., 2020).
Kernel methods. The analysis of kernel methods is more nuanced and depends on the interplay between the chosen kernel, the choice of regularization and the data distribution.  $L_{2}$ -generalization error bounds can be derived in the eigenbasis of the kernel's integral operator (Mcrae et al., 2022), where upper bounds of the form  $\sqrt{\boldsymbol{y}^{\top}k(\boldsymbol{X},\boldsymbol{X})^{-1}\boldsymbol{y}/n}$  promise good generalization when the regression function  $f^{*}$  is aligned with the dominant eigendirections of the kernel, or in other words, when  $\| f^{*}\|_{\mathcal{H}}$  is small. Most recent work focuses on high-dimensional limits, where the data dimensionality  $d\to \infty$ . For  $d\to \infty$ , the Hilbert space and its norm change, so that consistency results that demand bounded Hilbert norm of rotation-invariant kernels do not even include simple functions like sparse products (Donhauser et al., 2021, Lemma 2.1). In the regime  $d^{l + \delta}\leq n\leq d^{l + 1 - \delta}$ , rotation-invariant (neural) kernel methods (Ghorbani et al., 2021, Donhauser et al., 2021) can in fact only learn the polynomial parts up to order  $l$  of the regression function  $f^{*}$ , and fully-connected NTKs do so. Liang et al. (2020) uncover a related multiple descent phenomenon in kernel regression, where the risk vanishes for most  $n\to \infty$ , but peaks at  $n = d^i$  for all  $i\in \mathbb{N}$ . The slower  $d$  grows, the slower the optimal rate  $n^{-\frac{1}{2i + 1}}$  between the peaks. Note, however, that these bounds are only upper bounds, and whether they are optimal remains an open question to the best of our knowledge. Another recent line of work analyzes how different inductive biases, measured in  $\| \cdot \| _p$ -norm minimization,  $p\in [1,2]$ , (Donhauser et al., 2022) or in the filter size of convolutional kernels (Aerni et al., 2023), affects the generalization properties of minimum-norm interpolants. While the risk on noiseless training samples (bias) decreases with decreasing  $p$  or small filter size, the sensitivity to noise in the training data (variance) increases. Hence only 'weak inductive biases', that is large  $p$  or large filter sizes, enable harmless interpolation. Our results suggest that to achieve harmless interpolation in fixed dimension one has to construct and minimize more unusual norms than  $\| \cdot \| _p$ -norms.
Regularised kernel regression achieves optimal rates. With appropriate regularization, kernel ridge regularization with typical universal kernels like the Gauss, Matérn, and Laplace kernels is universally consistent (Steinwart and Christmann, 2008, Chapter 9). Steinwart et al. (2009, Corollary 6) even implies minimax optimal nonparametric rates for clipped kernel ridge regression with Sobolev kernels and  $f^{*} \in H^{\beta}$  where  $d/2 < \beta \leq s$  for the choice  $\rho_{n} = n^{-2s/(2\beta + d)}$ . Although  $f^{*}$  is not necessarily in the RKHS, KRR is adaptive and can still achieve optimal learning rates. Lower smoothness  $\beta$  of  $f^{*}$  as well as higher smoothness of the kernel should be met with faster decay of  $\rho_{n}$ . Optimal rates in Sobolev RKHS can also be achieved using cross-validation of the regularization  $\rho$  (Steinwart et al., 2009), early stopping rules based on empirical localized Rademacher (Raskutti et al., 2014) or Gaussian complexity (Wei et al., 2017) or smoothing of the empirical risk via kernel matrix eigenvalues (Averyanov and Celisse, 2020).
Lower bounds for kernel regression. Besides Rakhlin and Zhai (2019) and Buchholz (2022), Beaglehole et al. (2023) derive inconsistency results for kernel ridgeless regression given assumptions on the spectral tail in the Fourier basis. Mallinar et al. (2022) provide a characterization of kernel ridge regression into benign, tempered and catastrophic overfitting using a heuristic approximation of the risk via the kernel's eigenspectrum, essentially assuming that the eigenfunctions can be replaced by structureless Gaussian random variables. A general lower bound for ridgeless linear regression Holzmuller (2021) predicts bad generalization near the "interpolation threshold", where the dimension of the feature space is close to  $n$ , also known as the double descent phenomenon. In this regime, Ghosh and Belkin (2022) also consider overfitting by a fraction beyond the noise level and derive a lower bound for linear models.
Benign overfitting in fixed dimension. Only few works have established consistency results for interpolating models in fixed dimension. The first statistical guarantees for Nadaraya-Watson kernel
smoothing with singular kernels were given by Devroye et al. (1998). Optimal non-asymptotic results have only been established more recently. Belkin et al. (2019) show that Nadaraya-Watson kernel smoothing achieves minimax optimal convergence rates for  $a \in (0, d/2)$  under smoothness assumptions on  $f^*$ , when using singular kernels such as truncated Hilbert kernels  $K(u) = \| u \|_2^a \mathbb{1}_{\| u \| \leq 1}$ , which do not induce RKHS that only contain weakly differentiable functions (as our results do). By thresholding the kernel they can adjust the amount of overfitting without affecting the generalization bound. To the best of our knowledge, rigorously proving or disproving analogous bounds for kernel ridge regression remains an open question. Arnould et al. (2023) show that median random forests are able to interpolate consistently in fixed dimension because of an averaging effect introduced through feature randomization. They conjecture consistent interpolation for Breiman random forests based on numerical experiments.
Classification. For binary classification tasks, benign overfitting is a more generic phenomenon than for regression tasks (Muthukumar et al., 2021, Shamir, 2022). Consistency has been shown under linear separability assumptions (Montanari et al., 2019, Chatterji and Long, 2021, Frei et al., 2022) and through complexity bounds with respect to reference classes like the 'Neural Tangent Random Feature' model (Cao and Gu, 2019, Chen et al., 2021). Most recently, Liang and Recht (2023) have shown that the 0-1-generalization error of minimum RKHS-norm interpolants  $\hat{f}_0$  is upper bounded by  $\frac{\|\hat{f}_0\|_{\mathcal{H}}^2}{n}$  and analogously that kernel ridge regression  $\hat{f}_{\rho}$  generalizes as  $\frac{\boldsymbol{y}^\top(k(\boldsymbol{X},\boldsymbol{X}) + \rho\boldsymbol{I})^{-1}\boldsymbol{y}}{n}$ , where the numerator upper bounds  $\| \hat{f}_{\rho}\|_{\mathcal{H}}^{2}$ . Their bounds imply consistency as long as the total variation distance between the class conditionals is sufficiently large and the regression function has bounded RKHS-norm, and their Lemma 7 shows that the upper bound is rate optimal. Under a noise condition on the regression function  $f^{*}(\pmb {x}) = \mathbb{E}[y|\pmb {x}]$  for binary classification and bounded  $\| f^{*}\|_{\mathcal{H}}$ , our results together with Liang and Recht (2023) reiterate the distinction between benign overfitting for binary classification and inconsistent overfitting for least squares regression for a large class of distributions in kernel regression over Sobolev RKHS. Chapter 8 of Steinwart and Christmann (2008) discusses how the overlap of the two classes may influence learning rates under positive regularization. Using Nadaraya-Watson kernel smoothing, Wang and Scott (2022) offer the first consistency result for a simple interpolating ensemble method with data-independent base classifiers.
Connection to neural networks. It is known that neural networks can behave like kernel methods in certain infinite-width limits. For example, the function represented by a randomly initialized NN behaves like a Gaussian process with the NN Gaussian process (NNGP) kernel, which depends on details such as the activation function and depth of the NN (Neal, 1996, Lee et al., 2018, Matthews et al., 2018). Hence, Bayesian inference in infinitely wide NNs is GP regression, whose posterior predictive mean function is of the form  $f_{\infty,\rho}$ , where  $\rho$  depends on the assumed noise variance. Moreover, gradient flow training of certain infinitely wide NNs is similar to gradient flow training with the so-called neural tangent kernel (NTK) (Jacot et al., 2018, Lee et al., 2019, Arora et al., 2019), and the correspondence can be made exact using small modifications to the NN to remove the stochastic effect of the random initial function (Arora et al., 2019, Zhang et al., 2020). In other words, certain infinitely wide NNs trained with gradient flow learn functions of the form  $f_{t,0}$ .
When considering the sphere  $\Omega = \mathbb{S}^d$ , the NTK and NNGP kernels of fully-connected NNs are dot-product kernels, i.e.,  $k(\pmb{x},\pmb{x}^{\prime}) = \kappa (\langle \pmb{x},\pmb{x}^{\prime}\rangle)$  for some function  $\kappa :[-1,1]\to \mathbb{R}$ . Moreover, from Bietti and Bach (2021) and Chen and Xu (2021) it follows that the RKHSs of typical NTK and NNGP kernels for the ReLU activation function are equivalent to the Sobolev spaces  $H^{(d + 1) / 2}(\mathbb{S}^d)$  and  $H^{(d + 3) / 2}(\mathbb{S}^d)$ , respectively, cf Appendix B.4.
Regarding consistency, Ji et al. (2021) use the NTK correspondence to show that early-stopped wide NNs for classification are universally consistent under some assumptions. On the other hand, Holzmüller and Steinwart (2022) show that zero-initialized biases can prevent certain two-layer ReLU NNs from being universally consistent. Lai et al. (2023) show an inconsistency-type result for overfitting two-layer ReLU NNs with  $d = 1$ , but for fixed inputs  $X$ . They also note that an earlier inconsistency result by Hu et al. (2021) relies on an unproven result. Li et al. (2023) show that consistency with polynomial convergence rates is impossible for minimum-norm interpolants of common kernels including ReLU NTKs. Mallinar et al. (2022) conjecture tempered overfitting and therefore inconsistency for interpolation with ReLU NTKs based on their semi-rigorous result and the results of Bietti and Bach (2021) and Chen and Xu (2021). Xu and Gu (2023) establish consistency of overfitting wide 2-layer neural networks beyond the NTK regime for binary classification in very
high dimension  $d = \Omega(n^2)$  and for a quite restricted class of distributions (the mean difference  $\mu$  of the class conditionals needs to fulfill  $\mu = \Omega((d/n)^{1/4} \log^{1/4}(md/n))$  and  $\mu = O((d/n)^{1/2})$ ).
B Kernels and Sobolev spaces on the sphere
B.1 Background on Sobolev spaces
We say that two Hilbert spaces  $\mathcal{H}_1, \mathcal{H}_2$  are equivalent, written as  $\mathcal{H}_1 \cong \mathcal{H}_2$ , if they are equal as sets and the corresponding norms  $\| \cdot \|_{\mathcal{H}_1}$  and  $\| \cdot \|_{\mathcal{H}_2}$  are equivalent.
Let  $\Omega$  be an open set with  $C^\infty$  boundary. In this paper, we will mainly consider  $\ell_2$ -balls for  $\Omega$ . There are multiple equivalent ways to define a (fractional) Sobolev space  $H^s(\Omega)$ ,  $s \in \mathbb{R}_{\geq 0}$ , these are equivalent in the sense that the resulting Hilbert spaces will be equivalent. For example,  $H^s(\Omega)$  can be defined through restrictions of functions from  $H^s(\mathbb{R}^d)$ , through interpolation spaces, or through Sobolev-Slobodetski norms (see e.g. Chapter 5 and 14 in Agranovich, 2015 and Chapters 7-10 in Lions and Magenes, 2012). Some requirements on  $\Omega$  can be relaxed, for example to Lipschitz domains, by using more general extension operators (e.g. DeVore and Sharpley, 1993). Since our results are based on equivalent norms and not specific norms, we do not care which of these definitions is used. Further background on Sobolev spaces can be found in Adams and Fournier (2003), Wendland (2005) and Di Nezza et al. (2012).
B.2 General kernel theory and notation
There is a one-to-one correspondence between kernel functions  $k$  and the corresponding reproducing kernel Hilbert spaces (RKHS)  $\mathcal{H}_k$ . Mercer's theorem (Steinwart and Christmann, 2008, Theorem 4.49) states that for compact  $\Omega$ , continuous  $k$  and a Borel probability measure  $P_X$  on  $\Omega$  whose support is  $\Omega$ , the integral operator  $T_{k,P_X}: L_2(P_X) \to L_2(P_X)$  given by
$$
T _ {k, P _ {X}} f (\pmb {x}) = \int_ {\Omega} f (\pmb {x} ^ {\prime}) k (\pmb {x}, \pmb {x} ^ {\prime}) d P _ {X} (\pmb {x} ^ {\prime}),
$$
can be decomposed into an orthonormal basis  $(e_i)_{i\in I}$  of  $L_{2}(P_{X})$  and corresponding eigenvalues  $(\lambda_{i})_{i\in I}\geq 0,\lambda_{l}\searrow 0$  , such that
$$
T _ {k, P _ {X}} f = \sum_ {i \in I} \lambda_ {i} \langle f, e _ {i} \rangle e _ {i}, \qquad f \in L _ {2} (P _ {X}).
$$
We write  $\lambda_{i}(T_{k,P_{X}})\coloneqq \lambda_{i}$ . Moreover,  $k(\pmb {x},\pmb{x}^{\prime}) = \sum_{i\in I}\lambda_{i}e_{i}(\pmb {x})e_{i}(\pmb{x}^{\prime})$  converges absolutely and uniformly, and the RKHS is given by
$$
\mathcal {H} _ {k} = \left\{\sum_ {i \in I} a _ {i} \sqrt {\lambda_ {i}} e _ {i} \left| \sum_ {i \in I} a _ {i} ^ {2} <   \infty \right. \right\}. \tag {B.1}
$$
The corresponding inner product between  $f = \sum_{i\in I}a_i\sqrt{\lambda_i} e_i\in \mathcal{H}$  and  $g = \sum_{i\in I}b_i\sqrt{\lambda_i} e_i\in \mathcal{H}$  can then be written as
$$
\langle f, g \rangle_ {\mathcal {H}} = \sum_ {i \in I} a _ {i} b _ {i}. \tag {B.2}
$$
We use asymptotic notation  $O, \Omega, \Theta$  for integers  $n$  in the following way: We write
$$
f (n) = O (g (n)) \Leftrightarrow \exists C > 0 \forall n: f (n) \leq C g (n)
$$
$$
f (n) = \Omega (g (n)) \Leftrightarrow g (n) = O (f (n))
$$
$$
f (n) = \Theta (g (n)) \Leftrightarrow f (n) = O (g (n)) \text {a n d} g (n) = O (f (n)).
$$
Above, we require that the inequality  $f(n) \leq C g(n)$  holds for all  $n$  and not only for  $n \geq n_0$ . This implies that if  $f(n) = \Omega(g(n))$ , then  $f$  must be nonzero whenever  $g$  is nonzero. This is an important detail when arguing about equivalence of RKHSs, since it allows the following statement: If we have two kernels  $k, \tilde{k}$  with Mercer representations
$$
k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sum_ {i \in I} \lambda_ {i} e _ {i} (\boldsymbol {x}) e _ {i} \left(\boldsymbol {x} ^ {\prime}\right)
$$
$$
\tilde {k} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sum_ {i \in I} \tilde {\lambda} _ {i} e _ {i} (\boldsymbol {x}) e _ {i} (\boldsymbol {x} ^ {\prime})
$$
with identical eigenfunctions  $e_i$  and eigenvalues satisfying  $\lambda_i = \Theta(\tilde{\lambda}_i)$ , then the associated RKHSs are equivalent by (B.1) and (B.2).
B.3 Dot-product kernels on the sphere
A kernel of the form  $k(\pmb{x},\pmb{x}^{\prime}) = \kappa (\langle \pmb{x},\pmb{x}^{\prime}\rangle)$  for some function  $\kappa$  is called dot-product kernel. Dot-product kernels are rotationally invariant. Especially, NTKs and NNGPs of fully-connected NNs restricted to the sphere  $\mathbb{S}^d$  are dot-product kernels. Moreover, kernels like the Laplace, Matérn, and Gaussian kernels that only depend on the distance between their inputs are also dot-product kernels when restricted to the sphere  $\mathbb{S}^d$ . Therefore, in this section, we will assume that  $k:\mathbb{S}^d\times \mathbb{S}^d\to \mathbb{R}$  is a dot-product kernel.
We can then leverage some convenient results from the theory of dot-product kernels on the sphere, which are summarized in more detail by Hubbert et al. (2023). Let  $\{Y_{l,1},\ldots ,Y_{l,N_{l,d}}\}$  be a real orthonormal basis for the space of spherical harmonics of degree  $l$  within  $L_{2}(\mathbb{S}^{d})$ . Moreover, let  $\omega_{d}$  be the surface area of  $\mathbb{S}^d$ , then the  $\tilde{Y}_{l,i}\coloneqq \sqrt{\omega_dY_{l,i}}$  form a corresponding orthonormal basis w.r.t. the uniform distribution  $\mathcal{U}(\mathbb{S}^d)$ . Then, a Mercer representation of  $k$  is given by
$$
k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sum_ {l = 0} ^ {\infty} \mu_ {l} \sum_ {i = 1} ^ {N _ {l, d}} Y _ {l, i} (\boldsymbol {x}) Y _ {l, i} (\boldsymbol {x} ^ {\prime}) = \sum_ {l = 0} ^ {\infty} \tilde {\mu} _ {l} \sum_ {i = 1} ^ {N _ {l, d}} \tilde {Y} _ {l, i} (\boldsymbol {x}) \tilde {Y} _ {l, i} (\boldsymbol {x} ^ {\prime}),
$$
with  $\tilde{\mu}_l = \mu_l / \omega_d$ . Especially, the integral operator  $T_{k,\mathcal{U}(\mathbb{S}^d)}$  for the uniform distribution  $\mathcal{U}(\mathbb{S}^d)$  has eigenvalues  $\tilde{\mu}_l$  with multiplicity  $N_{l,d}$  and eigenfunctions  $\tilde{Y}_{l,i}$ . The RKHS of  $k$  is then given by
$$
\mathcal {H} _ {k} = \left\{\sum_ {l = 0} ^ {\infty} \sqrt {\mu_ {l}} \sum_ {i = 1} ^ {N _ {l, d}} a _ {l, i} Y _ {l, i} \left| \sum_ {l = 0} ^ {\infty} \sum_ {i = 1} ^ {N _ {l, d}} a _ {l, i} ^ {2} <   \infty \right. \right\}.
$$
Since the index  $l$  can be zero, we will denote decay asymptotics for  $l$  in the form  $\Theta((l + 1)^{-q})$  and not  $\Theta(l^{-q})$ , cf. our definition of  $\Theta$  notation in Appendix B.2.
Lemma B.1 (Sobolev dot-product kernels on the sphere). For a dot-product kernel  $k$  on  $\mathbb{S}^d$  as above, the RKHS  $\mathcal{H}_k$  is equivalent to the Sobolev space  $H^{s}(\mathbb{S}^{d})$ ,  $s > d / 2$ , if and only if  $\mu_{l} = \Theta ((l + 1)^{-2s})$ . In this case, we have
$$
\lambda_ {i} (T _ {k, \mathcal {U} (\mathbb {S} ^ {d})}) = \Theta (i ^ {- 2 s / d}).
$$
Proof. Step 0: Equivalence. If  $\mu_l = \Theta((l + 1)^{-2s})$ , it is stated in Section 3 in Hubbert et al. (2023) that  $\mathcal{H}_k \cong H^s(\mathbb{S}^d)$ . On the other hand, if  $\mu_l \neq \Theta((l + 1)^{-2s})$ , it is easy to see that  $\mathcal{H}_k$  is not equivalent to the RKHS of a kernel with  $\mu_l = \Theta((l + 1)^{-2s})$ . It remains to show  $\lambda_i(T_{k,\mathcal{U}(\mathbb{S}^d)}) = \Theta(i^{-2s/d})$ .
Step 1: Ordering the eigenvalues. Consider a permutation  $\pi : \mathbb{N}_0 \to \mathbb{N}_0$  such that
$$
\tilde {\mu} _ {\pi (0)} \geq \tilde {\mu} _ {\pi (1)} \geq \dots
$$
We can then define the partial sums
$$
S _ {l} := \sum_ {i = 0} ^ {l} N _ {\pi (i), d}.
$$
For  $S_{l - 1} <   i\leq S_l$  , we then have  $\lambda_{i}(T_{k,\mathcal{U}(\mathbb{S}^{d})}) = \tilde{\mu}_{\pi (l)}$
Step 2: Show  $\pi(i) = \Theta(i)$ . Let  $c, C > 0$  such that  $c(l + 1)^{-2s} \leq \tilde{\mu}_l \leq C(l + 1)^{-2s}$  for all  $l \in \mathbb{N}_0$ . For indices  $i, j \in \mathbb{N}_0$ , we have the implications
$$
\begin{array}{l} i > j \Rightarrow c (\pi (i) + 1) ^ {- 2 s} \leq \tilde {\mu} _ {\pi (i)} \leq \tilde {\mu} _ {\pi (j)} \leq C (\pi (j) + 1) ^ {- 2 s} \\ \Rightarrow \pi (i) + 1 \geq \left(\frac {c}{C}\right) ^ {1 / (2 s)} (\pi (j) + 1). \\ \end{array}
$$
Therefore, for  $i\geq 1$  and  $j\geq 0$
$$
\begin{array}{l} \pi (i) + 1 \geq \left(\frac {c}{C}\right) ^ {1 / (2 s)} \max  _ {i ^ {\prime} <   i} (\pi (i ^ {\prime}) + 1) \geq \left(\frac {c}{C}\right) ^ {1 / (2 s)} ((i - 1) + 1) \geq \Omega (i + 1), \\ \pi (j) + 1 \leq \left(\frac {C}{c}\right) ^ {1 / (2 s)} \min  _ {j ^ {\prime} > j} (\pi (j ^ {\prime}) + 1) \leq \left(\frac {C}{c}\right) ^ {1 / (2 s)} ((j + 1) + 1) \leq O (j + 1). \\ \end{array}
$$
We can thus conclude that  $\pi (i) + 1 = \Theta (i + 1)$ .
Step 3: Individual Eigenvalue decay. As explained in Section 2.1 in Hubbert et al. (2023), we have  $N_{l,d} = \Theta((l + 1)^{d - 1})$ . Therefore,