SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B Translation-invariant kernels and Fourier analysis
This section recalls basic facts about kernel methods and Fourier analysis, before proving Table 1.
B.1 Stylized analysis on the torus
When  $k$  is a translation-invariant kernel, i.e.  $k(x,x^{\prime}) = q(x - x^{\prime})$ , the integral operator  $K$  is a convolution against  $q$ . Let us expand on the friendly case provided by the torus  $\mathcal{X} = \mathbb{T}^{d} := \mathbb{R}^{d} / \mathbb{Z}^{d} = [0,1]^{d} / \sim$ , where  $\sim$  is the relation identifying opposite faces of the hypercube, and  $\rho_{\mathcal{X}}$  the uniform distribution. On the torus, a translation invariant kernel is defined through  $q$  being a one-periodic function on  $\mathbb{R}^{d}$ . Let  $\mathrm{d}x$  denote the Lebesgue measure on  $\mathbb{R}^{d}$ . The integral operator  $K:L^{2}(\mathcal{X},\mathrm{d}x)\to L^{2}(\mathcal{X},\mathrm{d}x)$  is the convolution
$$
K f (x) = \int_ {[ 0, 1 ] ^ {d}} k (x, x ^ {\prime}) f (x ^ {\prime}) \mathrm {d} x ^ {\prime} = \int_ {[ 0, 1 ] ^ {d}} q (x ^ {\prime} - x) f (x ^ {\prime}) \mathrm {d} x ^ {\prime} = q * f (x).
$$
For  $m \in \mathbb{Z}^d$ , define the Fourier function  $f_m : x \mapsto \exp(2i\pi \langle m, x \rangle)$ . One can check that the  $f_m$ 's form an orthonormal family that diagonalizes  $K$  with
$$
K f _ {m} = \widehat {q} _ {m} f _ {m}, \quad \text {w h e r e} \quad \widehat {q} _ {m} = \int_ {[ 0, 1 ] ^ {d}} q (x) \exp (2 i \pi \langle x, m \rangle)   \mathrm {d} x.
$$
Hence, using Pythagoras theorem, we can define the norm on  $\mathcal{F}$  through its action on Fourier coefficients as
$$
\| f \| _ {\mathcal {F}} ^ {2} = \left\langle f, K ^ {- 1} f \right\rangle_ {L ^ {2} (\rho_ {\mathcal {X}})} = \sum_ {m \in \mathbb {Z} ^ {d}} \widehat {q} _ {m} ^ {- 1} | \widehat {f} _ {m} | ^ {2} = \int_ {\mathbb {R} ^ {d}} \widehat {q} (\omega) ^ {- 1} | \widehat {f} (\omega) | ^ {2} \# (\mathrm {d} \omega),
$$
where  $\widehat{f}_m = \langle f, f_m \rangle_{L^2(\rho_{\mathcal{X}})}$ , and  $\#$  is the counting measure on  $\mathbb{Z}^d \subset \mathbb{R}^d$ .
Indeed, the Fourier transform of a function  $f \in L^{2}(\mathbb{T}^{d})$  can be defined as the mapping from  $\mathbb{N}$  to  $(\langle f_i, f \rangle)_{i \in \mathbb{N}}$  where  $(f_i)$  is a basis that diagonalizes all convolution operators (note that this definition is possible because  $\rho_{\mathcal{X}}$  is uniform on the torus).
B.1.1 First part of the proof of Proposition 1
Since  $K$  is diagonalized in Fourier, we compute the size of  $\mathcal{F}$  for  $a\in \{1,2\}$  with
$$
\mathcal {N} _ {a} (K) = \operatorname {T r} \left(K ^ {a} (K + 1) ^ {- a}\right) = \sum_ {m \in \mathbb {Z} ^ {d}} \frac {\widehat {q} _ {m} ^ {a}}{(\widehat {q} _ {m} + 1) ^ {a}}.
$$
For kernels whose scales are explicitly defined through  $q_{\sigma} = q(x / \sigma)$ , we have  $\widehat{q}_{\sigma}(\omega) = \sigma^{d}\widehat{q} (\sigma \omega)$  which leads to (14).
Similarly, the bound on the bias term follows from Fourier analysis by
$$
\mathcal {S} (K) = \left\| K (K + 1) ^ {- 1} f \right\| _ {L ^ {2} (\rho_ {\mathcal {X}})} ^ {2} = \sum_ {m \in \mathbb {Z} ^ {d}} \left| \widehat {f} _ {m} \right| \left(\frac {1}{\widehat {q} _ {m} + 1}\right) ^ {2},
$$
which provides (15).
B.1.2 Second part of the proof of Proposition 1
When  $\rho$  is a distribution that is absolutely continuous with respect to the Lebesgue measure and whose density is bounded from above, we get
$$
K \preceq \rho_ {\infty} K _ {\mathrm {d} x}, \quad \text {w i t h} \quad \rho_ {\infty} = \left\| \frac {\mathrm {d} \rho_ {\mathcal {X}}}{\mathrm {d} x} \right\| _ {L ^ {\infty} (\rho_ {\mathcal {X}})},
$$
where  $K_{\mathrm{dx}}$  is the integral operator associated to the kernel  $k$  on  $L^2(\mathrm{d}x)$ . Using the fact the effective dimension is an increasing function of the eigenvalues (since  $x \mapsto x / (x + 1)$  is increasing), and that eigenvalues are increasing with the Loewner order, this leads to
$$
\mathcal {N} (K) \leq \rho_ {\infty} \operatorname {T r} \left(\left(K _ {\mathrm {d} x} + 1\right) ^ {- 1} K _ {\mathrm {d} x}\right) = \rho_ {\infty} \int_ {\mathbb {R} ^ {d}} \frac {\widehat {q} (\omega)}{1 + \widehat {q} (\omega)} \mathrm {d} \omega .
$$
Note that those derivations are written informally (since  $K$  and  $K_{\mathrm{dx}}$  do not act on the same space), but could be made formal with the isomorphic covariance operators on  $\mathcal{H}$ , plus some technicalities to make sure  $\Sigma_{\mathrm{dx}}$  is well defined (assuming  $\varphi(X)$  has a fourth-order moment against Lebesgue, or approaching  $K_{\mathrm{dx}}$  within its action on compact subspaces of  $\mathcal{X}$  where it is bounded, before taking the limit of  $\mathcal{N}_{\mathrm{dx}}(K)$ ).
For the bias term, using the fact that  $L^2(\rho_{\mathcal{X}})$  is continuously embedded in  $L^2(\mathrm{d}x)$  and the isometry between the spatial and the Fourier domain, we get
$$
\left\| f - f ^ {*} \right\| _ {L ^ {2} \left(\rho_ {\mathcal {X}}\right)} \leq \rho_ {\infty} ^ {1 / 2} \left\| f - f ^ {*} \right\| _ {L ^ {2} (\mathrm {d} x)} = \rho_ {\infty} ^ {1 / 2} \left\| \widehat {f} - \widehat {f ^ {*}} \right\| _ {L ^ {2} (\mathrm {d} x)}.
$$
Finally, it should be noted that the norm associated with  $\mathcal{F}$  does not depend on the density of  $X$ , hence the formula can be written independently of  $\rho_{\mathcal{X}}$ . Indeed, under definition assumption, i.e. if  $q \in L^{1}(\mathrm{d}x)$ , this formula can even be written with a measure of infinite mass. For example, when  $\mathcal{X} = \mathbb{R}^d$ , one can consider the Fourier transform associated with  $L^2(\mathrm{d}x)$ , and get
$$
\left\| f \right\| _ {\mathcal {F}} ^ {2} = \int_ {\mathbb {R} ^ {d}} \widehat {q} (\omega) ^ {- 1} | \widehat {f} (\omega) | ^ {2} \mathrm {d} \omega , \quad \text {w h e r e} \quad \widehat {q} (\omega) = \int_ {\mathbb {R} ^ {d}} q (x) \exp (- 2 i \pi \langle x, \omega \rangle) \mathrm {d} x, \tag {31}
$$
although some care is needed to deal with the continuous version of the spectral theorem (the set of eigenvalues being non-countable). From there the same derivations as for Proposition 1 lead to the desired result.
B.2 Sobolev spaces
Recall the action of differentiation on the Fourier transform: for  $m \in \mathbb{N}^d$ ,  $|m| := \| m\|_1$ , and  $f \in L^2(\mathrm{d}x)$ ,
$$
\frac {\widehat {\partial^ {| m |} f}}{\prod_ {i \in [ d ]} \partial^ {m _ {i}} x _ {i}} (\omega) = (2 i \pi) ^ {| m |} \prod_ {i \in [ d ]} \omega_ {i} ^ {m _ {i}} \widehat {f} (\omega).
$$
This characterizes the pseudo-norm
$$
\| f \| _ {m} ^ {2} = \int_ {\mathbb {R} ^ {d}} \left\| \frac {\partial^ {| m |} f (x)}{\prod_ {i \in [ d ]} \partial^ {m _ {i}} x _ {i}} \right\| ^ {2} \mathrm {d} x = (2 \pi) ^ {2 | m |} \int_ {\mathbb {R} ^ {d}} \prod_ {i \in [ d ]} \omega_ {i} ^ {2 m _ {i}} \left| \widehat {f} (\omega) \right| ^ {2} \mathrm {d} \omega .
$$
This pseudo-norm is associated with the translation-invariant kernel such that  $\widehat{q} (\omega) = \prod_{i\in [d]}\omega_i^{-2m_i}$  as per (31). Note that  $q$  is well defined when  $\widehat{q}$  belongs to  $L^1 (\mathrm{d}x)$  (by Bochner's theorem), that is  $|m| > d$ . Those observations are usual to deduce that the Matérn kernels, which are defined from  $\widehat{q} (\omega)\propto (1 + \| \omega \|_2^2)^{-\beta}$ , correspond to the Sobolev spaces  $H^{\beta}(\mathrm{d}x)$  endowed with the norm
$$
\left\| f \right\| _ {H ^ {\beta}} ^ {2} = \sum_ {m; | m | \leq \beta} \left\| f \right\| _ {m} ^ {2}.
$$
It follows from Bochner's theorem that  $H^{\beta}$  is a reproducing kernel Hilbert space if and only if  $2\beta > d$ . Remarkably, the exponential kernel corresponds to the Matérn kernel with  $\beta = (d + 1)/2$  [24]. For the Gaussian kernel,  $\widehat{q}(\omega) = \pi^{-d/2} \exp(-\pi^2 \| \omega \|^2)$ , and the associated function class  $\mathcal{F}$  is analytic (by the Paley-Wiener theorem).
B.2.1 Functional sizes
Let us now express the capacity and bias bound within Sobolev spaces.
Proposition 11 (Sobolev capacity). When  $\widehat{q} (\omega) = \lambda^{-1}(1 + \| \omega \| ^2)^{-\beta}$  for  $\beta >d$ $\lambda \sigma^{-d}$  is bounded and  $\rho$  has a bounded density, we have
$$
\mathcal {N} _ {1} (\sigma , \lambda) \leq \frac {2 \beta \rho_ {\infty} \pi^ {(d + 1) / 2}}{\Gamma ((d - 1) / 2)} \lambda^ {- d / 2 \beta} \sigma^ {- d (2 \beta - d) / 2 \beta}.
$$
Moreover, when  $\mathcal{X} = \mathbb{T}^d$  and  $\rho_{\mathcal{X}}$  is uniform, we get
$$
\mathcal {N} _ {2} (\sigma = 1, \lambda) \geq \max  _ {l \in [ d ]} \frac {l \pi^ {(l + 1) / 2}}{2 ^ {l + 1} \Gamma ((l - 1) / 2)} \lambda^ {- l / 2 \beta}.
$$
Proof. In this setting, Proposition 1 leads to
$$
\begin{array}{l} \int_ {\mathbb {R} ^ {d}} \frac {1}{1 + \lambda \widehat {q _ {\sigma}} (\omega) ^ {- 1}} d \omega = \int_ {\mathbb {R} ^ {d}} \frac {1}{1 + \lambda \sigma^ {- d} \widehat {q} (\sigma \omega) ^ {- 1}} d \omega = \int_ {\mathbb {R} ^ {d}} \frac {1}{1 + \lambda \sigma^ {- d} (1 + \sigma^ {2} \| \omega \| ^ {2}) ^ {\beta}} d \omega \\ = \operatorname {s u r f} \left(\mathcal {S} ^ {d + 1}\right) \int_ {\mathbb {R} _ {+}} \frac {r ^ {d - 1} \mathrm {d} r}{1 + \lambda \sigma^ {- d} (1 + \sigma^ {2} r ^ {2}) ^ {\beta}} \\ = 2 \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \int_ {\lambda^ {1 / \beta} \sigma^ {- d / \beta}} ^ {\infty} \frac {(u - \lambda^ {1 / \beta} \sigma^ {- d / \beta}) ^ {d / 2 - 1} \mathrm {d} u}{\lambda^ {d / 2 \beta} \sigma^ {d - d ^ {2} / 2 \beta} (1 + u ^ {\beta})} \\ = 2 \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \lambda^ {- d / 2 \beta} \sigma^ {d (d - 2 \beta) / 2 \beta} \int_ {\mathbb {R} _ {+}} \frac {x ^ {d / 2 - 1} \mathrm {d} x}{1 + (x + \lambda^ {1 / \beta} \sigma^ {- d / \beta}) ^ {\beta}} \\ \leq 2 \pi \beta \operatorname {v o l} (\mathcal {S} ^ {d}) \lambda^ {- d / 2 \beta} \sigma^ {d (d - 2 \beta) / 2 \beta}, \\ \end{array}
$$
where we used the fact that
$$
\begin{array}{l} \int_ {0} ^ {\infty} \frac {x ^ {d / 2 - 1} \mathrm {d} x}{1 + (x + \lambda^ {1 / \beta} \sigma^ {- d / \beta}) ^ {\beta}} \leq \int_ {0} ^ {\infty} \frac {x ^ {d / 2 - 1} \mathrm {d} x}{\max (1 , \max (x ^ {\beta} , \lambda \sigma^ {- d}))} \\ \leq \int_ {0} ^ {1} x ^ {d / 2 - 1} \mathrm {d} x + \int_ {1} ^ {\infty} \frac {x ^ {d / 2 - 1} \mathrm {d} x}{x ^ {\beta}} = d / 2 - (d / 2 - \beta) = \beta , \\ \end{array}
$$
which is true as long as  $\beta >d / 2$  to ensure proper convergence of the last integral.
For the part on the torus, in order to get a sharp learning limit, we need to be slightly more precise. In particular, we want to relate the discrete Fourier transform integral of Proposition 1 with the continuous one through series-integral comparison, and get a lower bound on the last integral. We
will fix  $\sigma = 1$  for simplicity. A simple cut of  $\mathbb{R}^d$  into unit cubes, together with the fact that our integrand is decreasing, leads to
$$
\sum_ {m \in \mathcal {Z} ^ {d}} {\bf 1} _ {0 \notin m} \frac {\widehat {q} _ {m} ^ {2}}{(\widehat {q} _ {m} + \lambda) ^ {2}} \leq \int \frac {\widehat {q} (\omega) ^ {2}}{(\widehat {q} (\omega) + \lambda) ^ {2}} \mathrm {d} \omega \leq \sum_ {m \in \mathcal {Z} ^ {d}} 2 ^ {\# \{i \in [ d ] \mid m _ {i} = 0 \}} \frac {\widehat {q} _ {m} ^ {2}}{(\widehat {q} _ {m} + \lambda) ^ {2}}.
$$
We simplify it as
$$
\mathcal {N} _ {2} (\sigma , \lambda) \geq 2 ^ {- d} \int \frac {\widehat {q} (\omega) ^ {2}}{(\widehat {q} (\omega) + \lambda) ^ {2}} \mathrm {d} \omega .
$$
We now compute the integral with the same techniques as before:
$$
\begin{array}{l} \int \frac {\widehat {q} (\omega) ^ {2}}{(\widehat {q} (\omega) + \lambda) ^ {2}} d \omega = \int \frac {1}{(1 + \widehat {q} (\omega) ^ {- 1} \lambda) ^ {2}} d \omega = \int \frac {1}{(1 + (1 + \| \omega \| ^ {2}) ^ {\beta} \lambda) ^ {2}} d \omega \\ = 2 \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \int \frac {x ^ {d - 1}}{(1 + (1 + x ^ {2}) ^ {\beta} \lambda) ^ {2}} \mathrm {d} x \\ = 2 \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \lambda^ {- d / 2 \beta} \int \frac {x ^ {d - 1}}{\left(1 + \left(\lambda^ {1 / \beta} + x ^ {2}\right) ^ {\beta}\right) ^ {2}} \mathrm {d} x \\ \geq 2 \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \lambda^ {- d / 2 \beta} \int \frac {x ^ {d - 1}}{4 \max  \left(1 , 4 ^ {\beta} \max  \left(\lambda^ {2} , x ^ {4 \beta}\right)\right)} \mathrm {d} x \\ = 2 ^ {- 1} \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \lambda^ {- d / 2 \beta} \left(\int_ {0} ^ {1} x ^ {d - 1} \mathrm {d} x + 4 ^ {- \beta} \int_ {1} ^ {\infty} \frac {x ^ {d - 1}}{x ^ {4 \beta}} \mathrm {d} x\right) \\ = 2 ^ {- 1} \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \lambda^ {- d / 2 \beta} (d + 4 ^ {- \beta} (4 \beta - d)) \\ \geq 2 ^ {- 1} \pi \operatorname {v o l} \left(\mathcal {S} ^ {d}\right) \lambda^ {- d / 2 \beta} d. \\ \end{array}
$$
It should be noted that this last bound is somewhat too lax, as it tends to zero when the dimension increases. Since  $\sum_{m\in \mathbb{Z}^d}a(\| m\|)$  for  $a > 0$  is strictly increasing with  $d$ , we deduce that this lower bound holds for any  $k\leq d$ .
Proposition 12 (Gaussian capacity). When  $\widehat{q} (\omega) = \lambda^{-1}\exp (-\| \omega \| ^2)$  and  $\rho$  has a bounded density, we have
$$
\mathcal {N} _ {1} (\lambda , \sigma) \leq \frac {\rho_ {\infty} \pi^ {(d - 1) / 2} d}{2 \sigma^ {d}} L (\lambda^ {- 1} \sigma^ {d}),
$$
where  $L$  is defined by Eq. (32). In particular,  $L(x) \leq x$  when  $x < 1$ , and  $L(x) \lesssim \log(x)^{d/2}$  when  $x$  gets large. Moreover when  $\mathcal{X} = \mathbb{T}^d$  and  $\rho_{\mathcal{X}}$  is uniform, we get
$$
\mathcal {N} _ {2} (\lambda , \sigma) \geq \frac {\pi^ {(d - 1) / 2}}{2 ^ {d + 1} \sigma^ {d}} L (\lambda^ {- 1} \sigma^ {d}).
$$
Proof. With the Gaussian kernel, Proposition 1 leads to
$$
\begin{array}{l} \int_ {\mathbb {R} ^ {d}} \frac {1}{1 + \lambda \sigma^ {- d} \exp (\sigma^ {2} \| \omega \| ^ {2})} \mathrm {d} \omega = \operatorname {v o l} (\mathbb {S} ^ {d}) \int_ {\mathbb {R} _ {+}} \frac {2 x ^ {d - 1}}{1 + \lambda \sigma^ {- d} \exp (\sigma^ {2} x ^ {2})} \mathrm {d} x \\ = \operatorname {v o l} (\mathbb {S} ^ {d}) \sigma^ {- d} \int_ {\mathbb {R} _ {+}} \frac {u ^ {d / 2 - 1}}{1 + \lambda \sigma^ {- d} \exp (u)} \mathrm {d} u \\ = \operatorname {v o l} (\mathbb {S} ^ {d}) \Gamma (d / 2) \frac {- \operatorname {L i} _ {d / 2} (- \sigma^ {d} / \lambda)}{\sigma^ {d}}, \\ \end{array}
$$
where  $\mathrm{Li}_{d / 2}$  is the polylogarithm function, hence the definition of  $L$  as
$$
L (x) = - \operatorname {L i} _ {d / 2} (- x) = \sum_ {k = 1} ^ {\infty} \frac {(- 1) ^ {k + 1} x ^ {k}}{k ^ {d / 2}}. \tag {32}
$$
We recognize an alternating sequence, whose term amplitudes are decreasing as a function of  $k \in \mathbb{N}$  when  $x \leq 1$ , which explains that  $L(x)$  is smaller than the first term in this case. The expansion of the polylogarithm function at infinity leads to the upper bound when  $x$  goes to infinity.
When it comes to a lower bound, we can proceed as the precedent lemma with
$$
\mathcal {N} _ {2} (\sigma , \lambda) \geq 2 ^ {- d} \int \frac {\mathrm {d} \omega}{(1 + \lambda \widehat {q} (\omega) ^ {- 1}) ^ {2}} \geq 2 ^ {- d} \int \frac {\mathrm {d} \omega}{1 + \lambda \widehat {q} (\omega) ^ {- 1}},
$$
which corresponds to the integral computed for the upper bound. Once again, this also holds when  $d$  is replaced by any  $l \in [d]$ .
B.2.2 Adherence
Let us now turn our attention to the bias, i.e. the adherence of functions in those spaces. For proof readability, we will assume that  $\rho_{\mathcal{X}}$  has compact support. In this setting,  $W^{\alpha,2}$  is continuously embedded in  $C^{\alpha + d/2} = W^{\alpha + d/2,\infty}$ , which can be defined as
$$
C ^ {\alpha + d / 2} = \left\{f: \mathbb {R} ^ {d} \to R \Bigg | \| f \| _ {\alpha} := \operatorname * {e s s} _ {\omega \in \mathbb {R} ^ {d}} \sup _ {\omega \in \mathbb {R} ^ {d}} \left| \widehat {f} (\omega) \right| (1 + \| \omega \|) ^ {\alpha + d / 2} <   + \infty \right\}.
$$
Proposition 13 (Adherence of  $H^{\alpha}$  in  $H^{\beta}$ ). When  $\widehat{q}(\omega) = \lambda^{-1}(1 + \| \omega \|^2)^{-\beta}$ , hence  $\mathcal{F} = H^{\beta}$ , if  $\alpha > 2\beta$ , for any  $f^* \in H^\alpha(\rho_\chi)$ , and  $\lambda$  small enough, we have
$$
\mathcal {B} (\sigma , \lambda) \leq \lambda^ {2} \| K ^ {- 1} f \| _ {L ^ {2} (\rho_ {X})} ^ {2}.
$$
If  $\alpha < \beta$  and  $\rho_{\mathcal{X}}$  has a bounded density, then for any function  $f^{*} \in C^{\alpha + d/2}$
$$
\mathcal {B} (\sigma , \lambda) \leq \rho_ {\infty} \frac {4 \beta \pi^ {(d + 1) / 2} \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \beta}{(\beta^ {2} - \alpha^ {2}) \Gamma ((d - 1) / 2)} \lambda^ {\alpha / \beta} \sigma^ {(2 \beta - d) \alpha / \beta}. \tag {33}
$$
Proof. The first part results from previous considerations on the source condition since we have the inclusion  $f^{*} \in H^{\alpha} \subset H^{2\beta} = K(L^{2}(\rho_{\mathcal{X}}))$ . The second part follows from an  $L^1 - L^\infty$  Hölder inequality:
$$
\begin{array}{l} \mathcal {B} (\sigma , \lambda) \leq \rho_ {\infty} \int_ {\mathbb {R} ^ {d}} \frac {\left| \widehat {f} (\omega) \right| ^ {2}}{(\lambda^ {- 1} \sigma^ {d} \widehat {q} (\sigma \omega) + 1) ^ {2}} \mathrm {d} \omega \leq \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \int_ {\mathbb {R} ^ {d}} \frac {(1 + \| \omega \| ^ {2}) ^ {- (d / 2 + \alpha)}}{(\lambda^ {- 1} \sigma^ {d} (1 + \sigma^ {2} \| \omega \| ^ {2}) ^ {- \beta} + 1) ^ {2}} \mathrm {d} \omega \\ = 2 \pi \operatorname {v o l} (\mathbb {S} ^ {d}) \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \int_ {\mathbb {R} _ {+}} \frac {2 (1 + x ^ {2}) ^ {- (d / 2 + \alpha)} x ^ {d - 1}}{(\lambda^ {- 1} \sigma^ {d} (1 + \sigma^ {2} x ^ {2}) ^ {- \beta} + 1) ^ {2}} \mathrm {d} x \\ = 2 \pi \mathrm {v o l} (\mathbb {S} ^ {d}) \rho_ {\infty} \| f \| _ {\alpha} ^ {2} a ^ {\alpha} \sigma^ {2 \alpha} \int_ {a} ^ {\infty} \frac {(u - a + a \sigma^ {2}) ^ {- (d / 2 + \alpha)} (u - a) ^ {d / 2 - 1}}{(u ^ {- \beta} + 1) ^ {2}} \mathrm {d} u \\ \leq \frac {4 \beta \pi \operatorname {v o l} (\mathbb {S} ^ {d}) \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \beta}{\beta^ {2} - \alpha^ {2}} \lambda^ {\alpha / \beta} \sigma^ {(2 \beta - d) \alpha / \beta}, \\ \end{array}
$$
where  $a$  was set to  $\lambda^{1 / \beta}\sigma^{-d / \beta}$ , and the last integral can be bounded by
$$
\begin{array}{l} \int_ {a} ^ {\infty} \frac {(u - a + a \sigma^ {2}) ^ {- (d / 2 + \alpha)} (u - a) ^ {d / 2 - 1}}{(u ^ {- \beta} + 1) ^ {2}} d u \leq \int_ {0} ^ {\infty} \frac {u ^ {d / 2 + \beta - 1}}{(u + a (\sigma^ {2} - 1)) ^ {d / 2 + \alpha} (1 + u ^ {\beta}) ^ {2}} d u \\ \leq \int_ {0} ^ {\infty} \frac {u ^ {d / 2 + \beta - 1}}{(u + a (\sigma^ {2} - 1)) ^ {d / 2 + \alpha} (1 + u ^ {\beta}) ^ {2}} \mathrm {d} u \leq \int_ {0} ^ {1} \frac {u ^ {d / 2 + \beta - 1}}{u ^ {d / 2 + \alpha}} \mathrm {d} u + \int_ {1} ^ {+ \infty} \frac {u ^ {d / 2 + \beta - 1}}{u ^ {d / 2 + \alpha} u ^ {2} \beta} \mathrm {d} u \\ = \frac {1}{\beta - \alpha} + \frac {1}{\beta + \alpha} = \frac {\beta}{\alpha (\beta - \alpha)}. \\ \end{array}
$$
Recalling the volume of the sphere completes the proof.
Proposition 14 (Adherence of  $H^{\alpha}$  in the Gaussian RKHS). When  $\mathcal{F}$  is associated with the Gaussian kernel and  $\rho_{\mathcal{X}}$  has a bounded density, for any  $f^{*} \in C^{\alpha + d / 2}$  we have
$$
\mathcal {B} (\sigma , \lambda) \leq \frac {2 \pi^ {(d + 1) / 2} \rho_ {\infty} \| f \| _ {\alpha} ^ {2}}{\Gamma ((d - 1) / 2)} \left(\frac {1}{\sigma^ {2 d + 4 \alpha}} + \frac {2 \log (2) ^ {- (d / 2 + 2 \alpha)}}{d + 2 \alpha}\right) \sigma^ {2 \alpha} \log (\lambda^ {- 1} \sigma^ {d}) ^ {- \alpha}.
$$
Proof. We follow the same path as for the adherence of  $H^{\alpha}$  in  $H^{\beta}$ :
$$
\begin{array}{l} \mathcal {B} (\sigma , \lambda) \leq \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \int_ {\mathbb {R} ^ {d}} \frac {\left(1 + \| \omega \| ^ {2}\right) ^ {- (d / 2 + \alpha)}}{\left(\lambda^ {- 1} \sigma^ {d} \widehat {q} (\sigma \omega) + 1\right) ^ {2}} d \omega \\ = \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \int_ {\mathbb {R} ^ {d}} \frac {(1 + \| \omega \| ^ {2}) ^ {- (d / 2 + \alpha)}}{(\lambda^ {- 1} \sigma^ {d} \exp (- \sigma^ {2} \| \omega \| ^ {2}) + 1) ^ {2}} d \omega \\ = 2 \pi \operatorname {v o l} \left(\mathbb {S} ^ {d}\right) \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \int_ {\mathbb {R} _ {+}} \frac {\left(1 + x ^ {2}\right) ^ {- (d / 2 + \alpha)} x ^ {d - 1}}{\left(\lambda^ {- 1} \sigma^ {d} \exp \left(- \sigma^ {2} x ^ {2}\right) + 1\right) ^ {2}} \mathrm {d} x \\ = 2 \pi \operatorname {v o l} (\mathbb {S} ^ {d}) \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \sigma^ {2 \alpha} \int_ {1} ^ {+ \infty} \frac {\log (x) ^ {d / 2 - 1}}{(\sigma^ {2} + \log (x)) ^ {d + 2 \alpha} (a + x)} d x \\ \leq 2 \pi \operatorname {v o l} (\mathbb {S} ^ {d}) \rho_ {\infty} \| f \| _ {\alpha} ^ {2} \sigma^ {2 \alpha} \log (\lambda^ {- 1} \sigma^ {d}) ^ {- \alpha} \left(\frac {1}{\sigma^ {2 d + 4 \alpha}} + \frac {\log (2) ^ {- \alpha}}{\alpha}\right), \\ \end{array}
$$
where the integral can be bounded by
$$
\begin{array}{l} \int_ {1} ^ {+ \infty} \frac {\log (x) ^ {d / 2 - 1}}{(\sigma^ {2} + \log (x)) ^ {d + 2 \alpha} (a + x)} d x \leq \int_ {1} ^ {e} \frac {\log (x)}{\sigma^ {2 d + 4 \alpha}} d x + \int_ {e} ^ {+ \infty} \frac {1}{(\log (x)) ^ {d / 2 + 2 \alpha + 1} x} d x \\ = \frac {1}{\sigma^ {2 d + 4 \alpha}} + \frac {\log (2) ^ {- (d / 2 + 2 \alpha)}}{d / 2 + 2 \alpha}. \\ \end{array}
$$
The same type of derivations can be made for the bias in the lower bound.
Note that the proofs also work when the  $L^1 - L^\infty$  Hölder inequality is replaced with  $L^\infty - L^1$ , showcasing the norm of  $f$  in  $H^\alpha$  instead of in  $C^{\alpha + d/2}$ . We refer to Bach [4] for details.
Remark 15 (Blessing of dimensionality). It should be noted that all our integral calculations show a constant  $2\pi \rho_{\infty}\mathrm{vol}(\mathbb{S}^{d})$  which will be present in front of the excess risk. As  $d$  increases, this constant goes to zero faster than any exponential profile. To see that, note how the volume of the  $d$ -sphere is always smaller than twice the one of its inscribed hypercube, whose volume is  $d^{-d/2}$ . We do not have clear intuition to understand this behavior at the time of writing.
The previous derivations could be used to derive lower bound under highly specific priors on the structure of  $f^{*}$ . However, this proposition is somewhat deceptive compared to Theorems 2 and 3 that are more generic, easier to parse, and show more clearly the curse of dimensionality.
Proposition 16 (Lower-bound application example). For the Matérn kernel corresponding to  $\mathcal{F} = H^{\beta}$ , on the torus  $\mathbb{T}^d = \mathbb{R}^d / \mathbb{Z}^d$  with uniform measure  $\rho_{\chi}$ , when  $f^{*}: x \mapsto \cos(2\pi m^{\top}x)$  is a function with a single frequency  $m \in \mathbb{Z}^d$ ,
$$
\inf _ {\lambda > 0} \frac {\varepsilon^ {2} \mathcal {N} _ {2} (\lambda^ {- 1} K)}{n} + \mathcal {S} (\lambda^ {- 1} K) \geq \max _ {l \in [ d ]} \left(\frac {l \pi^ {(l - 1) / 2} \varepsilon^ {2}}{2 ^ {l - 1} \Gamma ((l - 1) / 2) n}\right) ^ {\gamma} \left(1 + \| m \| ^ {2}\right) ^ {\gamma l / 2},
$$
where  $\gamma = 4\beta /(4\beta +l)$  goes to one as  $\beta$  goes to infinity.
Proof. Using the bias and variance lower bounds decomposition and Proposition 11, we get, when  $f^{*} = f_{m}$  is a single frequency function  $\hat{f}(\omega) = \delta_{m}(\omega)$  with  $m \in \mathbb{Z}^d$ ,
$$
\begin{array}{l} \frac {\varepsilon^ {2} \mathcal {N} _ {2} (\lambda^ {- 1} K)}{n} + \mathcal {S} (\lambda^ {- 1} K) \geq \left(\frac {d \pi^ {(d - 1) / 2}}{2 ^ {d + 2} \Gamma ((d - 1) / 2) n} \lambda^ {- d / 2 \beta} + \frac {\lambda^ {2}}{\left((1 + \| m \| ^ {2}) ^ {- \beta} + \lambda\right) ^ {2}}\right) \\ \geq \frac {1}{2} \left(\frac {d \pi^ {(d - 1) / 2}}{2 ^ {d + 1} \Gamma ((d - 1) / 2) n} \lambda^ {- d / 2 \beta} + \min  (\lambda^ {2} (1 + \| m \| ^ {2}) ^ {2 \beta}, 1)\right). \\ \end{array}
$$
From the fact that