SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {V} _ {n} \right] - \mathcal {N} _ {2} (K) \right| \leq \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left| \mathcal {V} _ {n} - \mathcal {N} _ {2} (K) \right| \right] \\ \leq 2 \operatorname {T r} (\Sigma) \mathbb {P} \left(\| F _ {n} \| > 1 / 2 c\right) + 4 c \mathbb {E} \left[ \| F _ {n} \| | \| F _ {n} \| \leq 1 / 2 c \right] \mathbb {P} \left(\| F _ {n} \| \leq 1 / 2 c\right). \\ \end{array}
$$
We are left with the computation of two integrals. As long as  $1 \leq \| \Sigma \|$ , with  $a$  the coefficient appearing in the exponential
$$
\begin{array}{l} \mathbb {P} (\| F _ {n} \| \leq 1 / 2 c) \mathbb {E} \left[ \| F _ {n} \| | \| F _ {n} \| \leq 1 / 2 c \right] \leq 2 8 \mathcal {N} _ {1} (K) \int_ {0} ^ {1 / 2 c} x \exp \left(- 3 n x ^ {2} / 2 \mathcal {N} _ {+} (K)\right) d x \\ = 2 8 \mathcal {N} _ {1} (K) a ^ {- 1} \int_ {0} ^ {a ^ {1 / 2 c} / 2} x \exp (- x ^ {2}) d x \leq \frac {1 0 \mathcal {N} _ {1} (K) \mathcal {N} _ {+} (K)}{n}. \\ \end{array}
$$
Once again, the condition (23) can be relaxed with the following assumption.
Assumption 4. There exists a constant  $c$  such that, for all  $i \in \mathbb{N}$ ,
$$
\mathbb {E} \left[ \left| \left| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right| \right| ^ {i} \mid \| F _ {n} \| \leq 1 / 2 c \right] \leq c ^ {i} \mathbb {E} \left[ \left| \left| F _ {n} \right| ^ {i} \right| \| F _ {n} \| \leq 1 / 2 c \right]. \tag {24}
$$
As for Assumption 3, Assumption 4 holds when  $\mathcal{F}$  is finite-dimensional with  $c^2 = \left\| K^{-1} \right\|\left(\|K\| + 1\right)$ . It holds in general with  $c^2 = \|\Sigma\| + 1$ , although this would deteriorate the bound by a factor of  $\lambda$  when considering  $\Sigma$  to be  $\lambda^{-1}\Sigma$ .
We are finally ready to collect the different pieces.
Proposition 4. When  $f^{*} \in (\ker K)^{\perp}$  and  $1 \leq \| \Sigma \|$ , under the technical Assumption 4, the variance term can be bounded from above and below by
$$
\frac {\varepsilon^ {2}}{n} \left| \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {V} _ {n} \right] - \mathcal {N} _ {2} (K) \right| \leq \varepsilon^ {2} \mathcal {N} _ {1} ^ {2} (K) \left(\frac {2 8 \operatorname {T r} (\Sigma)}{n} \exp \left(- \frac {c ^ {2} n}{(4 + 2 c) \mathcal {N} _ {+} (K)}\right) + \frac {4 0 c \mathcal {N} _ {+} (K)}{n ^ {2}}\right). \tag {25}
$$
A.3.4 Discussion to the variance bound
Once again, the bound presented here is somewhat unsatisfying, as it will not necessarily decrease faster than  $\mathcal{N}(K) / n$ . If one considers target functions that are far away from  $\mathcal{F}$  and requires a large search space (i.e.  $\mathcal{S}(\lambda^{-1}K)$  decreases slowly with  $\lambda$ , and given a fixed number of samples  $n$  the
optimal  $\lambda_{n}$  is found for  $\mathcal{N}(\lambda_nK)$  quite large compared to  $n$ ), so that  $\mathcal{N}_{+}(\lambda^{-1}K)\mathcal{N}(\lambda^{-1}K) / n$  does not go to zero. Several directions can be taken to improve the bound. For example, when  $Y$  is bounded by  $M$ , the noise  $\varepsilon^2$  can be replaced by  $M^2$ , and it is possible to get an upper bound of the form
$$
\mathbb {E} _ {\mathcal {D} _ {n}} [ \mathcal {E} (f _ {n} ^ {(\mathrm {t h r e s .})}) ] \leq \frac {8 M ^ {2}}{n} \mathcal {N} _ {1} (K) + \inf  _ {f \in \mathcal {F}} \left(\| f - f ^ {*} \| _ {L ^ {2}} ^ {2} + \| f \| _ {\mathcal {F}} ^ {2}\right),
$$
for a truncated version  $f_{n}^{(\mathrm{thres.})}$  of the estimator (8) as proved by Mourtada et al. [19] for ridge-less regression and extended to ridge regression in Mourtada et al. [20]. Moreover, retaking the analysis of Mourtada and Rosasco [18], one can get a lower bound of the form
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {n}} \mathcal {V} _ {n} \geq \frac {n}{n + 1} \mathbb {E} _ {\mathcal {D} _ {n + 1}} \operatorname {T r} \left(\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} \Sigma_ {n + 1} \Sigma_ {n + 1, (n + 1) / n} ^ {- 1}\right) \\ \geq \frac {n}{n + 1} \mathbb {E} _ {\mathcal {D} _ {n}} \operatorname {T r} \left(\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1}\right) - \mathbb {E} _ {\mathcal {D} _ {n + 1}} \operatorname {T r} \left(\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} \Sigma_ {n + 1, (n + 1) / n} ^ {- 1}\right), \\ \end{array}
$$
which might lead to some lower bound with
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {n + 1}} \operatorname {T r} \left(\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} \Sigma_ {n + 1, (n + 1) / n} ^ {- 1}\right) = \frac {n + 1}{n} \mathbb {E} _ {\mathcal {D} _ {n}, X} \operatorname {T r} \left(\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} \left(\Sigma_ {n, 1} + \varphi (X) \otimes \varphi (X)\right) ^ {- 1}\right) \\ \leq \frac {n + 1}{n} \mathbb {E} _ {\mathcal {D} _ {n}} \operatorname {T r} \left(\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1}\right) \mathbb {E} _ {X} \left[ \left\| (\Sigma_ {n, 1} + \varphi (X) \otimes \varphi (X)) ^ {- 1} \right\| \right]. \\ \end{array}
$$
We also note that it should not be too hard to replace  $F_{n}$  by  $\Sigma^{1 / 2}\Sigma_1^{-1}E_n\Sigma^{1 / 2}\Sigma_1^{-1}$ , which would lead to  $\varepsilon^2\mathcal{N}_2(K) / n$  instead of  $\varepsilon^2\mathcal{N}_1(K) / n$  in the right-hand side of (11).
A.3.5 Full theorem
Collecting the precedent results leads to the following theorem.
Theorem 8. Under the technical Assumptions 3 and 4, as long as  $1 \leq \| \Sigma\|$ , when  $f^{*}$  belongs to the closure of  $\mathcal{F}$  in  $L^2(\rho_{\mathcal{X}})$ , the estimator (8) verifies
$$
\begin{array}{l} \left| \mathbb {E} _ {\mathcal {D} _ {n}} [ \mathcal {E} (f _ {n}) ] - \frac {\varepsilon^ {2} \mathcal {N} _ {2} (K)}{n} - \mathcal {S} (K) \right| \leq 4 0 \mathcal {N} _ {1} (K) \left(a _ {n} \cdot \frac {\varepsilon^ {2} \mathcal {N} _ {1} (K)}{n} + a _ {n} ^ {1 / 2} \mathcal {S} (K)\right) \tag {26} \\ + 5 6 \mathcal {N} _ {1} (K) \left(\frac {\mathrm {T r} (\Sigma) \varepsilon^ {2}}{n} + \| f ^ {*} \| _ {L ^ {2}} ^ {2}\right) \exp (- c a _ {n}), \\ \end{array}
$$
where  $a_{n} = \mathcal{N}_{+}(K) / n$  and some constant  $c$ .
As long as its right-hand side decreases faster than  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$ , Theorem 8 states that  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$  behaves like
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} (f _ {n}) \right] \simeq \frac {\varepsilon^ {2} \mathcal {N} _ {2} (K)}{n} + \mathcal {S} (K).
$$
When optimizing for  $\lambda$  when  $K$  is actually  $\lambda^{-1}K$ , assuming that  $\mathcal{N}_1\simeq \mathcal{N}_2$ , the right-hand side of Theorem 8 decreases faster than  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$  if and only if  $\mathcal{N}_1(K)a_n^{-1 / 2}$  goes to zero with  $n$ . This implies  $\mathcal{N}(K)^3\leq n$ , which is a much stronger condition than the high-sample regime condition  $\mathcal{N}(K)\leq n$ .
A.4 Interpolation spaces, capacity and source conditions
This section discusses the values of  $\mathcal{N}(K)$  and  $S(K)$ , and prove the last statements of Theorem 4.
A.4.1 Variances
We begin with simple facts about the variance term.
Proposition 5 (Relation between variances). For any kernel  $k$ , we have the relation
$$
\mathcal {N} _ {2} (K) \leq \mathcal {N} _ {1} (K) \leq \mathcal {N} _ {+} (K). \tag {27}
$$
Figure 8: Effective dimensions  $\mathcal{N}_1$  (left) and  $\mathcal{N}_2$  (right) as a function of  $(\lambda, \sigma)$  in one dimension (top) and two dimension (bottom) when  $\rho_{\mathcal{X}}$  is uniform on  $\mathcal{X} = [-1, 1]^d$ , and  $k$  is the Gaussian kernel.
Proof. Once again, the precise study of the variance is easier in  $\mathcal{H}$  with the operator  $\Sigma$  rather than in  $L^2$ . First of all, notice that  $K = SS^{*}$  has the same spectrum as  $\Sigma = S^{*}S$ , so that, for  $a\in [1,2]$ ,
$$
\mathcal {N} _ {a} (K) = \operatorname {T r} \left(\left(K + 1\right) ^ {- a} K ^ {a}\right) = \operatorname {T r} \left(\left(\Sigma + 1\right) ^ {- a} \Sigma^ {a}\right) = \sum_ {\mu \in \operatorname {s p e c} (\Sigma)} \frac {\mu^ {a}}{\left(1 + \mu\right) ^ {a}}.
$$
This shows the first part of the inequality (27):
$$
\left(0 \leq \frac {x}{x + 1} \leq 1 \quad \Rightarrow \quad \frac {x}{x + 1} \leq \frac {x ^ {a}}{(x + 1) ^ {a}}\right) \quad \Rightarrow \quad \mathcal {N} _ {2} (K) \leq \mathcal {N} _ {1} (K).
$$
For the second part of the inequality, we need to reformulate  $\mathcal{N}_{+}(K)$ , which is the object of the next Lemma. In view of Lemma 6, the last inequality in (27) follows from
$$
\mathcal {N} _ {2} (K) = \mathbb {E} _ {X} \left[ \operatorname {T r} ((\Sigma + 1) ^ {- 1} \varphi (X) \otimes \varphi (X) \right] \leq \operatorname * {e s s} _ {X} \sup  \operatorname {T r} ((\Sigma + 1) ^ {- 1} \varphi (X) \otimes \varphi (X)) = \mathcal {N} _ {+} (K).
$$
The ends the proof of the proposition.
Lemma 6.  $\mathcal{N}_{+}(K)$  can be expressed in  $\mathcal{H}$  as
$$
\mathcal {N} _ {+} (K) = \operatorname * {e s s} _ {x \sim \rho_ {\mathcal {X}}} \left\| (\Sigma + 1) ^ {- 1 / 2} \varphi (x) \right\| ^ {2}.
$$
Proof. Observe that, for  $x \in \mathcal{X}$ ,
$$
\left\| (\Sigma + 1) ^ {- 1 / 2} \varphi (x) \right\| ^ {2} = \varphi (x) ^ {\top} (\Sigma + 1) ^ {- 1} \varphi (x) = \operatorname {T r} \left((\Sigma + 1) ^ {- 1} \varphi (x) \otimes \varphi (x)\right).
$$
Let us introduce the operator
$$
S _ {x}: \mathcal {H} \to L ^ {2} (\rho_ {\mathcal {X}}), \quad \theta \mapsto (x ^ {\prime} \mapsto \varphi (x) ^ {\top} \theta).
$$
$$
\langle S _ {x} \theta , g \rangle = \mathbb {E} [ g (X) \varphi (x) ^ {\top} \theta ] = \langle \theta , \mathbb {E} _ {X} [ g (X) \varphi (x) ] \rangle
$$
we get  $S_x^* g = \mathbb{E}[g(X)\varphi (x)]$ . Similarly, one can check that
$$
K _ {x} (g) (x ^ {\prime}) = (S _ {x} S _ {x} ^ {*} g) (x ^ {\prime}) = (S _ {x} (\mathbb {E} _ {X} [ g (X) ] \varphi (x))) (x ^ {\prime}) = \varphi (x) ^ {\top} \varphi (x) \mathbb {E} _ {X} [ g (X) ] = \mathbb {E} [ g ] k (x, x),
$$
$$
\Sigma_ {x} \theta = S _ {x} ^ {*} S _ {x} \theta = \mathbb {E} _ {X} [ \varphi (x) ^ {\top} \theta ] \varphi (x) = (\varphi (x) \otimes \varphi (x)) \theta ,
$$
from which we deduce that there exists  $\varepsilon \in \{-1,1\}$  such that
$$
\varepsilon \left\| (K + 1) ^ {- 1} K _ {x} \right\| = \operatorname {T r} \left(K + 1\right) ^ {- 1} K _ {x}) = \operatorname {T r} (\Sigma + 1) ^ {- 1} \varphi (x) \otimes \varphi (x)) = \left\| (\Sigma + 1) ^ {- 1 / 2} \varphi (x) \right\|.
$$
Necessarily  $\varepsilon = 1$  since the right term is positive. Taking the essential supremum ends the proof.
The following is a reinterpretation of Proposition 29 of Cabannes et al. [5].
Proposition 7 (Capacity condition under interpolation inequalities). When  $K^p(L^2(\rho_{\mathcal{X}}))$  is continuously embedded in  $L^\infty(\rho_{\mathcal{X}})$  with  $p \leq 1/2$ , there exists a constant  $c$  such that
$$
\mathcal {N} _ {+} \left(\lambda^ {- 1} K\right) \leq c \lambda^ {- 2 p}. \tag {28}
$$
When the function  $x \to k(x, x)$  is bounded, the RKHS associated with  $k$  verifies
$$
\mathcal {N} _ {+} (\lambda^ {- 1} K) = O (\lambda^ {- 1}).
$$
Proof. The continuous embedding means that there exists a constant  $c$  such that, for any  $\lambda \geq 0$ ,
$$
\| K ^ {p} f \| _ {\infty} \leq c \| f \| _ {2}.
$$
Stated in  $\mathcal{H}$ , we get
$$
\left\| S \theta \right\| _ {\infty} \leq c \left\| K ^ {- p} S \theta \right\| _ {2} = c \left\| \Sigma^ {1 / 2 - p} \theta \right\|
$$
for every  $\theta \in \mathcal{H}$ . In other terms,
$$
\operatorname * {e s s} _ {x} \sup  _ {\varphi (x) ^ {\top}} \left| \varphi (x) ^ {\top} \theta \right| \leq c \left\| \Sigma^ {1 / 2 - p} \theta \right\|.
$$
Let us denote by  $(\lambda_i,\theta_i)$  the eigenvalue decomposition of  $\Sigma$ . Then
$$
\operatorname * {e s s} _ {x} \sup  (\varphi (x) ^ {\top} \theta) ^ {2} \leq c ^ {2} \left\| \Sigma^ {1 / 2 - p} \theta \right\| ^ {2} = c ^ {2} \sum_ {i \in \mathbb {N}} \lambda_ {i} ^ {1 - 2 p} (\theta_ {i} ^ {\top} \theta) ^ {2}.
$$
When considering  $\theta = \theta_{i}$ , this leads to
$$
\left| \theta^ {\top} \varphi (x) \right| \leq c \lambda_ {i} ^ {1 / 2 - p}.
$$
$$
\begin{array}{l} \mathcal {N} _ {\infty} ^ {1 / 2} (\lambda^ {- 1} K) = \sup  _ {x} \left\| (\Sigma + \lambda) ^ {- 1 / 2} \varphi (x) \right\| = \sup  _ {x} \sup  _ {\theta ; \| \theta \| \leq 1} \theta^ {\top} \Sigma_ {\lambda} ^ {- 1 / 2} \varphi (x) \\ = \sup  _ {x} \sup  _ {\theta ; \| \theta \| \leq 1} \sum_ {i \in \mathbb {N}} \frac {\theta^ {\top} \theta_ {i} \theta_ {i} ^ {\top} \varphi (x)}{(\lambda + \lambda_ {i}) ^ {1 / 2}} \leq c \sup  _ {a; \sum a _ {i} ^ {2} \leq 1} \sum_ {i \in \mathbb {N}} \frac {a _ {i} \lambda_ {i} ^ {1 / 2 - p}}{(\lambda + \lambda_ {i}) ^ {1 / 2}} \\ = c \sup  _ {i \in \mathbb {N}} \frac {\lambda_ {i} ^ {1 / 2 - p}}{(\lambda + \lambda_ {i}) ^ {1 / 2}} = c \sup  _ {t \in \operatorname {s p e c} (K)} \frac {t ^ {1 / 2 - p}}{(\lambda + t) ^ {1 / 2}} \leq c \sup  _ {t \geq 0} \frac {t ^ {1 / 2 - p}}{(\lambda + t) ^ {1 / 2}} \\ = c (2 p) ^ {- p} (1 - 2 p) ^ {1 / 2 - p} \lambda^ {- p}, \\ \end{array}
$$
where the last equality follows from basic calculus.
The second inequality follows from the fact that  $K^{1/2}(L^2) \hookrightarrow L^\infty$  as soon as  $\varphi$  is bounded since, for any  $f = \varphi(\cdot)^\top \theta \in \mathcal{F} = SH = K^{1/2}(L^2)$ ,
$$
| f (x) | = \left| \varphi (x) ^ {\top} \theta \right| \leq \| \varphi \| _ {\infty} \| \theta \| = \| \varphi \| _ {\infty} \| f \| _ {\mathcal {F}} = \| \varphi \| _ {\infty} \left\| K ^ {- 1 / 2} f \right\| _ {2}.
$$
The previous characterization of  $\mathcal{N}_{+}$  leads to the claim.
A.4.2 Bias
We now focus our attention on the bias term.
Proposition 8 (Source condition). When  $f^{*} \in K^{q}(L^{2}(\rho_{\mathcal{X}}))$  with  $q \leq 1$ , there exists a constant  $c$  such that, for any  $\lambda \geq 0$ ,
$$
\mathcal {S} \left(\lambda^ {- 1} K\right) \leq c \lambda^ {2 q}. \tag {29}
$$
$$
\mathcal {S} \left(\lambda^ {- 1} K\right) \leq 2 \inf  _ {q \in [ 0, 1 ], f \in K ^ {q} \left(L ^ {2}\right)} \lambda^ {2 q} \| K ^ {- q} f \| _ {2} ^ {2} + \| f - f ^ {*} \| _ {2} ^ {2}. \tag {30}
$$
Proof. The proof is straightforward. If  $f^{*} = K^{q}g$  with  $g \in L^{2}$ , then
$$
\begin{array}{l} \lambda^ {- 2} \mathcal {S} (\lambda^ {- 1} K) ^ {1 / 2} = \left\| (K + \lambda) ^ {- 1} f ^ {*} \right\| _ {2} = \left\| (K + \lambda) ^ {- 1} K ^ {q} g \right\| _ {2} \\ \leq \left\| (K + \lambda) ^ {q - 1} \right\| _ {2} \left\| (K + \lambda) ^ {- q} K ^ {q} \right\| _ {2} \| g \| _ {2} \leq \lambda^ {q - 1} \left\| K ^ {- p} f ^ {*} \right\| _ {2}. \\ \end{array}
$$
Squaring this term and multiplying it by  $\lambda^2$  leads to the result.
The second equation is due to the decomposition
$$
\begin{array}{l} \lambda^ {- 2} \mathcal {S} (\lambda^ {- 1} K) = \left\| (K + \lambda) ^ {- 1} f ^ {*} \right\| _ {2} ^ {2} \leq 2 \left\| (K + \lambda) ^ {- 1} f \right\| _ {2} ^ {2} + 2 \left\| (K + \lambda) ^ {- 1} (f ^ {*} - f) \right\| _ {2} ^ {2} \\ \leq 2 \left\| (K + \lambda) ^ {- 1} f \right\| _ {2} ^ {2} + 2 \lambda^ {- 2} \left\| f ^ {*} - f \right\| _ {2} ^ {2}. \\ \end{array}
$$
The previous derivations explain the final result.
A.5 Application of Theorem 4 to Taylor expansions
A natural idea to estimate a target function  $f^{*} \in C^{\alpha}$ , with  $C^{\alpha}$  the space of  $\alpha$ -Hölder functions, is to estimate its Taylor expansion. This is done with local polynomials which consists in concatenating into  $\varphi(x)$  a finite number of features of the form  $x \to \mathbf{1}_{x \in A} x^{k}$  for  $k \in [\beta]$  for some  $\beta \in \mathbb{N}$  and  $A$  in a partition of  $\mathcal{X}$ ; together with the ridgeless estimator of (7). In this setting  $\mathcal{N}_2(K, 0)$  is equal to the number of polynomials of degree at most  $\beta$  with  $d$  variables times the size of the partition considered. The number of such polynomials is  $h_{\beta}(1_d)$ , where  $h_{\lfloor \beta \rfloor}$  is the complete homogeneous symmetric polynomial of degree  $\beta$  in  $d$  variables and  $1_d$  is the vector of all ones. Thus,
$$
\mathcal {N} _ {2} (K, 0) = h _ {\beta} (1 _ {d}) = \left( \begin{array}{c} d + \beta \\ \beta \end{array} \right) = \frac {(d + \beta) !}{d ! \beta !} \in \left(1 + \frac {\beta}{d}\right) ^ {d} \cdot \left[ 1, e ^ {d} \right],
$$
where the last inequalities is due to the fact that  $\binom{n}{k} \in \left[(n/k)^k, (ne/k)^k\right]$ .
When  $\mathcal{X} = [0,1)$  with uniform distribution, and  $f = f^{*}$  is assumed to be  $(\alpha ,L_{\alpha})$  -Holder,i.e.,
$$
\left| f ^ {(\lfloor \alpha \rfloor)} (x) - f ^ {(\lfloor \alpha \rfloor)} (y) \right| \leq L _ {\alpha} | x - y | ^ {\alpha - \lfloor \alpha \rfloor},
$$
by fitting Taylor expansions on intervals  $[(i - 1) / m, i / m)$  for  $i \in [m]$  and  $m \in \mathbb{Z}_+$  with polynomials
$$
\varphi (x) = \left(\left(x - \frac {2 i - 1}{2 m}\right) ^ {j} \cdot {\bf 1} _ {x \in [ \frac {i - 1}{m}, \frac {i}{m})}\right) _ {i \in [ m ], j \in [ 0, \lfloor \alpha \rfloor ]},
$$
one can ensure that [see 12, Lemma 11.1]
$$
\| \Pi_ {\mathcal {F}} f ^ {*} - f ^ {*} \| _ {2} \leq \| \Pi_ {\mathcal {F}} f ^ {*} - f ^ {*} \| _ {\infty} \leq \frac {L}{2 ^ {\alpha} \left\lfloor \alpha \right\rfloor ! m ^ {\alpha}},
$$
where  $\Pi_{\mathcal{F}}$  denotes the orthogonal projection from  $L^2$  onto  $\mathcal{F}$ . The same type of result also holds for  $\mathcal{X} = [0,1]^d$  when using  $m^d$  multivariate polynomials of degree less than  $\lfloor \alpha \rfloor$ . Balancing the bias and the variance term, we get an excess risk that behaves as (up to higher order term)
$$
\mathbb {E} \left[ \| f _ {n} - f ^ {*} \| ^ {2} \right] \lesssim \inf _ {m \in \mathbb {Z} _ {+}; \beta \in [ \alpha ]} \left(\frac {\varepsilon^ {2} (m e) ^ {d} (1 + \beta / d) ^ {d}}{n} + \frac {L _ {\beta} ^ {2}}{2 ^ {2 \beta} \beta ! ^ {2} m ^ {2 \beta}}\right) = \inf _ {\beta \in [ \alpha ]} c _ {\beta} n ^ {- 2 \beta / (d + 2 \beta)},
$$
for some constant  $c_{\beta}$  that depends on  $\beta$ ,  $d$  and grows with  $L_{\beta}$ ,  $\sigma^2$ , the infimum being found for  $m \propto (L_{\beta}^2 n)^{1/(d+2\beta)}$ . This argument of the minimizer illustrates how the size of the window should depend on how smooth  $f^*$  is expected to be among the functions in  $C^\alpha$ . Interestingly, it equally shows that the partition size does not deteriorate with the dimension of the input space. Nor will deteriorate the percentage of the total volume contained in each region of the partition. However, the radius of those regions will scale as  $r = v^{1/d}$  for  $v$  the volume of those regions, meaning that when this volume will shrink to zero, the radius will shrink slower as the dimension grows, which will lead to a slower minimization of the approximation error. Ultimately as  $n$  grows, the last upper bound will end up behaving according to the "stationary behavior" in  $c_{\alpha} n^{-2\alpha/(2\alpha+d)}$ .
A.6 Application of Theorem 4 to Fourier expansions
Rather than local properties, one could leverage global smoothness properties, such as fast decays of Fourier coefficients, e.g.  $f^{*} \in H^{\alpha}$  with  $H^{\alpha}$  the Sobolev space of functions that are  $\alpha$ -times differentiable with derivatives in  $L^2(\rho_{\mathcal{X}})$ .
Fourier coefficient estimators can be built implicitly built from translation-invariant kernels, i.e.  $k(x,x^{\prime}) = \lambda^{-1}q((x - x^{\prime}) / \sigma)$  for  $q:\mathbb{R}^d\to \mathbb{R}$ ,  $\sigma$  a bandwidth parameter and  $\lambda >0$  a regularization parameter. Examples are provided by the Matérn kernels, defined from  $\hat{q} (\omega) = (1 + \| \omega \| ^2)^{-\beta}$ , the exponential kernel, which corresponds to a Matérn kernel of low smoothness  $\beta = (d + 1) / 2$ , and the Gaussian kernel, which can be seen as the limit of a Matérn kernel to infinite smoothness, defined as  $q(x) = \exp (-\| x\|^{2})$ . The corresponding estimators 8 can be proven to guarantee the convergence rates as Theorem 1, namely
$$
\mathbb {E} \left[ \| f _ {n} - f ^ {*} \| ^ {2} \right] \leq \inf  _ {\beta <   \alpha} c _ {\beta} n ^ {- 2 \beta / (2 \beta + d)},
$$
where  $c_{\beta}$  notably relates to the norm of  $f^{*}$  in  $H^{\beta}$ .
Proof. For the Matérn kernels, the generalization error reads, with  $\tau = 2\beta - d$ , up to constants and higher order terms, according to Table 1,
$$
\mathbb {E} \left[ \| f _ {n} - f ^ {*} \| ^ {2} \right] \lesssim \frac {\left(\sigma^ {\tau} \lambda\right) ^ {- d / 2 \beta}}{n} + \left(\sigma^ {\tau} \lambda\right) ^ {\alpha / \beta}.
$$
This is optimized for
$$
\sigma^ {\tau} \lambda = n ^ {- 2 \beta / (2 \alpha + d)},
$$
leading to minimax convergence rates in  $O(n^{-2\alpha/(2\alpha + d)})$ .
For the Gaussian kernel, we get
$$
\mathbb {E} \left[ \| f _ {n} - f ^ {*} \| ^ {2} \right] \lesssim \frac {\sigma^ {- d} \log (\lambda^ {- 1} \sigma^ {d}) ^ {d / 2}}{n} + \sigma^ {2 \alpha} \log (\lambda^ {- 1} \sigma^ {d}) ^ {- \alpha},
$$
which is optimized for
$$
\sigma^ {- 2} \log (\lambda^ {- 1} \sigma^ {d}) = n ^ {2 / (2 \alpha + d)},
$$
leading to the same minimax convergence rate. In particular, when  $\sigma$  is fixed, this leads to
$$
\lambda = \lambda_ {n} = \sigma^ {d} \exp (- \sigma^ {2} n ^ {2 / (2 \alpha + d)}).
$$
Based on Theorem 4, this is true as long as  $\mathcal{N}(\lambda)\mathcal{N}_{\infty}^{1/2}(\lambda)/n^{1/2}$  goes to zero with  $n$ , which imposes some constraints on  $\alpha$  when assuming  $f^* \in H^\alpha$ . However, considering a generalization of Mourtada et al. [20] from linear regression to RKHS, the upper bound is actually true without this constraint, which allows to prove the convergence rates in  $O(n^{2\alpha/(2\alpha+d)})$  for any  $\alpha$ .
A.7 Application of Theorem 4 to Sobolev spaces
We now turn ourselves to an informal more generic reformulation of the previous facts on Fourier expansions estimation based on well-known facts in approximation theory [30, 10].
Proposition 9 (Informal source condition). When  $\mathcal{F} = H^{\beta}$  and  $f^{*}\in H^{\alpha}$ , it holds
$$
f ^ {*} \in K ^ {\alpha / 2 \beta} (L ^ {2} (\rho_ {\mathcal {X}})).
$$
Proof. In essence, as explained in Appendix B,  $K$  takes a function in  $L^2(\rho_{\mathcal{X}})$  and multiply its Fourier transform by  $\hat{q}(\omega)^{-1} = (1 + \| \omega \|^2)^\beta$ , with  $q$  defining the Matérn kernel, making it  $2\beta$ -smooth in the Sobolev sense. In harmonic settings where the Fourier functions diagonalize  $K$  and  $\hat{q}(\omega)$  parameterizes the spectrum of  $K$ , the fractional operator  $K^p$  can be seen as multiplying the Fourier transform of  $f$  by  $\hat{q}(\omega)^{-p}$ , making it  $2p\beta$ -smooth. This fact can be extended beyond those harmonic settings, notably with interpolation inequalities. On the opposite direction, any  $\alpha$ -smooth function can be multiplied by  $q(\omega)^{\alpha/2\beta}$  in Fourier while staying in  $L^2(\rho_{\mathcal{X}})$ , so that, if  $f^*$  is  $\alpha$ -smooth, it belongs to  $K^{\alpha/2\beta}$ .
Proposition 10 (Informal interpolation inequality). When  $\mathcal{F} = H^{\beta}$  is the space of  $\alpha$ -Sobolev functions,
$$
K ^ {d / 2 \beta} (L ^ {2}) \hookrightarrow L ^ {\infty}.
$$
Proof. Note that  $\mathcal{F} = S\mathcal{H} = K^{1/2}(L^2)$ . We have seen informally in the proof of the previous lemma how  $K^p(L^2) \subset H^{2p\beta}$ . Now, let us recall the Sobolev embedding theorems [1]. Under mild assumptions on  $\rho_{\mathcal{X}}$ , for  $k, r, l, s > 0$
$$
W ^ {k, r} (\rho_ {\mathcal {X}}) \hookrightarrow W ^ {l, s} (\rho_ {\mathcal {X}}), \qquad \text {a s l o n g a s} \qquad \frac {1}{r} - \frac {k}{d} \leq \frac {1}{s} - \frac {l}{d}.
$$
We want to use it with  $k = 2p\beta$ ,  $r = 2$ ,  $l = 0$  and  $s = +\infty$ , which leads to  $p = 4\beta / d$ .
These results could be used to explain the scaling with respect to  $\lambda$  in Table 1, which we will derive formally in Appendix B. They can also be used to show convergence rates in  $O(n^{-2\alpha /(2\alpha +d)})$  for any target functions  $f^{*}\in H^{\alpha}$  when utilizing a kernel such that  $\mathcal{F} = H^{\beta}$  (in terms of sets equally,  $\mathcal{F}$  being eventually endowed with an other norm than Sobolev norms).