SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\inf  _ {x \in \mathbb {R}} a x ^ {- \alpha} + b x ^ {2} = \left(\alpha^ {2 / (2 + \alpha)} + \alpha^ {- \alpha / (2 + \alpha)}\right) b ^ {\alpha / (2 + \alpha)} a ^ {2 / (2 + \alpha)} \geq b ^ {\alpha / (2 + \alpha)} a ^ {2 / (2 + \alpha)},
$$
Number of samples
Number of samples
Figure 9: (Right) Noise-free convergence rates for  $f^{*}(x) = x_{1}^{5}$  with  $k(x,y)\propto (1 + x^{\top}y)^{5}$ . We observed similar deterioration of convergence rates as a function of the dimension  $d$  as on Figure 2. The fact that the error is not exactly zero when  $d = 1$  and  $n\geq 5$  is due to a small regularization added in our algorithm to avoid running into computational issues when inverting a matrix online. (Left) Convergence rates for  $f^{*}(x) = \cos (4\pi x_{1})$  on the torus  $\mathcal{X} = \mathbb{T}^d$  with the (periodic) exponential kernel  $k(x,y) = q(-100\| x - y\| ^2 /d)$ . We observe similar behavior as on Figure 2, the picture being worse because the kernel weights all frequencies in the Fourier domain, and not only the first  $\binom{d+5}{5}$  ones.
we have that, whatsoever  $\lambda$  is (if it is fixed for all  $n$  independently of the realization  $\mathcal{D}_n$ ).
$$
\begin{array}{l} \frac {d \pi^ {(d - 1) / 2}}{2 ^ {d + 1} \Gamma ((d - 1) / 2) n} \lambda^ {- d / 2 \beta} + \lambda^ {2} (1 + \| m \| ^ {2}) ^ {2 \beta} \\ \geq \left(\frac {d \pi^ {(d - 1) / 2}}{n 2 ^ {d - 1} \Gamma ((d - 1) / 2)}\right) ^ {4 \beta / (4 \beta + d)} \left(1 + \| m \| ^ {2}\right) ^ {2 \beta d / (4 \beta + d)}. \\ \end{array}
$$
Once again, this lower bound is also true when  $d$  is replaced by any  $l \in [d]$ .
C.1 Online solving of problems of increasing size
In order to solve a big number of least-squares problems with increasing numbers of samples, one can use recursive matrix inversion. When  $A \in \mathbb{R}^{n \times n}$ ,  $x \in \mathbb{R}^n$  and  $b \in \mathbb{R}$ , one can check that
$$
\left( \begin{array}{c c} A & x \\ x ^ {\top} & b \end{array} \right) ^ {- 1} = \left( \begin{array}{c c} A ^ {- 1} + c y y ^ {\top} & - c y \\ - c y ^ {\top} & c \end{array} \right),
$$
$$
c = \frac {1}{b - x ^ {\top} y}, \qquad \text {a n d} \qquad y = A ^ {- 1} x.
$$
This allows efficient computation of matrix inversion online as the number of samples increases.
C.2 Example of convergence rates for "atomic" functions
Polynomial estimation. Consider the target function  $f^{*}(x) = 63x_{1}^{5} - 70x_{1}^{3} + 15x_{1}$ , learned with the polynomial kernel  $k(x,y) = (1 + x^{\top}y)^{p}$  with  $p = 5$  on  $\mathcal{X} = [0,1]^d$  and  $\rho_{\mathcal{X}}$  uniform, in the interpolation regime  $\lambda = 0$ . In this setting,  $\mathcal{F}$  is exactly the space of polynomials of degree no larger than 5, which follows from the fact that  $k$  can be rewritten through a vector  $\varphi$  that enumerates monomials:
$$
(1 + x ^ {\top} y) ^ {p} = \sum_ {i = 0} ^ {p} \binom {p} {i} \sum_ {(i _ {j}) _ {j}: \sum_ {j} i _ {j} = i} \binom {i} {(i _ {j}) _ {j}} \prod x _ {j} ^ {i _ {j}} y _ {j} ^ {i _ {j}} = \varphi (x) ^ {\top} \varphi (y).
$$
As a consequence,  $f^{*}$  belongs to  $\mathcal{F}$ , and we expect the bias term to be zero. Yet, we expect the variance, hence the generalization error, to behave as  $\varepsilon^2 \dim \mathcal{F} / n$ , where  $\varepsilon$  corresponds to some notion of variability between labels. The dimension of the class of polynomials in dimension  $d$  of degree no larger than  $p$  is  $\binom{p+d}{d}$ . In particular, in dimension  $d = 100$ , a polynomial of degree at
most  $p = 5$  can have up to one hundred million coefficients, so that one needs about one hundred million observations to enter the high-sample regime and expect an excess risk  $\mathcal{E}(f_n)$  of order  $\varepsilon^2$  as per Theorem 4. While one could fit a polynomial of lower maximum degree, e.g. 4 instead of 5, since the  $f^*$  considered here is actually orthogonal to all polynomials of lower degree, there is no hope to obtain better rates. On Figure 2, the target function is  $f^*(x) = x_1^5$ , and the polynomial kernel is normalized as
$$
k (x, y) = \left(\frac {1 + x ^ {\top} y}{1 + d}\right) ^ {5}
$$
to avoid computational issues. The noise level  $\varepsilon$  is set to  $10^{-2}$ , and the lower bound in  $\varepsilon^2$  dim  $\mathcal{F} / n$  is plotted on Figure 2. In practice, this lower bound describes well the learning dynamic when the number of samples is high compared to the effective dimension of the space of functions considered.
Same examples in Fourier. To transpose the previous example in the Fourier domain, one can consider  $f^{*}(x) = \cos (\omega_{0}x_{1})$  together with some translation-invariant kernel. We illustrate this case on Figure 9. The deterioration of the rates with respect to dimension can be understood precisely. In harmonic settings, such as on the torus with uniform measure, one can consider  $f^{*}$  as an eigenfunction of the integral operator  $K$  associated with the eigenvalue  $\lambda_{\omega_0}$ . The lower bound on the bias is given by
$$
\mathcal {S} (\lambda^ {- 1} K) = \frac {\lambda^ {2}}{(\lambda + \lambda_ {\omega_ {0}}) ^ {2}}.
$$
When  $k$  is translation-invariant,  $k(x,y) = q(x - y)$ , we get that
$$
\operatorname {T r} (K) = \sum_ {\omega \in \mathbb {Z} ^ {d}} \lambda_ {\omega} = \operatorname {T r} \left(\mathbb {E} [ \varphi (X) \varphi (X) ^ {\top} ]\right) = \mathbb {E} [ \varphi (X) ^ {\top} \varphi (X) ] = \mathbb {E} [ k (X, X) ] = q (0).
$$
When  $q(0)$  does not depend on  $d$ , this quantity is constant. On the other hand, we expect  $\lambda_{\omega}$  to decrease with  $\| \omega \|$ . But since the number of frequencies below  $\| \omega_0 \|$  grows exponentially with the dimension, in order to keep this sum constant  $\lambda_{\omega_0}$  has to decrease exponentially fast with the dimension, hence the bias will increase exponentially fast with the dimension.
Example of "wrongfully" arbitrarily fast convergence rates. To further emphasize the importance of constants and transitory regimes, let us discuss an even simpler example. Assume that one wants to learn a polynomial of a unknown degree  $s \in \mathbb{N}$  in a noiseless setting; or equivalently, learn an analytical function such that  $f^{(s + 1)} = 0$  for an unknown  $s \in \mathbb{N}$ . This polynomial can be learned exactly when provided with as many points as the unknown coefficients in the polynomial, meaning that the generalization error will almost surely goes to zero when provided enough points. As a consequence,
$$
\forall h: \mathbb {N} \to \mathbb {R}, \qquad \mathcal {E} (f _ {n}) \leq O (h (n)),
$$
where  $f_{n}$  is defined in (7) with  $\mathcal{F}$  the space of polynomials of any degree. In other terms, we are able to prove arbitrarily fast convergence rates. Yet, such convergence rates hide constants that are the real quantities governing convergence behaviors of any learning procedure. Figure 9 shows how the number of coefficients in a Taylor expansion of order  $s$  is once again the right quantity to look at.
C.3 Different convergence rates profiles
Figure 4 was computed with the Gaussian kernel on either the one-dimensional torus or  $\mathbb{R}$ . One hundred runs were launched and averaged to get a meaningful estimate of the expected excess risk. Convergence rates were computed for different hyperparameters, and the best set of hyperparameters (changing with respect to the number of samples but constant over run) was taken to show the best achievable convergence rate.
Fast then slow profile. Let us focus on the example provided by  $f^{*}: x \mapsto \exp(-\max(x^{2}, M)) - \exp(-M)$  (note that we subtract  $\exp(-M)$  so that the function goes to zero at infinity, which removes the burden of learning a constant offset with the Gaussian kernel). Note that, for  $q_{\sigma} = \exp(-x^{2}/\sigma)$ , the convolution  $q_{\sigma} * f$  for a large  $\sigma$  will not modify  $f$  much, while making it analytical. This follows from Fourier analysis: if  $f$  is integrable, its Fourier transform is bounded; since a convolution corresponds to a product in Fourier, and since the Fourier transform of  $q_{\sigma}$  decays exponentially fast, so does  $f * q_{\sigma}$ , which implies its analytical property. As a consequence, all functions are close to
Figure 10: Excess risk when the target at the top is taken as  $f^{*}(x) = \exp (-\max (x^{2},M)) - \exp (M)$  with  $M = 1 / 4$ , and  $x\in \mathbb{R}$  with unit Gaussian distribution. Note that the learning of the smooth part is more efficient when the regularizer is big, which forces the reconstruction to be smooth.
Figure 11: Example of a smooth target function with a  $C^1$ -singularity whose estimation is expected to showcase convergence rates that decrease first fast and then slowly. The  $x$ -axis represents the input space  $\mathcal{X} = [-2,2]$ , the  $y$ -axis represent the output space  $f^{*}(x)$  and  $f_{n,\lambda}$  for  $n = 10^{2}$  and  $n = 10^{4}$ . The first fast decrease of excess risk corresponds to the easy estimation of the coarse details of the function, while the slow decrease thereafter corresponds to the precise estimation of the  $C^1$ -singularity. The target function  $x \mapsto \exp(-\max(x^2, 1)) - \exp(-1)$  is represented in blue, the estimation with  $10^{4}$  samples is represented in orange, and the one with  $10^{2}$  samples is represented in dashed green. The left picture zooms in on the estimation of the singularity. We see that the increase from  $10^{2}$  to  $10^{4}$  does not lead to a much better estimate.
analytical functions, whose approximation should exhibit convergence rates in  $O(n^{-1})$ . In particular, for  $f^{*}$  defined as before for a some  $M$  large enough, without enough observations one will not be able to distinguish between  $f^{*}$  and  $q_{\sigma} * f^{*}$ , and as the number of sample first increases, one will learn quite fast a smooth version of  $f^{*}$ . After a certain number of samples, the learning will stall until enough points are provided to distinguish between  $f^{*}$  and its smoothing, and learn the  $C^1$ -singularity of the former. Figures 10 and 11 illustrate this observation. Note that similar reasoning could be made for any RKHS that is dense in  $L^2(\rho_{\mathcal{X}})$ .
Slow then fast profile. The slow then fast profile was computed with  $\mathcal{X} = \mathbb{S}^1$ ,  $\rho_{\mathcal{X}}$  being uniform and  $f^{*}:x\mapsto \cos (2\pi \omega x)$  with  $\omega = 20$ . One hundred runs were launched and averaged to get an estimate of the excess risk of the estimator in (8) for different values of  $\sigma$  and  $\lambda$ . Again, the best results for different sample sizes were reported to get an estimate of convergence rates on Figure 4. A log-log-log-log plot of the results is provided on Figure 12.
C.4 Exploring the low-sample regime
Looking at the empirical weights. While the previous paragraph discusses the weights  $\alpha_{X}(x)$  when given access to the full distribution, similar derivations can be made when accessing a finite number of samples. Indeed, kernel ridge regression reads as
$$
f _ {\lambda , n} (x) = \sum_ {i \in [ n ]} Y _ {i} \widehat {\alpha} _ {i} (x), \qquad \widehat {\alpha} (x) = (\widehat {K} + n \lambda) ^ {- 1} \widehat {K} _ {x} \in \mathbb {R} ^ {n},
$$
$$
\widehat {K} = \big (k (X _ {i}, X _ {j}) \big) _ {i, j \in [ n ]} \in \mathbb {R} ^ {n \times n}, \qquad \widehat {K} _ {x} = \big (k (X _ {i}, x) \big) _ {i \in [ n ]} \in \mathbb {R} ^ {n}.
$$
Figure 12: Excess of risk when the target at the top is taken as  $f^{*}(x) = \cos (2\pi \omega x)$  with  $\omega = 20$ , and  $x \in \mathbb{S}^1 = \mathbb{R} / \mathbb{Z}$  uniform on the circle. Observe how the risk first stalls, before learning the function quite fast. The two graphs  $\{(n,\mathcal{N}_a(\lambda ,\sigma)\}$  for  $a \in \{1,2\}$  are plotted with the blue lines.
High-sample ratio: 0.32
High-sample ratio: 0.53
High-sample ratio: 1.06
High-sample ratio: 2.11
High-sample ratio: 4.23
High-sample ratio: 8.45
Figure 13: Same picture as Figure 6 yet with  $\sigma = .05$ , which leads to an effective dimension  $\mathcal{N} = 85$ .
Note that  $\widehat{\alpha}_i(x) = \widehat{\alpha}_{X_i|\mathbb{X}}(x)$  where  $\mathbb{X} = (X_1,\dots ,X_n)$  is the input dataset. As a consequence,
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ f _ {n} (x) \right] = \sum_ {i \in [ n ]} \mathbb {E} _ {\mathcal {D} _ {n}} \left[ Y _ {i} \widehat {\alpha} _ {X _ {i} | \mathbb {X}} (x) \right] = n \dots \mathbb {E} _ {\mathcal {D} _ {n}} \left[ Y _ {1} \widehat {\alpha} _ {X _ {1} | \mathbb {X}} (x) \right].
$$
In other terms,  $f_{n}$  is a bias estimator whose average is defined as
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ f _ {n} \right] = \mathbb {E} _ {(X, Y)} \left[ Y \widehat {\alpha} _ {X} \right], \quad \widehat {\alpha} _ {X} = n \cdot \mathbb {E} _ {\mathbb {X}} \left[ \widehat {\alpha} _ {X _ {1} | \mathbb {X}} \mid X _ {1} = X \right]. \tag {34}
$$
These are the weights plotted on Figures 6 and 13. In order to compute those weights efficiently, one can use the block matrix inversion. Using the sliced indices matrix notations, we have
$$
\begin{array}{l} \widehat {\alpha} _ {X _ {n} \mid \mathcal {D} _ {n}} (x) = [ (\widehat {K} + n \lambda) ^ {- 1} \widehat {K} _ {X} ] _ {n} = [ (\widehat {K} + n \lambda) ^ {- 1} ] _ {n,:} \times \widehat {K} _ {x} \\ = \left[ (\widehat {K} + n \lambda) ^ {- 1} \right] _ {n,: n - 1} \times \left[ \widehat {K} _ {x} \right] _ {: n - 1} + \left[ (\widehat {K} + n \lambda) ^ {- 1} \right] _ {n, n} \times \left[ \widehat {K} _ {x} \right] _ {n} \\ = - (b - x ^ {\top} A ^ {- 1} x) ^ {- 1} ([ \widehat {K} _ {x} ] _ {n} - x ^ {\top} A ^ {- 1} \times [ \widehat {K} _ {x} ] _ {n - 1}). \\ \end{array}
$$
$$
\widehat {K} + n \lambda I = \left( \begin{array}{c c} A & x \\ x ^ {\top} & b \end{array} \right) = \left( \begin{array}{c c} [ \widehat {K} ] _ {: n - 1,: n - 1} + n \lambda & [ \widehat {K} ] _ {n,: n - 1} \\ [ \widehat {K} ] _ {n,: n - 1} ^ {\top} & [ \widehat {K} ] _ {n, n} + n \lambda \end{array} \right).
$$
Denoting
$$
\tilde {K} = \big (k (X _ {i}, X _ {j}) \big) _ {i, j \in [ n - 1 ]} \in \mathbb {R} ^ {n - 1 \times n - 1} \tilde {K} _ {x} = \big (k (X _ {i}, x) \big) _ {i \in [ n - 1 ]} \in \mathbb {R} ^ {n - 1},
$$
$$
\widehat {\alpha} _ {X | \mathcal {D} _ {n}} (x) = \left(k (X, X) - Z _ {X} ^ {\top} Z _ {X} + n \lambda\right) ^ {- 1} \left(k (X, x) - Z _ {X} ^ {\top} Z _ {x}\right), \qquad Z _ {x} = (\tilde {K} + n \lambda) ^ {- 1 / 2} \tilde {K} _ {x}.
$$