SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for  $q:\mathbb{R}^d\to \mathbb{R}$  a basic function,  $\sigma$  a bandwidth parameter, and  $\lambda$  a regularization parameter. However, the number of frequencies smaller than a cut-off frequency, i.e., the number of trigonometric monomials  $x\mapsto e^{im^{\top}x}$  with  $m\in \mathbb{Z}^d$  such that  $\| m\| \leq \omega$ , grows exponentially with the dimension, and this approach will not escape from the curse of dimensionality.
Other approaches, such as windowed Fourier estimation, or wavelets expansion estimation (aiming to reconstruct both fine local details together with coarse large-scale behaviors), could be thought of and described through the lens of RKHS. Yet, whatsoever the definition of smoothness considered (i.e., Hölder, Sobolev, Besov), all those methods will hit an inherent limit: the number of "smooth" functions increases really fast as the dimension increases. Indeed, since their proofs simply consist in finding  $D$  linearly independent function, lower bound theorems akin to Theorems 2 and 3 could be derived without difficulties for other notions of smoothness.
All the previously described methods are usually endowed with a few hyperparameters that modify the integral operator  $K$  and the norm  $\| \cdot \|_{\mathcal{F}}$  defining the estimator (8). Geometrically, a change of the front regularization parameter  $\lambda$  leads to an isotropic rescaling of the ball  $\| \cdot \|_{\mathcal{F}}^{-1}\{1\}$  inside  $L^2 (\rho_X)$ , while a change of other hyperparameters could favor certain directions, or even remove some functions in  $\mathcal{F}$ . In practice, fitting hyperparameters through cross-validation can be understood as implicitly searching to balance and minimize  $\mathcal{N}(K)$  and  $\mathcal{S}(K)$ . This fitting allows all those methods to reach the performance of Theorem 1 in  $O(n^{-2\alpha /(2\alpha +d)})$  as we explain in Appendix.
Kernel	F	N(σ,λ)	S(σ,λ;Hα)
Gaussian	FCω	σ−d log(λ−1σd)d/2	σ2α log(λ−1σd)−α
Matérn	Hβ	σ−d(2β−d)/2βλ−d/2β	σ(2β−d)α/βλα/β
Exponential	H(d+1)/2	(σλ)−d/(d+1)	(σλ)2α/(d+1)
Table 1: Example of translation-invariant kernels, their associated function classes, upper bounds (up to multiplicative constants) on their sizes as a function of the bandwidth  $\sigma$  and regularization parameter  $\lambda$ , as well as on the bias when approximating a function in the Sobolev space  $H^{\alpha}$ . Here  $C^{\omega}$  stands for the set of analytical functions. Proofs and details are to be found in Appendix B.
3.3 Exploration of Transitory Regimes
Given  $n$  samples, Theorem 4 suggests to tune  $K$  so as to minimize  $\varepsilon^2\mathcal{N}(K) / n + \mathcal{S}(K)$ . Interestingly, while theory tends to focus on deriving convergence rates in  $O(n^{-2\alpha /(2\alpha +d)})$  that maximize the coefficient  $\alpha$ , Theorem 4 can also be leveraged to characterize tightly the expected decay of the generalization error when accessing a small number of samples. To ground the discussion, we will focus on a stylized setting where  $\mathcal{N}$  and  $S$  can be studied in detail as a function of hyperparameters.
Proposition 1 (Capacity and bias bounds). When  $\rho_{\mathcal{X}}$  is uniform on the torus  $\mathbb{T}^d = \mathbb{R}^d / \mathbb{Z}^d$ , and  $k$  is a translation-invariant kernel  $k(x, y) = \lambda^{-1} q((x - y) / \sigma)$ , the capacity of the space defined through  $k$  with regularization  $\lambda$  and bandwidth  $\sigma$  verifies, for  $a \in \{1, 2\}$ ,
$$
\mathcal {N} _ {a} (\sigma , \lambda) = \int_ {\mathbb {R} ^ {d}} \left(\frac {\widehat {q} (\sigma \omega)}{\widehat {q} (\sigma \omega) + \lambda \sigma^ {- d}}\right) ^ {a} \# (\mathrm {d} \omega), \tag {14}
$$
where  $\#$  is the counting measure on  $\mathbb{Z}^d\subset \mathbb{R}^d$ , and  $\widehat{q}$  is the (discrete) Fourier transform of  $q$ . Similarly, the biases quantifying the adherence of  $f^{*}$  in  $\mathcal{F}$  verify
$$
\mathcal {S} (\sigma , \lambda) = \int_ {\mathbb {R} ^ {d}} \frac {\left| \widehat {f} ^ {*} (\omega) \right| ^ {2}}{\left(\sigma^ {d} \widehat {q} (\sigma \omega) + \lambda\right) ^ {2}} \# (\mathrm {d} \omega). \tag {15}
$$
Moreover on  $\mathcal{X} = \mathbb{R}^d$ , if  $\rho_{\mathcal{X}}$  has a density bounded above by  $\rho_{\infty} < +\infty$ , then (14) and (15) become upper bounds for  $\mu$  the Lebesgue measure and  $\widehat{f}$  the continuous Fourier transform, at the cost of extra constants in front of their right-hand sides (respectively  $\rho_{\infty}$  and  $\max(\rho_{\infty}, 1)$  for  $\mathcal{N}_a$  and  $\mathcal{S}$ ).
Proof Sketch. This relatively standard fact follows from the assumption on  $\rho_{\mathcal{X}}$  which implies that  $K$  is diagonal in the Fourier domain.
Proposition 1 unlocks a precise sense of the effective dimension for the Gaussian kernel, defined with  $q(x) = \exp(-\|x\|^2)$ , the exponential kernel, with  $q(x) = \exp(-\|x\|)$ , and the Sobolev kernel, with  $\widehat{q}(\omega) = (1 + \|\omega\|^2)^{-\beta}$ , as well as the bias term  $S$  when approximating a function  $f \in H^\alpha$  with those kernels. This is reported in Table 1 and proved in Appendix B.
High-sample regimes in harmonics settings. Proposition 1 is useful to describe formally different convergence rates profiles that one may expect in practice. In particular, the linearity of the bias characterization (15) is theoretically useful to decorrelate the estimation of different power laws appearing in the Fourier transform of  $f$ . More precisely, if
$$
\left| \widehat {f} ^ {*} (\omega) \right| ^ {2} = \int_ {\alpha} ^ {\infty} c _ {\gamma} (1 + \| \omega \| ^ {2}) ^ {- \gamma} \mu (\mathrm {d} \gamma),
$$
with  $c_{\gamma}$  being the inverse of the constant in front of the characterization of  $S(\sigma, \lambda; H^{\gamma})$  in Table 1, and  $\mu$  some measure with a profile that ensures the good definition of  $f^{*} \in H^{\alpha}$ , we have, taking for example  $k$  as the Gaussian kernel,
$$
\mathcal {S} (\sigma , \lambda ; f ^ {*}, k) \simeq \int_ {\alpha} ^ {\infty} \left(\sigma^ {2} \log \left(\lambda^ {- 1} \sigma^ {d}\right) ^ {- 1}\right) ^ {\gamma} \mu (\mathrm {d} \gamma).
$$
For example, when  $\mathcal{F} = \mathrm{im}K^{1 / 2} = H^{\beta}$  and  $f^{*}\in H^{\alpha}$ , it typically holds that  $\mathcal{N}_{+}(\lambda^{-1}K) = O(\lambda^{-2p})$  with  $p = d / 4\beta$ , and  $\mathcal{S}(\lambda^{-1}K) = O(\lambda^{2r})$  with  $r = \alpha /2\beta$ , which can be used to prove Theorem 1 by tuning  $\lambda = n^{-2\beta /(2\alpha +d)}$ .
Figure 4: Composite convergence rates profile. The  $x$ -axis corresponds to the number of samples, while the  $y$ -axis corresponds to the excess risk. From the analysis in Section 3.3, one can build different convergence rates profiles. For example, regular functions with relatively high-frequencies are going to be hard to learn with few samples but really easy after a certain number of samples, roughly equals to the number of harmonics with lower-frequencies); while regular low-frequency functions with singularity are going to show convergence rates where the coarse details of the functions are learned with few samples, but the reconstructions of fine-grained details will required much more samples. The former profile is illustrated with the left figure, and the latter on the right figure. For any sample sizes, excess risk is reported for the best hyperparameters, found with cross-validation. More details are provided in Appendix C.3.
In particular, with  $\sigma^2\log (\lambda^{-1}\sigma^d)^{-1} = n^{-r}$ , we get the following convergence rate profile, with  $c_{\mathcal{N}}$  the constant in front of the characterization of  $\mathcal{N}(\sigma ,\lambda)$  in Table 1,
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} \left(f _ {n}\right) \right] \simeq \inf  _ {r \in \mathbb {R}} \left(\varepsilon^ {2} c _ {\mathcal {N}} n ^ {- 1 + r d / 2} + \int_ {\alpha} ^ {\infty} n ^ {- \gamma r} \mu (\mathrm {d} \gamma)\right) (1 + O \left(a _ {n}\right)). \tag {16}
$$
This characterization enables us to easily create target functions exhibiting different convergence profiles, as long as we stay in the high-sample regimes where our bounds are meaningful (i.e. when the factor in  $1 + O(a_{n})$  is relatively constant).
- Fast then slow profile. The first type of profile is built from  $\mu$  that charges most of its mass on fast decays, but also puts some small mass on slow decays. It corresponds to target functions that are roughly well approximated by highly smooth functions, but whose exact reconstruction needs to incorporate less regular functions. The smooth part of the function will be learned quickly, yet the non-smooth part will be learned slowly. Typical examples of such a profile are provided by non-smooth functions that can be turned into infinitely differentiable ones after introducing infinitesimal perturbations, such as  $f^{*}(x) = \exp(-\min(|x|^{2}, M))$ , which is only  $C^{0}$ , but where one can expect to learn fast before stalling to estimate the  $C^{1}$ -singularity. Another example is given by a function made of a sum of one low-frequency cosine with large amplitude easy to learn together with one high-frequency cosine with small amplitude much harder to learn. We illustrate these profiles on Figure 4.
- Slow then fast profile. The second type of profile that can be created is for functions that are supported on a few eigenfunctions of  $K$  associated with small eigenvalues. A typical example of this profile in one dimension would be  $f^{*}(x) = \cos (\omega x)$  for a high frequency  $\omega$ . For this target function, no meaningful learning can be done when the search space is too small, because small search spaces do not contain high-frequency functions. On the other hand, when the search space is big enough, the bias quickly goes to zero, allowing for fast learning as long as one controls the estimation error. When provided with few samples, one would prefer a small search space to avoid blowing up of the estimation error, and learning will stall until enough samples are collected to explore bigger search spaces, where  $f^{*}$  could be learned quickly. We illustrate this profile on Figure 4.
Those examples illustrate how, given a target function and a range on the number of available samples, convergence behaviors might fall in regimes that do not correspond to the steady convergence rate in  $O(n^{-2\alpha / 2\alpha + d})$  predicted by Theorem 1. The intuition beyond those examples is not specific to translation-invariant kernels in harmonics settings, but holds more generically for abstract RKHS.
Empirical study of low-sample regimes. In this work, we have focused on "under-parameterized" situations where the parameters were set to have more samples than the effective dimension of the
Figure 5: Level lines of the weights  $x \to \alpha_x(x_0)$  (17) for a given  $x_0 \in \mathcal{X}$ , when  $\mathcal{X}$  is the torus  $\mathbb{R}^2/\mathbb{Z}^2$  and the kernel is taken as the Gaussian kernel with the Riemannian metric on the torus (think of an unrolled donut). Parameters are taken as  $\sigma = 1$  together with  $\lambda = 10^6$  (left),  $\lambda = 10^2$  (middle) or  $\lambda = 1$  (right). From this picture, one can build examples of non-smooth functions where the kernel inductive bias will have adversarial effects.
resulting functional space  $\mathcal{F}_{t(n)}$ . In a deep learning world, where many phenomena are understood as taking place in the "over-parameterized" regime, it is of interest to compare our perspective with the double descent phenomenon. Figure 1 shows the excess risk as a function of two of the three parameters  $(n,\sigma ,\lambda)$ , as well as the graph defined by  $\{(n,\sigma ,\lambda)\mid \mathcal{N}_1(\lambda ,\sigma) = n\}$ . It illustrates a double descent phenomena with "phase" transition governed by the passage from the low-sample to the high-sample regime. To delve into this, we investigate into regression weight learned by kernel ridge regression. When given access to the knowledge of the full distribution  $\rho$ , the estimator in (8) can be rewritten as
$$
f _ {\infty , \lambda} = \mathbb {E} [ Y \alpha_ {X} ], \quad \alpha_ {X}: x \rightarrow (K + \lambda I) ^ {- 1} k (X, x). \tag {17}
$$
As such, kernel ridge regression can be seen as learning in an unsupervised fashion the weights  $\alpha : \mathcal{X} \to L^2(\rho)$ , which then indicate how to fold the input space to use information provided by the labels. At a high level, one can think of a scheme, given some input points, to perform finite differences and leverage the result to build an estimate of the target function from Taylor expansions, whatsoever would be the label observations. Figure 5 shows how, when  $\lambda$  is not too big, the reconstruction  $f_{\infty, \lambda}(x_0)$  ( $x_0$  being the same point at the bluest center on the different pictures on this Figure) depends on observations made far away from  $x_0$  according to some periodic pattern, implicitly assuming that the target function should be regular when looked at in the Fourier domain. Similarly, one can look at the weights  $\widehat{\alpha}_X$  satisfying  $\mathbb{E}_{\mathcal{D}_n}[f_n(x)] = \mathbb{E}_{(X,Y)}[\widehat{\alpha}_X(x)Y]$ , and whose closed form is given in Appendix C.4. Those weights are shown on Figures 6. They present weird behaviors when the number of data  $n$  is closed to the search space size  $\mathcal{N}_2(K)$ . Note that this double descent phenomenon actually takes place in the regularized setup, and not in the interpolation regime, contrarily to prior works on the matter [e.g. 28, 21].
In this paper, we have shown how subtle is the saying that smoothness allows to break the curse of dimensionality. In essence, without implicit bias and in presence of noise, one needs to be in the high-sample regime where the size of the search space  $\mathcal{F}$  is smaller than the number of samples to avoid overfitting. As the input dimension grows, many more smooth functions can be defined. This constrains the diversity of functions within  $\mathcal{F}$ , which will typically be devoid of fine-grained details (linked with high-order, eventually trigonometric, polynomials), hence unable to harness high-order smoothness without accessing a large number of samples  $n$ .
Future work. Since we have shown that smoothness alone is not a strong enough prior to build efficient learning algorithms in high-dimensions, other priors could be investigated. As such, sparsity assumptions, multi-index models, feature learning or multi-scale behaviors might offer more realistic models to break the curse of dimensionality. How deep learning models exploit such priors has been an active line of research, although linking theoretical results with "interpretable" observations in neural networks remains challenging, and theory has not yet provided that many meaningful insights for practitioners.
High-sample ratio: 0.60
High-sample ratio: 1.00
High-sample ratio: 2.00
High-sample ratio: 4.00
High-sample ratio: 8.00
High-sample ratio: 16.00
Figure 6: Weights  $x \mapsto \widehat{\alpha}_x(x_0)$  such that  $\mathbb{E}_{\mathcal{D}_n}[f_n(x)] = \mathbb{E}_{(X,Y)}[\hat{\alpha}_X(x)Y]$ , for  $x_0 = -1/2$  (blue),  $x_0 = 0$  (orange) and  $x_0 = 1/2$  (green), when  $\mathcal{X} = [-1,1]$  and  $\rho \chi$  is uniform. The weight  $\alpha_x(x_0)$  quantifies how much the prediction at  $x_0$  is inferred from the output observed at the point  $x$ . The weights are computed with the Gaussian kernel with bandwidth  $\sigma = .1$  and  $\lambda = 10^{-5}$ , which yields an effective dimension  $\mathcal{N}_2(K) = 45$ , and for  $n \in \{27,45,90,180,360,720\}$ , which explains the high-sample ratio  $n / \mathcal{N}_2(K)$  seen on the title of the different plots. When this ratio is close to one, the weights present weird behaviors, which could explain the bad performance when transitioning from low-sample to high-sample regimes.
Furthermore, going beyond the sole selection of a few hyperparameters, it would be interesting to understand more aggressive model selection. In particular, given some observations  $(X_{i},Y_{i})$  and some hypothesis classes  $(\mathcal{F}_t)_t$ , it seems natural to trade a term that fits the data as per (7), together with a regularization term  $\min_t\| f\|_{\mathcal{F}_t}$  that selects  $\mathcal{F}_t$  so that  $f_{n}$  has a small  $\mathcal{F}_t$  norm. We understand this as a lex parsimoniae, where each  $\mathcal{F}_t$  encodes different notions of simplicity (e.g. different priors) while  $f_{n}$  only needs to satisfy one of them.
Finally, while this work heavily relies on the least-square loss, practitioners tend to favor other losses such as the cross-entropy. How losses deform and modify the size of the search space  $\mathcal{F}$  and its adherence properties to some target functions  $f^{*}$  is an open-question -not to mention its adherence properties when the final predictor is built as a decoding  $y(x) = \arg \max_{y} f(y|x)$  in order to learn a discrete  $y$  from a score  $f(y|x)$  that relates to  $\mathbb{P}_{(X,Y)}\sim \rho (Y = x\mid X = x)$ .
Experiments reproduction. All the code to run figures is available at https://github.com/facebookresearch/rates.
Acknowledgements. VC would like to thank Alberto Bietti, Jaouad Mourtada and Francis Bach for useful discussions. SV was partially supported by the MUR Excellence Department Project MatMod@TOV awarded to the Department of Mathematics, University of Rome Tor Vergata.
[1] Robert Adams and John Fournier. Sobolev Spaces. Academic Press, 1975.
[2] Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 1950.
[3] Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv preprint arXiv:2303.01372, 2023.
[28] Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under- to over-parametrization affects loss landscape and generalization. Journal of Physics A: Mathematical and Theoretical, 2019.
[29] Charles Stone. Consistent nonparametric regression. The Annals of Statistics, 1977.
[30] Hans Triebel. Interpolation Theory, Function Spaces, Differential Operators. North-Holland Publishing Co., 1978.
[31] Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
[32] Ding-Xuan Zhou. The covering number in learning theory. Journal of Complexity, 2002.
Figure 7: Two other cuts of Figure 1.
A Generic Proofs and Discussions
A.1 What do we mean by Transitory Regimes?
In essence, by transitory regimes we mean any finite-time behavior that does not match an expected long-time horizon "stationary" behavior. More precisely, let  $\Gamma = \{(n,\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)])|n\in \mathbb{N}\}$  be the graph of the expected excess risk. Theorem 4 provides a lower-upper bound of the form  $\Gamma \subset \{(n,cn^{-\gamma}(1 + ah(n)))|n\in \mathbb{N},a\in [-1,1]\}$  with  $c,\gamma$  two constants and  $h$  a function that goes to zero when its argument goes to infinity. This shows that, as  $n$  grows large,  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$  will behave as  $cn^{-\gamma}$ . However, this stationary behavior in  $cn^{-\gamma}$  might take time to kick in, and when only accessing a small number of samples  $n$ , our bound does not lead to strong constraints on  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$ , which might arguably exhibit a very different profile. We illustrate this idea on Figure 3.
A.2 Generic Lower Bounds
Let us recall a relatively standard lower bound for linear regression.
Theorem 5 (Linear regression minimax rates, [17]). Let  $\rho_{\theta} \in \Delta_{\mathcal{X} \times \mathcal{Y}}$  with  $\mathcal{X} = \mathbb{R}^d$  and  $\mathcal{Y} = \mathbb{R}$  be defined with its marginal over  $\mathcal{X}$  being  $\rho_{\mathcal{X}}$ , and its conditionals being  $(Y|X) = \theta^\top X + \varepsilon^2 \sim \mathcal{N}(0,1)$  with  $\mathcal{N}$  the Gaussian distribution. Let  $\mathcal{D}_n \sim \rho_\theta^{\otimes n}$  be a dataset of  $n$  samples  $(X_i, Y_i)$ , and  $\mathcal{A}$  be an algorithm that map  $\mathcal{D}_n$  to a guess for  $\theta$ . Under the assumption that  $n > d$  and that  $\sum X_i X_i^\top$  is almost always invertible, we have
$$
\sup _ {\theta} \mathbb {E} _ {\mathcal {D} _ {n} \sim \rho_ {\theta} ^ {\otimes n}} \left[ \| \theta - \mathcal {A} (\mathcal {D} _ {n}) \| ^ {2} \right] \geq \frac {\varepsilon^ {2} d}{n - d + 1}.
$$
Replacing  $X$  by  $\varphi (X)$  , we get the following corollary.
Corollary 6. For any learning algorithm  $\mathcal{A}$  targeting a function  $f^{*}(X) = \varphi (X)^{\top}\theta$  with  $\varphi :\mathcal{X}\to \mathbb{R}^D$  and  $\theta \in \mathbb{R}^D$ , there exists a  $\theta$  and a distribution  $\rho_{\mathcal{X}}$  on  $X$  such that under Assumption 2,
$$
\mathbb {E} _ {\mathcal {D} _ {n} \sim \rho} \left[ \| \theta - \mathcal {A} (\mathcal {D} _ {n}) \| ^ {2} \right] \geq \frac {\varepsilon^ {2} \dim \operatorname {S p a n} \varphi (\mathcal {X})}{n}.
$$
The proof of Theorem 2 (resp. Theorem 3) follows by considering  $\varphi$  to be the concatenation of all the  $\binom{d+\alpha}{\alpha}$  polynomials of degree at most  $\alpha$  with  $d$  variables (resp. all the trigonometric polynomials  $x \mapsto \exp(-im^{\top}x)$  with  $\|m\|_{\infty} \leq \omega$ ). To be totally rigorous, one should incorporate the assumptions of Theorem 5, yet if we remove the extra assumption, one can actually show that the worse excess risk could be infinite [17, Proposition 1], so this does not cast shadow on our results, but would only make them stronger. We choose to present weaker results that are easier to understand and parse for the reader.
Those theorems could be proven from scratch by considering a well-thought Bayesian prior on  $\theta$  that has generated the dataset  $\mathcal{D}_n$ , before lower bounding
$$
\begin{array}{l} \inf  _ {\mathcal {A}} \sup  _ {\theta} \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \| \mathcal {A} (\mathcal {D} _ {n}) - \theta \| ^ {2} | \theta \right] \geq \inf  _ {\mathcal {A}} \mathbb {E} _ {\theta} \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \| \mathcal {A} (\mathcal {D} _ {n}) - \theta \| ^ {2} | \theta \right] \\ = \inf  _ {\mathcal {A}} \mathbb {E} _ {(\theta , \mathcal {D} _ {n})} \left[ \| \mathcal {A} (\mathcal {D} _ {n}) - \theta \| ^ {2} \right] \\ = \mathbb {E} _ {(\theta , \mathcal {D} _ {n})} \left[ \| \mathbb {E} [ \theta | \mathcal {D} _ {n} ] - \theta \| ^ {2} \right], \\ \end{array}
$$
and computing the last term explicitly. We conjecture that it should be equally possible to compute with this technique a precise "exponentially bad" lower bound on the optimal minimax constants appearing in front of  $n^{-2\alpha/(2\alpha+d)}$  in Theorem 1.
It is equally possible to retake the proof techniques of Theorems 2 and 3 to prove lower bound that are targeted to match the upper bound in  $O(n^{-2\alpha/(2\alpha+d)})$ .
Theorem 7. For any RKHS  $\mathcal{F}$  and distribution  $\rho_{\mathcal{X}}$  such that the spectrum of the integral operator verifies  $\operatorname{spec} K = \left\{n^{-2\alpha /d} \mid n \in \mathbb{N}\right\}$ , for any algorithm  $\mathcal{A}$  and value  $D > 0$ , there exists a function satisfying  $\|f^{*}\|_{\mathcal{F}} \leq D$  such that under Assumption 2,
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} (\mathcal {A} (\mathcal {D} _ {n})) \right] \geq \frac {1}{6} \frac {\varepsilon^ {4 \alpha / (2 \alpha + d)} D ^ {2 d / (2 \alpha + d)}}{n ^ {2 \alpha / (2 \alpha + d)}}.
$$
Proof. The proof consists in finding a subproblem that reduces to linear regression. Under the assumptions of Theorem 5 and the additional assumption that  $\| \theta^{*}\|_{2}\leq D$ , it is possible to show the minimax lower bound
$$
\mathbb {E} _ {\theta} \mathbb {E} _ {\mathcal {D} _ {n} \sim \rho_ {\theta} ^ {\otimes n}} \left[ \| \theta - \mathcal {A} (\mathcal {D} _ {n}) \| ^ {2} \right] \geq \frac {1}{6} \min \left\{\frac {\varepsilon^ {2} d}{n}, D ^ {2} \right\},
$$
for some Bayesian prior on  $\theta$  [see 4, Chapter on "lower bound", section on Bayesian analysis].