SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {E} := \left\{(\alpha_ {\ell}) _ {\ell = 0} ^ {\infty}: \sum_ {\ell = 0} ^ {\infty} \frac {| \alpha_ {\ell} | ^ {2}}{\min  \left(1 , \lambda_ {\ell} ^ {- s}\right)} \leq 1 \right\} \subseteq \ell^ {2} (\mathbb {N}). \tag {132}
$$
Note that the eigenvalues are distributed according to the bound proved in Theorem 4.4. Consider  $M$  functions/sequences  $f_{1}, f_{2}, \ldots, f_{M} \in \mathcal{E}$  such that  $\|f_{i} - f_{j}\|_{\ell^{2}(\mathbb{N})} \geq \delta$  for all  $i \neq j$ , for some  $M$  and  $\delta$  (to be set later). In other words, let  $\{f_{1}, f_{2}, \ldots, f_{M}\}$  denote a  $\delta$ -packing of the set  $\mathcal{E}$  in the  $\ell^{2}(\mathbb{N})$ -norm.
Consider a pair of random variables  $(Z, J)$  as follows: first  $J \in [M] \coloneqq \{1, 2, \dots, M\}$  and  $x_i \in \mathcal{M}$ ,  $i = 1, 2, \dots, n$ , are chosen uniformly and independently at random, and then,  $Z = (f_J(x_i) + \epsilon_i)_{i=1}^n \in \mathbb{R}^n$ , where  $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$  are independent Gaussian variates. Let  $\mathbb{P}_j(.) = \mathbb{P}(.|J = j)$  denote the conditional law of  $(Z, J)$ , given the observation  $J = j$ . A straightforward computation shows that  $D_{\mathrm{KL}}(\mathbb{P}_i||\mathbb{P}_j) = \frac{n}{2\sigma^2} \| f_i - f_j \|_{\ell^2(\mathbb{N})}^2 \geq \frac{n\delta}{2\sigma^2}$  for all  $i, j \in [M]$ .
According to Fano's method, one can get the minimax bound
$$
\inf  _ {\hat {f}} \sup  _ { \begin{array}{l} f ^ {\star} \in \mathcal {H} _ {\text {i n v}} ^ {s} (\mathcal {M}) \\ \| f ^ {\star} \| _ {\mathcal {H} _ {\text {i n v}} ^ {s} (\mathcal {M})} = 1 \end{array} } \mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} (f ^ {\star}) \right] \geq \frac {1}{2} \delta^ {2}, \tag {133}
$$
if  $\log (M)\geq 2I(Z;J) + 2\log (2)$ . Using the Yang-Barron method [65], this condition is satisfied if
$$
\epsilon^ {2} \geq \log N _ {\mathrm {K L}} (\epsilon) \tag {134}
$$
$$
\log M \geq 4 \epsilon^ {2} + 2 \log (2), \tag {135}
$$
where  $N_{\mathrm{KL}}(\epsilon)$  denotes the  $\epsilon$ -covering number of the space of distributions  $\mathbb{P}(.|f)$  for some  $f \in \mathcal{E}$  (defined similarly as we have  $\mathbb{P}(.|J = j) = \mathbb{P}(.|f = f_j)$ ), in the square-root KL-divergence. However, since  $D_{\mathrm{KL}}(\mathbb{P}_f||\mathbb{P}_g) = \frac{n}{2\sigma^2}\|f - g\|_{\ell^2(\mathbb{N})}^2$ , this equals to the  $\epsilon$ -covering of the space  $\mathcal{E}$  in the  $\ell^2(\mathbb{N})$ -norm. In other words, we have
$$
N _ {\mathrm {K L}} (\epsilon) = N _ {\ell^ {2} (\mathbb {N})} \left(\frac {\epsilon \sigma \sqrt {2}}{\sqrt {n}}\right). \tag {136}
$$
Now note that, for any  $M$  such that  $\log M\geq \log N_{\ell^2 (\mathbb{N})}(\delta)$ , there exists a  $\delta$ -packing of the space  $\mathcal{E}$  in the  $\ell^2 (\mathbb{N})$ -norm (see the packing and covering numbers relationship [65]).
In summary, we get the minimax rate of  $\frac{1}{2}\delta^2$  if the following inequalities are satisfied:
$$
\epsilon^ {2} \geq \log N _ {\ell^ {2} (\mathbb {N})} \left(\frac {\epsilon \sigma \sqrt {2}}{\sqrt {n}}\right) \tag {137}
$$
$$
\log N _ {\ell^ {2} (\mathbb {N})} (\delta) \geq 4 \epsilon^ {2} + 2 \log (2), \tag {138}
$$
for some pair  $(\epsilon, \delta)$ . Thus, our goal is to obtain tight lower/upper bounds for  $\log N_{\ell^{2}(\mathbb{N})}(.)$ .
Lemma E.1. For any positive  $\zeta$
$$
\log N _ {\ell^ {2} (\mathbb {N})} (\zeta / 2) \geq N \left(\zeta^ {- \frac {2}{s}}; G\right) \log (2) \tag {139}
$$
$$
\log N _ {\ell^ {2} (\mathbb {N})} (\sqrt {2} \zeta) \leq N \left(\zeta^ {- \frac {2}{s}}; G\right) (s / d + \log (4)) + \mathcal {O} \left(\zeta^ {- \frac {d - 1}{s}}\right), \tag {140}
$$
where the quantity  $N(\zeta^{-\frac{2}{s}};G)$  is defined in Theorem 4.4.
First, let us show how the above lemma concludes the proof of Theorem 4.3. According to the lemma, we just need to check the following inequalities:
$$
\epsilon^ {2} \geq (s / d + \log (4)) \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \left(\frac {\epsilon \sigma}{\sqrt {n}}\right) ^ {- d / s} + \mathcal {O} \left(n ^ {- \frac {d - 1}{2 s}}\right) \tag {141}
$$
$$
N \left(\left(2 \delta\right) ^ {\frac {- 2}{s}}; G\right) \log (2) \geq 4 \epsilon^ {2} + 2 \log (2). \tag {142}
$$
Without loss of generality, let us discard the big-O error terms in the above analysis. In our final adjustment, we can add a constant multiplicative factor to ensure that the bound is asymptotically valid. To get the largest possible  $\delta$ , we set
$$
\epsilon^ {2} = (s / d + \log (4)) \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \left(\frac {\epsilon \sigma}{\sqrt {n}}\right) ^ {- d / s}, \tag {143}
$$
$$
\epsilon^ {2} = \left((s / d + \log (4)) \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {s / (s + d / 2)} \times \left(\frac {\sigma^ {2}}{n}\right) ^ {- d / (s + d / 2)}. \tag {144}
$$
Therefore, the following inequality needs to be satisfied:
$$
N \left((2 \delta) ^ {\frac {- 2}{s}}; G\right) \log (2) \geq \log (4) + 4 \left(\left(s / d + \log (4)\right) \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {s / (s + d / 2)} \left(\frac {\sigma^ {2}}{n}\right) ^ {- d / (s + d / 2)}. \tag {145}
$$
Using asymptotic analysis, the inequality holds when
$$
\frac {\log (2) \omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) (2 \delta) ^ {- d / s} \geq 4 \left(\left(s / d + \log (4)\right) \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {s / (s + d / 2)} \left(\frac {\sigma^ {2}}{n}\right) ^ {- d / (s + d / 2)}. \tag {146}
$$
Rearranging the terms shows that
$$
4 \delta^ {2} \leq \left(\frac {\omega_ {d}}{(2 \pi) ^ {d}} \frac {\sigma^ {2} \operatorname {v o l} (\mathcal {M} / G)}{n}\right) ^ {s / (s + d / 2)} \times \underbrace {\left(\frac {4}{\log (2) (s / d + 2 \log (2)) ^ {- s / (s + d / 2)}}\right) ^ {- 2 s / d}} _ {:= 8 C _ {\kappa}}, \tag {147}
$$
where  $C_{\kappa}$  only depends on  $\kappa = 2s / d - 1$ . Since this gives a minimax lower bound of  $\frac{1}{2}\delta^2$ , the proof is complete.
The rest of this section is devoted to the proof of Lemma E.1.
Proof of Lemma E.1. Define the following truncated ellipsoid:
$$
\tilde {\mathcal {E}} := \left\{(\alpha_ {\ell}) _ {\ell = 0} ^ {\infty} \in \mathcal {E}: \forall \ell \geq \Delta + 1: \alpha_ {\ell} = 0 \right\} \subseteq \mathcal {E}, \tag {148}
$$
where  $\Delta$  is a parameter defined as
$$
\Delta := \max  \left\{\ell : \lambda_ {\ell} \leq \zeta^ {\frac {- 2}{s}} \right\} = N \left(\zeta^ {\frac {- 2}{s}}; G\right). \tag {149}
$$
Note that  $\sum_{\ell = \Delta +1}^{\infty}|\alpha_{\ell}|^{2}\leq \zeta^{2}$  for all  $(\alpha_{\ell})_{\ell = 0}^{\infty}\in \mathcal{E}$
To construct a  $\zeta /2$ -covering set, note that according to the definition of the truncated ellipsoid,  $\mathcal{B}_{\Delta}(\zeta)\subseteq \tilde{\mathcal{E}}\subseteq \mathcal{E}$ , where  $\mathcal{B}_{\Delta}(\zeta)$  denotes the ball with radius  $\zeta$  (in  $\ell^2$ -norm) is the Euclidean space of dimension  $\Delta$ . Using standard bounds in the literature [65], we get
$$
\log N _ {\ell^ {2} (\mathbb {N})} (\zeta / 2) \geq \Delta \log (2). \tag {150}
$$
To get the upper bound on the  $\sqrt{2}\zeta$ -covering number, using an argument based on the volume of the ellipsoid  $\tilde{\mathcal{E}}$  [65], we conclude
$$
\begin{array}{l} \log N _ {\ell^ {2} (\mathbb {N})} (\sqrt {2} \zeta) \leq \Delta \log (4 / \zeta) + \frac {1}{2} \sum_ {\ell = 0} ^ {\Delta} \log \left(\lambda_ {\ell} ^ {- s}\right) (151) \\ = \Delta \log (4 / \zeta) - \frac {s}{2} \sum_ {\ell = 0} ^ {\Delta} \log \left(\lambda_ {\ell}\right) (152) \\ = \Delta \log (4 / \zeta) - \frac {s}{2} \int_ {1} ^ {\zeta^ {- \frac {2}{s}}} \log (t) d N (t; G) (153) \\ = \Delta \log (4 / \zeta) - \frac {s}{2} \log \left(\zeta^ {\frac {- 2}{s}}\right) N \left(\zeta^ {\frac {- 2}{s}}; G\right) + \frac {s}{2} \int_ {1} ^ {\zeta^ {\frac {- 2}{s}}} N (t; G) \frac {d t}{t} (154) \\ = \Delta \log (4) + \frac {s}{2} \int_ {1} ^ {\zeta^ {- \frac {2}{s}}} N (t; G) \frac {d t}{t}. (155) \\ \end{array}
$$
By Theorem 4.4,
$$
\begin{array}{l} \int_ {1} ^ {\zeta^ {- \frac {2}{s}}} N (t; G) \frac {d t}{t} \leq \frac {2}{d} N \left(\zeta^ {- \frac {2}{s}}; G\right) + \mathcal {O} \left(\zeta^ {- \frac {d - 1}{s}}\right) (156) \\ = \frac {2}{d} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \zeta^ {- d / s} + \mathcal {O} \left(\zeta^ {- \frac {d - 1}{s}}\right). (157) \\ \end{array}
$$
$$
\log N _ {\ell^ {2} (\mathbb {N})} (\sqrt {2} \zeta) \leq \Delta (s / d + \log (4)) + \mathcal {O} \left(\zeta^ {- \frac {d - 1}{s}}\right). \tag {158}
$$