SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition C.1 (Covering number [38]). Let  $\mathcal{F}$  be a functional class and  $x_{1},\dots ,x_{n}$  be a set of  $n$  observations. An  $\epsilon$ -cover of  $\mathcal{F}$  with respect to  $P_{n}$  is a set of functions  $f_{1},\dots ,f_{N}$  such that for any  $f\in \mathcal{F}$ , there exists  $j\in [N]$  such that
$$
\| f - f _ {j} \| _ {P _ {n}} \leq \epsilon .
$$
The covering number of  $\mathcal{F}$  with respect to  $P_{n}$  at scale  $\epsilon$ , denoted as  $\mathcal{N}(\mathcal{F}, P_n, \epsilon)$ , is the smallest  $N$  such that there exists an  $\epsilon$ -cover of  $\mathcal{F}$  with respect to  $P_{n}$  with size  $N$ .
Definition C.2 (Packing number [56]). Let  $\mathcal{F}$  be a functional class and  $x_{1},\dots ,x_{n}$  be a set of  $n$  observations. An  $\epsilon$ -packing of  $\mathcal{F}$  with respect to  $P_{n}$  is a set of functions  $f_{1},\dots ,f_{N}$  such that for any  $i,j\in [N]$ ,
$$
\| f _ {i} - f _ {j} \| _ {P _ {n}} \geq \epsilon .
$$
The packing number of  $\mathcal{F}$  with respect to  $P_{n}$  at scale  $\epsilon$ , denoted as  $\mathcal{P}(\mathcal{F}, P_n, \epsilon)$ , is the largest  $N$  such that there exists an  $\epsilon$ -packing of  $\mathcal{F}$  with respect to  $P_{n}$  with size  $N$ .
The following well-known lemma establishes the equivalence between packing and covering numbers.
Lemma C.1 (Lemma 4.2.8 [56]). For  $\epsilon >0$
$$
\mathcal {P} (\mathcal {F}, P _ {n}, 2 \epsilon) \leq \mathcal {N} (\mathcal {F}, P _ {n}, \epsilon) \leq \mathcal {P} (\mathcal {F}, P _ {n}, \epsilon).
$$
Finally, we define the fat-shattering dimension of a function class.
Definition C.3 (Fat-shattering dimension [26]). Let  $\mathcal{F}$  be a functional class. Given a subset  $S = \{\pmb{x}_1, \dots, \pmb{x}_n\} \subseteq \mathbb{R}^p$ , the class  $\mathcal{F}$ $\epsilon$ -shatters  $S$ , if there exists  $c = (c_1, \dots, c_n) \in \mathbb{R}^n$ , such that for every  $\pmb{e} \in \{0, 1\}^n$ , there exists  $f \in \mathcal{F}$  such that
$$
f (\boldsymbol {x} _ {i}) \geq c _ {i} + \epsilon / 2 \text {f o r} e _ {i} = 1, \text {a n d} f (\boldsymbol {x} _ {i}) \leq c _ {i} - \epsilon / 2 \text {f o r} e _ {i} = 0.
$$
The fat-shattering dimension of  $\mathcal{F}$  at scale  $\epsilon$ , denoted as  $\mathrm{fat}_{\epsilon}(\mathcal{F})$  is the size of the largest subset  $S$  that is  $\epsilon$ -shattered by  $\mathcal{F}$ .
The following theorem relates the fat-shattering dimension to the covering number discussed above.
Theorem C.1. If there exists  $v > 0$  such that
$$
\operatorname {f a t} _ {x \epsilon} (\mathcal {F}) \leq \frac {v}{x} \forall x \geq 1,
$$
$$
\log \mathcal {N} (\mathcal {F}, P _ {n}, C \epsilon) \leq C v.
$$
Proof. The theorem is a direct result of Theorems 6.3 of [50] and Lemma C.1.  $\square$
C.2 Technical Lemmas
Let  $\mathcal{U},\mathcal{L}$  denote the set of internal nodes and leaves of a depth  $d$  tree, respectively. Let us define the class of soft trees with depth  $d$  and data dimension  $p$  as
$$
\mathcal {F} (d, p) = \left\{f (\boldsymbol {x}; \boldsymbol {W}, \boldsymbol {o}): B _ {p} ^ {2} \mapsto \mathbb {R}, \bar {u} (f (\boldsymbol {x}; \boldsymbol {W}, \boldsymbol {o})) \leq B \right\} \tag {C.2}
$$
where  $B_p^2$  is the Euclidean ball of dimension  $p$ ,  $\mathbf{W} \in \mathbb{R}^{|\mathcal{U}| \times p}$  and  $\mathbf{o} \in \mathbb{R}^{|\mathcal{L}|}$  and
$$
\bar {u} (f (\boldsymbol {x}; \boldsymbol {W}, \boldsymbol {o})) = \max  _ {k \in \mathcal {U}} \| \boldsymbol {w} _ {k} \| _ {2} \vee \max  _ {l \in \mathcal {L}} | o _ {l} |
$$
with  $\boldsymbol{w}_k \in \mathbb{R}^p$  being the vector of weights for node  $k$  from  $\boldsymbol{W}$ . Our first result is characterizing the fat-shattering dimension of  $\mathcal{F}(d,p)$ .
Lemma C.2. Let  $A = 2 \vee 2^{d + 2}B \vee B^2 dL2^{d + 3}$ . For  $\epsilon > 0$ ,
$$
\operatorname {f a t} _ {\epsilon} (\mathcal {F} (d, p)) \leq \left(\frac {A}{\epsilon}\right) ^ {p}. \tag {C.3}
$$
Proof. First, suppose  $\epsilon > 2B2^d$ . We show that  $\mathsf{fat}_{\epsilon}(\mathcal{F}(d,p)) = 0$ . To this end, note that for  $x$  and  $f \in \mathcal{F}(d,p)$ ,
$$
| f (\boldsymbol {x}) | = \left| \sum_ {l \in \mathcal {L}} P \left(\left\{\boldsymbol {x} \rightarrow l \right\}\right) o _ {l} \right| \leq \sum_ {l \in \mathcal {L}} \left| o _ {l} \right| \leq 2 ^ {d} B \tag {C.4}
$$
where  $P$  is defined in Appendix A. As a result, for  $f_{1}, f_{2} \in \mathcal{F}(d,p)$  and  $\pmb{x}$ ,
$$
\left| f _ {1} (\boldsymbol {x}) - f _ {2} (\boldsymbol {x}) \right| \leq 2 B 2 ^ {d} <   \epsilon \tag {C.5}
$$
showing no single point can be  $\epsilon$ -shattered. Therefore,  $\operatorname{fat}_{\epsilon}(\mathcal{F}(d,p)) = 0$ .
Let  $0 < \epsilon \leq 2B2^{d}$  and suppose  $\{\pmb{x}_1,\dots ,\pmb{x}_N\}$  is  $\epsilon$ -shattered by  $\mathcal{F}(d,p)$  at level  $\pmb{c}$ . We show that for  $i,j\in [N]$ ,  $\| \pmb{x}_i - \pmb{x}_j\| _2\geq \epsilon /(B^2 dL2^d)$ . To this end, assume there exists  $i,j\in [N]$  such that  $\| \pmb{x}_i - \pmb{x}_j\| _2 < \epsilon /(B^2 dL2^d)$ . Without loss of generality assume  $c_{i}\geq c_{j}$  and choose  $\pmb {e}\in \{0,1\} ^p$  such that  $e_i = 1,e_j = 0$ . Suppose  $f(\cdot ;\pmb {W},\pmb {o})\in \mathcal{F}(d,p)$  shatters the data with  $\pmb{e}$ . Particularly,
$$
f \left(\boldsymbol {x} _ {i}; \boldsymbol {W}, \boldsymbol {o}\right) \geq c _ {i} + \epsilon / 2, f \left(\boldsymbol {x} _ {j}; \boldsymbol {W}, \boldsymbol {o}\right) \leq c _ {j} - \epsilon / 2. \tag {C.6}
$$
For  $k\in \mathcal{U},l\in \mathcal{L}$  , if  $k\in \mathcal{A}(l)$  we have
$$
\begin{array}{l} \left| r _ {k, l} \left(\boldsymbol {x} _ {i}\right) - r _ {k, l} \left(\boldsymbol {x} _ {j}\right) \right| = \left| \phi \left(\boldsymbol {w} _ {k} ^ {T} \boldsymbol {x} _ {i}\right) - \phi \left(\boldsymbol {w} _ {k} ^ {T} \boldsymbol {x} _ {i}\right) \right| \\ \stackrel {(a)} {\leq} L \left| \boldsymbol {w} _ {k} ^ {T} \boldsymbol {x} _ {i} - \boldsymbol {w} _ {k} ^ {T} \boldsymbol {x} _ {j} \right| \\ \stackrel {(b)} {\leq} L B \left\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right\| _ {2} \tag {C.7} \\ \end{array}
$$
where  $r_{k,l}$  is defined in Appendix A,  $\pmb{w}_k\in \mathbb{R}^p$  is the vector of weights for node  $k$  from  $\pmb{W}$ ,  $(a)$  is by (A1) and  $(b)$  is by  $\bar{u} (f(\pmb {x};\pmb {W},\pmb {o}))\leq B$  (recall the Assumption (A2)). Next,
$$
\begin{array}{l} | P (\{\boldsymbol {x} _ {i} \rightarrow l \}) - P (\{\boldsymbol {x} _ {j} \rightarrow l \}) | = \left| \prod_ {k \in \mathcal {A} (l)} r _ {k, l} (\boldsymbol {x} _ {i}) - \prod_ {k \in \mathcal {A} (l)} r _ {k, l} (\boldsymbol {x} _ {j}) \right| \\ \leq d L B \| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \| _ {2} \tag {C.8} \\ \end{array}
$$
where the inequality is by (C.7) and the fact that  $r_{k,l} \in [0,1]$  is bounded. Finally,
$$
\begin{array}{l} | f (\boldsymbol {x} _ {i}; \boldsymbol {W}, \boldsymbol {o}) - f (\boldsymbol {x} _ {j}; \boldsymbol {W}, \boldsymbol {o}) | = \left| \sum_ {l \in \mathcal {L}} P (\{\boldsymbol {x} _ {i} \rightarrow l \}) o _ {l} - \sum_ {l \in \mathcal {L}} P (\{\boldsymbol {x} _ {j} \rightarrow l \}) o _ {l} \right| \\ \leq \sum_ {l \in \mathcal {L}} | o _ {l} | | P (\{\boldsymbol {x} _ {i} \rightarrow l \}) - P (\{\boldsymbol {x} _ {j} \rightarrow l \}) | \\ \leq 2 ^ {d} B ^ {2} d L \left\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right\| _ {2} \tag {C.9} \\ \end{array}
$$
where the last inequality is true as  $|\mathcal{L}| = 2^d$  and  $|o_l| \leq B$  by  $\bar{u}(f(\pmb{x}; \pmb{W}, \pmb{o})) \leq B$ . As we assumed  $\| \pmb{x}_i - \pmb{x}_j \|_2 < \epsilon / (B^2 dL^{2^d})$ , we have from (C.9)
$$
| f (\boldsymbol {x} _ {i}; \boldsymbol {W}, \boldsymbol {o}) - f (\boldsymbol {x} _ {j}; \boldsymbol {W}, \boldsymbol {o}) | <   \epsilon
$$
while on the other hand, by the shattering property (C.6),
$$
f \left(\boldsymbol {x} _ {i}; \boldsymbol {W}, \boldsymbol {o}\right) - f \left(\boldsymbol {x} _ {j}; \boldsymbol {W}, \boldsymbol {o}\right) \geq \underbrace {\left(c _ {i} - c _ {j}\right)} _ {\geq 0} + 2 \epsilon / 2 \geq \epsilon \tag {C.10}
$$
which is a contradiction. Therefore, for every  $i,j\in [N]$ , we must have  $\| \pmb {x}_i - \pmb {x}_j\| _2\geq \epsilon /(B^2 dL2^d)$  or in other words,  $\{\pmb {x_1},\dots ,\pmb {x}_N\}$  is a  $\epsilon /(B^{2}dL^{2})$  -packing of  $B_p^2$ . Thus,
$$
\begin{array}{l} N \stackrel {(a)} {\leq} \mathcal {N} \left(B _ {p} ^ {2}, \| \cdot \| _ {2}, \epsilon / \left(2 B ^ {2} d L 2 ^ {d}\right)\right) \\ \stackrel {(b)} {\leq} \left(\frac {4 B ^ {2} d L 2 ^ {d}}{\epsilon} + 1\right) ^ {p} \\ \leq \left(\frac {4 B ^ {2} d L 2 ^ {d}}{\epsilon} + 1 \vee 2 B 2 ^ {d}\right) ^ {p} \\ \stackrel {(c)} {\leq} \left(\frac {4 B ^ {2} d L 2 ^ {d}}{\epsilon} + \frac {1 \vee 2 B 2 ^ {d}}{\epsilon}\right) ^ {p} \leq \left(\frac {A}{\epsilon}\right) ^ {p} \\ \end{array}
$$
where  $\mathcal{N}(B_p^2,\| \cdot \| _2,\epsilon)$  is the  $\epsilon$ -covering number of  $B_{p}^{2}$ ,  $(a)$  is by Lemma C.1,  $(b)$  is true by Corollary 4.2.13 of [56] and  $(c)$  is true as  $\epsilon \leq 2^{d + 1}B$ .
Lemma C.3. There exists an absolute constant  $C > 0$  such that for  $\epsilon > 0$ ,
$$
\log \mathcal {N} _ {p} (\mathcal {F} (d, p), P _ {n}, \epsilon) \leq C \left(\frac {C A}{\epsilon}\right) ^ {p}.
$$
Proof. From Lemma C.2, for  $x \geq 1$
$$
\begin{array}{l} \operatorname {f a t} _ {x \epsilon} (\mathcal {F} (d, p)) \leq \left(\frac {A}{x \epsilon}\right) ^ {p} \\ = \left(\frac {A}{\epsilon}\right) ^ {p} \frac {1}{x ^ {p}} \\ \leq \left(\frac {A}{\epsilon}\right) ^ {p} \frac {1}{x} \tag {C.11} \\ \end{array}
$$
where the last inequality is true as  $p \geq 1$ . The proof is complete by Theorem C.1.
In the next step, we consider trees that only use a subset of variables, such as  $\mathcal{I} \subseteq [p]$ . Formally, we let
$$
\mathcal {F} _ {0} (\mathcal {J}, d) = \left\{f \left(\left\{x _ {j} \right\} _ {j \in \mathcal {J}}; \boldsymbol {W}, \boldsymbol {o}\right): B _ {p} ^ {2} \mapsto \mathbb {R},, \bar {u} \left(f \left(\left\{x _ {j} \right\} _ {j \in \mathcal {J}}; \boldsymbol {W}, \boldsymbol {o}\right)\right) \leq B \right\}. \tag {C.12}
$$
Lemma C.4. For  $\epsilon > 0$ ,
$$
\log \mathcal {N} (\mathcal {F} _ {0} (\mathcal {I}, d), P _ {n}, \epsilon) \leq C \left(\frac {C A}{\epsilon}\right) ^ {| \mathcal {I} |}.
$$
Proof. Let  $P_{n,\mathcal{J}}$  be the empirical distribution supported on coordinates in  $\mathcal{J}$
$$
P _ {n, \mathcal {I}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \delta_ {\boldsymbol {x} _ {\mathcal {I}}}
$$
where  $\pmb{x}_{\mathcal{I}}$  is the sub-vector of  $\pmb{x}$  with coordinates in  $\mathcal{I}$ . Then,
$$
\mathcal {N} \left(\mathcal {F} _ {0} (\mathcal {J}, d), P _ {n}, \epsilon\right) \leq \mathcal {N} \left(\mathcal {F} (d, | \mathcal {J} |), P _ {n, \mathcal {J}}, \epsilon\right)
$$
The result follows from Lemma C.3.
Next, let us define the classes of functions with interactions in  $\mathbb{J} \subseteq 2^{[p]}$ ,
$$
\mathcal {F} _ {1} (d, \mathbb {J}) = \left\{\sum_ {\mathcal {J} \in \mathbb {J}} f _ {\mathcal {J}} (\boldsymbol {x}; \boldsymbol {W} _ {\mathcal {J}}, \boldsymbol {o} _ {\mathcal {J}}): B _ {p} ^ {2} \mapsto \mathbb {R}, f _ {\mathcal {J}} (\cdot ; \boldsymbol {W} _ {\mathcal {J}}, \boldsymbol {o} _ {\mathcal {J}}) \in \mathcal {F} _ {0} (\mathcal {J}, d) \right\} \tag {C.13}
$$
and the class of functions with at most  $s$  interactions of size at most  $t$  (i.e.  $t$ -dimensional interactions),
$$
\mathcal {F} ^ {*} (d, t, s) = \left\{f _ {1} + f _ {2}: \mathbb {J} \subseteq 2 ^ {[ p ]}, | \mathbb {J} | \leq s, | \mathcal {J} | \leq t \forall \mathcal {J} \in \mathbb {J}, f _ {1} \in \mathcal {F} _ {1} (d, \mathbb {J}), f _ {2} \in \mathcal {F} _ {1} (d, \mathbb {J} ^ {*}) \right\} \tag {C.14}
$$
for a fixed  $\mathbb{J}^*$  with  $|\mathbb{J}^*| \leq s$  and  $|\mathcal{I}| \leq t$  for  $\mathcal{I} \in \mathbb{J}^*$ .
Lemma C.5. Suppose  $0 < \epsilon \leq 2sCA$  and  $1\leq t\leq 2$ . Then,
$$
\log \mathcal {N} (\mathcal {F} ^ {*} (d, t, s), P _ {n}, \epsilon) \leq 2 s \log p + 2 C s \left(\frac {2 C s A}{\epsilon}\right) ^ {t}.
$$
Proof. Let  $S(\mathcal{F}_0(\mathcal{J},d))$  be a minimal  $\epsilon /2s$ -cover of  $\mathcal{F}_0(\mathcal{J},d)$ . Hence,  $|\mathcal{S}(\mathcal{F}_0(\mathcal{J},d))| = \mathcal{N}(\mathcal{F}_0(\mathcal{J},d),P_n,\epsilon /2s)$ . Define
$$
\mathbb {S} = \left\{\mathbb {J} \subseteq 2 ^ {[ p ]}, | \mathcal {J} | \leq t \forall \mathcal {J} \in \mathbb {J}, | \mathbb {J} | \leq s \right\} \tag {C.15}
$$
$$
\mathcal {S} ^ {*} = \bigcup_ {\mathbb {J} \in \mathbb {S}} \left(\sum_ {\mathcal {J} \in \mathbb {J}} \mathcal {S} \left(\mathcal {F} _ {0} (\mathcal {J}, d)\right) + \sum_ {\mathcal {J} \in \mathbb {J} ^ {*}} \mathcal {S} \left(\mathcal {F} _ {0} (\mathcal {J}, d)\right)\right) \tag {C.16}
$$
where  $A + B = \{a + b : a \in A, b \in B\}$  denotes the Minkowski sum. Note that from Lemma C.4, for  $\mathbb{J} \in \mathbb{S}$  and  $\mathcal{J} \in \mathbb{J}$ ,
$$
\mathcal {N} \left(\mathcal {F} _ {0} (\mathcal {J}, d), P _ {n}, \epsilon / 2 s\right) \leq \exp \left(C \left(\frac {2 s C A}{\epsilon}\right) ^ {| \mathcal {I} |}\right) \leq \exp \left(C \left(\frac {2 s C A}{\epsilon}\right) ^ {t}\right) \tag {C.17}
$$
as  $2sCA / \epsilon \geq 1$ . For the rest of the proof, we let
$$
N _ {t} (\epsilon) = \exp \left(C \left(\frac {C A}{\epsilon}\right) ^ {t}\right).
$$
Suppose  $f\in \mathcal{F}^{*}(d,t,s)$
$$
f (\boldsymbol {x}) = \sum_ {\mathcal {I} \in \mathbb {J} _ {0}} f _ {\mathcal {I}} (\boldsymbol {x}; \boldsymbol {W} _ {\mathcal {I}}, \boldsymbol {o} _ {\mathcal {I}}) + \sum_ {\mathcal {I} \in \mathbb {J} ^ {*}} f _ {\mathcal {I}} (\boldsymbol {x}; \boldsymbol {W} _ {\mathcal {I}}, \boldsymbol {o} _ {\mathcal {I}}) \tag {C.18}
$$
for some  $\mathbb{J}_0\in \mathbb{S}$ . For any  $\mathcal{I}\in \mathbb{J}_0\cup \mathbb{J}^*$ , let  $\tilde{f}_{\mathcal{I}}\in S(\mathcal{F}_0(\mathcal{I},d))$  be such that  $\| \tilde{f}_{\mathcal{I}} - f_{\mathcal{I}}\|_{P_n}\leq \epsilon /2s$ . Let
$$
\tilde {f} (\boldsymbol {x}) = \sum_ {\mathcal {J} \in \mathbb {J} _ {0}} \tilde {f} _ {\mathcal {J}} (\boldsymbol {x}; \tilde {\boldsymbol {W}} _ {\mathcal {J}}, \tilde {\boldsymbol {o}} _ {\mathcal {J}}) + \sum_ {\mathcal {J} \in \mathbb {J} ^ {*}} \tilde {f} _ {\mathcal {J}} (\boldsymbol {x}; \tilde {\boldsymbol {W}} _ {\mathcal {J}}, \tilde {\boldsymbol {o}} _ {\mathcal {J}}). \tag {C.19}
$$
Note that  $\tilde{f}\in S^{*}$  . Next,
$$
\begin{array}{l} \| f - \tilde {f} \| _ {P _ {n}} = \left\| \sum_ {\mathcal {J} \in \mathbb {J} _ {0}} (f _ {\mathcal {J}} - \tilde {f} _ {\mathcal {J}}) + \sum_ {\mathcal {J} \in \mathbb {J} ^ {*}} (f _ {\mathcal {J}} - \tilde {f} _ {\mathcal {J}}) \right\| _ {P _ {n}} \\ \leq \sum_ {\mathcal {J} \in \mathbb {J} _ {0}} \| f _ {\mathcal {J}} - \tilde {f} _ {\mathcal {J}} \| _ {P _ {n}} + \sum_ {\mathcal {J} \in \mathbb {J} ^ {*}} \| f _ {\mathcal {J}} - \tilde {f} _ {\mathcal {J}} \| _ {P _ {n}} \\ \leq 2 s \epsilon / 2 s = \epsilon \tag {C.20} \\ \end{array}
$$
as  $|\mathbb{J}_0|, |\mathbb{J}^*| \leq s$ . As a result,  $\mathcal{S}^*$  is a  $\epsilon$ -cover for  $\mathcal{F}^*(d, t, s)$ . Moreover,
$$
\begin{array}{l} \mathcal {N} \left(\mathcal {F} ^ {*} (d, t, s), P _ {n}, \epsilon\right) \leq | \mathcal {S} ^ {*} | \\ \leq \sum_ {\mathbb {J} \in \mathbb {S}} \prod_ {\mathcal {I} \in \mathbb {J}} | \mathcal {S} (\mathcal {F} _ {0} (\mathcal {I}, d)) | \prod_ {\mathcal {I} \in \mathbb {J} ^ {*}} | \mathcal {S} (\mathcal {F} _ {0} (\mathcal {I}, d)) | \\ \leq \sum_ {\mathbb {J} \in \mathbb {S}} \prod_ {\mathcal {J} \in \mathbb {J}} \mathcal {N} \left(\mathcal {F} _ {0} (\mathcal {J}, d), P _ {n}, \epsilon / 2 s\right) \prod_ {\mathcal {J} \in \mathbb {J} ^ {*}} \mathcal {N} \left(\mathcal {F} _ {0} (\mathcal {J}, d), P _ {n}, \epsilon / 2 s\right) \\ \stackrel {(a)} {\leq} \sum_ {\mathbb {J} \in \mathbb {S}} N _ {t} (\epsilon / 2 s) ^ {2 s} \\ \stackrel {(b)} {\leq} p ^ {2 s} N _ {t} \left(\epsilon / 2 s\right) ^ {2 s} \tag {C.21} \\ \end{array}
$$
where  $(a)$  is by (C.17) and  $(b)$  is true as  $|\mathbb{S}| \leq (p^t)^s \leq p^{2s}$ . As a result,
$$
\begin{array}{l} \log \mathcal {N} \left(\mathcal {F} ^ {*} (d, t, s), P _ {n}, \epsilon\right) \leq 2 s \log p + 2 s \log N _ {t} (\epsilon / 2 s) \\ \leq 2 s \log p + 2 C s \left(\frac {2 C s A}{\epsilon}\right) ^ {t}. \tag {C.22} \\ \end{array}
$$
Lemma C.6. Suppose  $0 < \delta \leq 1 \land 2sCA$ . Then, for  $a = \exp(-1)$
$$
\int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} \left(\mathcal {F} ^ {*} (d , 1 , s) , P _ {n} , \epsilon\right)} d \epsilon \leq (\sqrt {2 s \log p} + C s A) \sqrt {\delta} \tag {C.23}
$$
$$
\int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) , P _ {n} , \epsilon)} d \epsilon \leq \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma / \delta) ^ {a}.
$$
Proof. Note that if  $\delta > 4\sigma$ , the lemma is trivial. Therefore, we assume  $\delta \leq 4\sigma$ . From Lemma C.5
$$
\begin{array}{l} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , t , s) , P _ {n} , \epsilon)} \leq \sqrt {2 s \log p + 2 C s \left(\frac {2 C s A}{\epsilon}\right) ^ {t}} \\ \leq \sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A) ^ {t / 2} \epsilon^ {- t / 2}. \tag {C.24} \\ \end{array}
$$
Hence, as  $\delta \leq 1$
$$
\begin{array}{l} \int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} \left(\mathcal {F} ^ {*} (d , 1 , s) , P _ {n} , \epsilon\right)} d \epsilon \leq \int_ {0} ^ {\delta} \sqrt {\log \mathcal {N} \left(\mathcal {F} ^ {*} (d , 1 , s) , P _ {n} , \epsilon\right)} d \epsilon \\ \leq \int_ {0} ^ {\delta} \sqrt {2 s \log p} d \epsilon + \int_ {0} ^ {\delta} 2 C s A \epsilon^ {- 1 / 2} d \epsilon \\ \leq \sqrt {2 s \log p} \delta + C s A \delta^ {1 / 2} \\ \leq \sqrt {2 s \log p} \delta^ {1 / 2} + C s A \delta^ {1 / 2} \tag {C.25} \\ \end{array}
$$
$$
\begin{array}{l} \int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) , P _ {n} , \epsilon)} d \epsilon \leq \int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) , P _ {n} , \epsilon)} d \epsilon \\ \leq \int_ {0} ^ {\delta} \sqrt {2 s \log p} d \epsilon + \int_ {\delta^ {2} / 4 \sigma} ^ {\delta} 2 C s A \sqrt {2 C s} \epsilon^ {- 1} d \epsilon \\ \leq \sqrt {2 s \log p} + 2 \sqrt {2 C s} C s A \log (4 \sigma / \delta). \tag {C.26} \\ \end{array}
$$
Next, consider the function  $h(x) = x^a - \log(x)$  where  $a = \exp(-1)$ . We will show  $h(x) \geq 0$  for  $x > 0$ . Note that
$$
h ^ {\prime} (x) = a x ^ {a - 1} - \frac {1}{x}
$$
therefore  $h(a^{-1 / a}) = h'(a^{-1 / a}) = 0$ . Moreover,
$$
h ^ {\prime \prime} (x) = \frac {a (a - 1) x ^ {a} + 1}{x ^ {2}}
$$
so  $h''(a^{-1/a}) = (1 + a(a - 1)a^{-1}) / (a^{-2/a}) > 0$ , showing that  $a^{-1/a}$  is a minimum for  $h(x)$ , and therefore  $h(x) \geq 0$  for  $x > 0$ . Hence, from (C.26),
$$
\begin{array}{l} \int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) , P _ {n} , \epsilon)} d \epsilon \leq \sqrt {2 s \log p} + \sqrt {2 C s} 2 C s A \log (4 \sigma / \delta) \\ \leq \sqrt {2 s \log p} + \sqrt {2 C s} 2 C s A (4 \sigma / \delta) ^ {a} \\ \leq \sqrt {2 s \log p} (4 \sigma / \delta) ^ {a} + \sqrt {2 C s} 2 C s A (4 \sigma / \delta) ^ {a} \tag {C.27} \\ \end{array}
$$
as  $\delta \leq 4\sigma$  or  $4\sigma /\delta \geq 1$
Lemma C.7. Under the assumptions of Theorem 2, for any sequence  $a_{n}, n \geq 1$  such that  $a_{n} > 0$  and  $\lim_{n \to \infty} a_{n} = 0$ , there exists a positive sequence  $b_{n}$  such that  $\lim_{n \to \infty} b_{n} = 0$  and
$$
\int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) , P _ {n} , \epsilon)} d \epsilon \leq \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma / \delta) ^ {a _ {n}} \forall \delta \leq b _ {n}.
$$
Proof. Consider the function  $h_n(x) = x^{a_n} - \log (x)$ . Note that
$$
\lim  _ {x \rightarrow \infty} \frac {x ^ {a _ {n}}}{\log x} = \lim  _ {x \rightarrow \infty} a _ {n} x ^ {a _ {n}} = \infty
$$
therefore, there exists a sequence  $\gamma_{n}$  such that  $\lim_{n\to \infty}\gamma_n = \infty$  and
$$
h _ {n} (x) \geq 0, \forall x \geq \gamma_ {n}.
$$
Therefore, if  $\delta \leq 4\sigma /\gamma_n\wedge 4\sigma \wedge 1 =: b_n$  , from (C.26),
$$
\begin{array}{l} \int_ {\delta^ {2} / 4 \sigma} ^ {\delta} \sqrt {\log \mathcal {N} (\mathcal {F} ^ {*} (d , 2 , s) , P _ {n} , \epsilon)} d \epsilon \leq \sqrt {2 s \log p} + 2 \sqrt {2 C s} C s A \log (4 \sigma / \delta) \\ \leq \sqrt {2 s \log p} + 2 \sqrt {2 C s} C s A (4 \sigma / \delta) ^ {a _ {n}} \\ \leq \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) \left(4 \sigma / \delta\right) ^ {a _ {n}}. \tag {C.28} \\ \end{array}
$$
Lemma C.8. For any  $\epsilon, \delta > 0$ ,
$$
\mathcal {N} \left(\mathcal {F} ^ {*} (d, t, s) \cap \left\{\| f \| _ {P _ {n}} \leq \delta \right\}, P _ {n}, \epsilon\right) \leq \mathcal {N} \left(\mathcal {F} ^ {*} (d, t, s), P _ {n}, \epsilon\right). \tag {C.29}
$$
Proof. Let  $\mathcal{S} = \{f_1, \dots, f_N\}$  be a minimal  $\epsilon$ -cover of  $\mathcal{F}^*(d, t, s)$  where  $N = \mathcal{N}(\mathcal{F}^*(d, t, s), P_n, \epsilon)$ . Let  $\mathcal{S}^* = \{f_1^*, \dots, f_N^*\}$  where for  $i \in [N]$ ,
$$
f _ {i} ^ {*} = \left\{ \begin{array}{l l} f _ {i} & \text {i f} \| f _ {i} \| _ {P _ {n}} \leq \delta \\ f _ {i} \delta / \| f _ {i} \| _ {P _ {n}} & \text {i f} \| f _ {i} \| _ {P _ {n}} > \delta \end{array} \right. \in \mathcal {F} ^ {*} (d, t, s) \cap \{\| f \| _ {P _ {n}} \leq \delta \}.
$$
We show  $S^{*}$  is a  $\epsilon$ -cover for  $\mathcal{F}^{*}(d, t, s) \cap \{\|f\|_{P_{n}} \leq \delta\}$ .
Let  $f \in \mathcal{F}^*(d, t, s) \cap \{\|f\|_{P_n} \leq \delta\}$ . Let  $j \in [N]$  be such that  $\|f - f_j\|_{P_n} \leq \epsilon$ . Let us consider the following cases:
Case 1: If  $\| f_j \|_{P_n} \leq \delta$ , then  $f_j \in S^*$  showing  $f_j^* = f_j$  covers  $f$ .
Case 2: Suppose  $\| f_{j} \|_{P_{n}} > \delta$ . For any  $f$ , let  $\mathbf{f} = (f(\mathbf{x}_{1}), \dots, f(\mathbf{x}_{n})) / \sqrt{n} \in \mathbb{R}^{n}$ . With this notation,  $\| f \|_{P_{n}} = \| \mathbf{f} \|_{2}$ . As a result,
$$
\begin{array}{l} \epsilon^ {2} \geq \| f - f _ {j} \| _ {P _ {n}} ^ {2} \\ = \left\| \left(f - f _ {j} ^ {*}\right) + \left(f _ {j} ^ {*} - f _ {j}\right) \right\| _ {P _ {n}} ^ {2} \\ = \left\| \left(f - f _ {j} ^ {*}\right) + \left(\frac {\delta}{\| f _ {j} \| _ {P _ {n}}} - 1\right) f _ {j} \right\| _ {P _ {n}} ^ {2} \\ = \left\| \left(\mathbf {f} - \mathbf {f} _ {j} ^ {*}\right) + \left(\frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} - 1\right) \mathbf {f} _ {j} \right\| _ {2} ^ {2} \\ \geq \| \mathbf {f} - \mathbf {f} _ {j} ^ {*} \| _ {2} ^ {2} + 2 \left(\frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} - 1\right) \mathbf {f} _ {j} ^ {T} (\mathbf {f} - \mathbf {f} _ {j} ^ {*}) \\ = \| \mathbf {f} - \mathbf {f} _ {j} ^ {*} \| _ {2} ^ {2} + 2 \left(\frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} - 1\right) \mathbf {f} _ {j} ^ {T} \mathbf {f} - 2 \left(\frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} - 1\right) \frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} \| \mathbf {f} _ {j} \| _ {2} ^ {2} \\ \stackrel {(a)} {\geq} \| \mathbf {f} - \mathbf {f} _ {j} ^ {*} \| _ {2} ^ {2} + 2 \left(\frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} - 1\right) \| \mathbf {f} _ {j} \| _ {2} \delta - 2 \left(\frac {\delta}{\| \mathbf {f} _ {j} \| _ {2}} - 1\right) \delta \| \mathbf {f} _ {j} \| _ {2} \\ = \left\| \mathbf {f} - \mathbf {f} _ {j} ^ {*} \right\| _ {2} ^ {2} = \left\| f - f _ {j} ^ {*} \right\| _ {P _ {n}} ^ {2} \tag {C.30} \\ \end{array}
$$
where  $(a)$  is true as  $\| \mathbf{f}_j\| _2 > \delta$  and  $\| \mathbf{f}\| _2\leq \delta$ . This shows  $f_{j}^{*}$  covers  $f$ , completing the proof.
$$
\delta_ {1} = \frac {\left[ 6 4 \sigma \left(\sqrt {2 s \log p} + C s A\right) \right] ^ {2 / 3}}{n ^ {1 / 3}} \tag {C.31}
$$
$$
\delta_ {2} = \frac {\left[ 1 6 \left(\sqrt {2 s \log p} + \sqrt {2 C s} (2 C s A)\right) (4 \sigma) ^ {1 + a} \right] ^ {1 / (2 + a)}}{n ^ {1 / 2 (2 + a)}}.
$$
Claim: For  $t\in \{1,2\}$