SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} | \mathbf {E} (\mathcal {R} _ {S} (\boldsymbol {\theta})) - \mathcal {R} _ {D} (\boldsymbol {\theta}) | \\ = \sum_ {j = 1} ^ {d} \left(\mathbf {E} \frac {1}{M} \sum_ {i = 1} ^ {M} \left| \frac {\partial (f (\boldsymbol {x} _ {i}) - \phi (\boldsymbol {x} _ {i} ; \boldsymbol {\theta}))}{\partial x _ {j}} \right| ^ {2} - \int_ {(0, 1) ^ {d}} \left| \frac {\partial (f (\boldsymbol {x}) - \phi (\boldsymbol {x} ; \boldsymbol {\theta}))}{\partial x _ {j}} \right| ^ {2} d \boldsymbol {x}\right) \\ + \mathbf {E} \frac {1}{M} \sum_ {i = 1} ^ {M} | (f (\boldsymbol {x} _ {i}) - \phi (\boldsymbol {x} _ {i}; \boldsymbol {\theta})) | ^ {2} - \int_ {(0, 1) ^ {d}} | (f (\boldsymbol {x}) - \phi (\boldsymbol {x}; \boldsymbol {\theta})) | ^ {2} d \boldsymbol {x} \\ \leq (B + 1) \sum_ {j = 1} ^ {d} \left(\mathbf {E} \left| \frac {1}{M} \sum_ {i = 1} ^ {M} \frac {\partial (f (\boldsymbol {x} _ {i}) - \phi (\boldsymbol {x} _ {i} ; \boldsymbol {\theta}))}{\partial x _ {j}} - \int_ {(0, 1) ^ {d}} \frac {\partial (f (\boldsymbol {x}) - \phi (\boldsymbol {x} ; \boldsymbol {\theta}))}{\partial x _ {j}} d \boldsymbol {x} \right|\right) \\ + (B + 1) \mathbf {E} \left| \frac {1}{M} \sum_ {i = 1} ^ {M} \left(f (\boldsymbol {x} _ {i}) - \phi (\boldsymbol {x} _ {i}; \boldsymbol {\theta})\right) - \int_ {(0, 1) ^ {d}} \left(f (\boldsymbol {x}) - \phi (\boldsymbol {x}; \boldsymbol {\theta})\right) d \boldsymbol {x} \right| \\ \leq 2 (B + 1) \left(d \mathbf {R} _ {M} \left(D \tilde {\Phi}\right) + \mathbf {R} _ {M} (\tilde {\Phi})\right) \tag {120} \\ \end{array}
$$
where the last inequality is due to Lemma 12.
7.5.2 Bounding the Rademacher complexity and the proof of Theorem 5
In this subsection, we aim to estimate the Rademacher complexity using the covering number. We then estimate the covering number using the pseudo-dimension.
Definition 14 (covering number [3]). Let  $(V, \| \cdot \|)$  be a normed space, and  $\Theta \in V$ .  $\{V_1, V_2, \ldots, V_n\}$  is an  $\varepsilon$ -covering of  $\Theta$  if  $\Theta \subset \cup_{i=1}^{n} B_{\varepsilon, \| \cdot \|}(V_i)$ . The covering number  $\mathcal{N}(\varepsilon, \Theta, \| \cdot \|)$  is defined as
$$
\mathcal {N} (\varepsilon , \Theta , \| \cdot \|) := \min  \{n: \exists \varepsilon \text {- c o v e r i n g o v e r} \Theta \text {o f s i z e} n \}.
$$
Definition 15 (Uniform covering number [3]). Suppose the  $\mathcal{F}$  is a class of functions from  $\mathcal{F}$  to  $\mathbb{R}$ . Given  $n$  samples  $\mathbf{Z}_n = (z_1, \ldots, z_n) \in \mathcal{X}^n$ , define
$$
\mathcal {F} | _ {\mathbf {Z} _ {n}} = \left\{\left(u \left(z _ {1}\right), \dots , u \left(z _ {n}\right)\right): u \in \mathcal {F} \right\}.
$$
The uniform covering number  $\mathcal{N}(\varepsilon, \mathcal{F}, n)$  is defined as
$$
\mathcal {N} (\varepsilon , \mathcal {F}, n) = \max  _ {\mathbf {Z} _ {n} \in \mathcal {X} ^ {n}} \mathcal {N} \left(\varepsilon , \mathcal {F} | _ {\mathbf {Z} _ {n}}, \| \cdot \| _ {\infty}\right),
$$
where  $\mathcal{N}(\varepsilon, \mathcal{F}|_{\mathbf{Z}_n}, \| \cdot \|_{\infty})$  denotes the  $\varepsilon$ -covering number of  $\mathcal{F}|_{\mathbf{Z}_n}$  w.r.t the  $L_{\infty}$ -norm.
Then we use a lemma to estimate the Rademacher complexity using the covering number.
Lemma 13 (Dudley's theorem [3]). Let  $\mathcal{F}$  be a function class such that  $\sup_{f\in \mathcal{F}}\| f\|_{\infty}\leq B$ . Then the Rademacher complexity  $\mathbf{R}_n(\mathcal{F})$  satisfies that
$$
\mathbf {R} _ {n} (\mathcal {F}) \leq \inf  _ {0 \leq \delta \leq B} \left\{4 \delta + \frac {1 2}{\sqrt {n}} \int_ {\delta} ^ {B} \sqrt {\log 2 \mathcal {N} (\varepsilon , \mathcal {F} , n)} \mathrm {d} \varepsilon \right\}
$$
To bound the Rademacher complexity, we employ Lemma 13, which bounds it by the uniform covering number. We estimate the uniform covering number by the pseudo-dimension based on the following lemma.
Lemma 14 ([3]). Let  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $[-B, B]$ . For any  $\varepsilon > 0$ , we have
$$
\mathcal {N} (\varepsilon , \mathcal {F}, n) \leq \left(\frac {2 e n B}{\varepsilon P d i m (\mathcal {F})}\right) ^ {P d i m (\mathcal {F})}
$$
for  $n\geq Pdim(\mathcal{F})$
The remaining problem is to bound  $\mathrm{Pdim}(\widetilde{\Phi})$  and  $\mathrm{Pdim}(D\widetilde{\Phi})$ . Based on [4],  $\mathrm{Pdim}(\widetilde{\Phi}) = O(L^2 N^2 \log_2 L \log_2 N)$ . For the  $\mathrm{Pdim}(D\widetilde{\Phi})$ , we can estimate it by Theorem 2.
Now we can estimate generalization error based on Lemma 12.
Proof of Theorem 5. Let  $J = \max \{\mathrm{Pdim}(D\widetilde{\Phi}), \mathrm{Pdim}(\widetilde{\Phi})\}$ . Due to Lemma 13, 14 and Theorem 2, for any  $M \geq J$ , we have
$$
\begin{array}{l} \mathbf {R} _ {M} (D \widetilde {\Phi}) \leq 4 \delta + \frac {1 2}{\sqrt {M}} \int_ {\delta} ^ {B} \sqrt {\log 2 \mathcal {N} (\varepsilon , D \widetilde {\Phi} , M)} \mathrm {d} \varepsilon \\ \leq 4 \delta + \frac {1 2}{\sqrt {M}} \int_ {\delta} ^ {B} \sqrt {\log 2 \left(\frac {2 e M B}{\varepsilon \mathrm {P d i m} (D \widetilde {\Phi})}\right) ^ {\mathrm {P d i m} (D \widetilde {\Phi})}} \mathrm {d} \varepsilon \\ \leq 4 \delta + \frac {1 2 B}{\sqrt {M}} + 1 2 \left(\frac {\operatorname {P d i m} (D \widetilde {\Phi})}{M}\right) ^ {\frac {1}{2}} \int_ {\delta} ^ {B} \sqrt {\log \left(\frac {2 e M B}{\varepsilon \operatorname {P d i m} (D \widetilde {\Phi})}\right)} \mathrm {d} \varepsilon . \tag {121} \\ \end{array}
$$
By the direct calculation for the integral, we have
$$
\int_ {\delta} ^ {B} \sqrt {\log \left(\frac {2 e M B}{\varepsilon \mathrm {P d i m} (D \widetilde {\Phi})}\right)} \mathrm {d} \varepsilon \leq B \sqrt {\log \left(\frac {2 e M B}{\delta \mathrm {P d i m} (D \widetilde {\Phi})}\right)}.
$$
Then choosing  $\delta = B\left(\frac{\mathrm{Pdim}(D\widetilde{\Phi})}{M}\right)^{\frac{1}{2}} \leq B$ , we have
$$
\mathbf {R} _ {M} (D \widetilde {\Phi}) \leq 2 8 B \left(\frac {\operatorname {P d i m} (D \widetilde {\Phi})}{M}\right) ^ {\frac {1}{2}} \sqrt {\log \left(\frac {2 e M}{\operatorname {P d i m} (D \widetilde {\Phi})}\right)}. \tag {122}
$$
Therefore, due to Theorem 2, there is a constant  $C_4$  independent with  $L,N,M$  such as
$$
\mathbf {R} _ {M} \left(D \widetilde {\Phi}\right) \leq C _ {4} \frac {N L \left(\log_ {2} L \log_ {2} N\right) ^ {\frac {1}{2}}}{\sqrt {M}} \log M. \tag {123}
$$
$\mathbf{R}_M(\widetilde{\Phi})$  can be estimate in the similar way. Due to Lemma 12, we have that there is a constant  $C_5 = C_5(B,d,C_1,C_2)$  such that
$$
\mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {D}) - \mathcal {R} _ {D} (\boldsymbol {\theta} _ {D}) + \mathbf {E} \mathcal {R} _ {D} (\boldsymbol {\theta} _ {S}) - \mathbf {E} \mathcal {R} _ {S} (\boldsymbol {\theta} _ {S}) \leq C _ {5} \frac {N L \left(\log_ {2} L \log_ {2} N\right) ^ {\frac {1}{2}}}{\sqrt {M}} \log M. \tag {124}
$$