SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} (I): \Pi \left\{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \| \mathbf {B} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \| _ {1} \leq \Delta_ {n} / 6 \right\}; \\ (\mathrm {I I}): \Pi \left\{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \| \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) \| _ {1} \leq \Delta_ {n} / 6 \right\}; \\ \left. \Pi \left\{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \| \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}) \| _ {1} \leq \Delta_ {n} / 6 \right\}; \right. \tag {23} \\ (I) ^ {\prime}: \Pi \{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \| \mathbf {A} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) \| _ {1} \leq \Delta_ {n} / 6 \}; \\ (\mathrm {I I}) ^ {\prime}: \Pi \left\{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \| \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1) *}) \| _ {1} \leq \Delta_ {n} / 6 \right\}; \\ \left. \left(\Pi \Pi^ {\prime}\right): \Pi \left\{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \left\| \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1) *}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}) \right\| _ {1} \leq \Delta_ {n} / 6 \right\}. \right. \\ \end{array}
$$
Corresponding to  $\mathbf{B}^{(k)}$  for (6), we write  $\mathbf{B} = (\beta_{1},\dots,\beta_{K})$  and  $\mathbf{B}_0 = (\beta_{10},\dots,\beta_{K0})$ . Then for (I), we have that
$$
\begin{array}{l} \Pi \Bigl \{\pmb {\theta}: \mathbb {E} _ {\mathbf {x}} \| \mathbf {B} ^ {\top} \psi_ {2} (\mathbf {x}; \pmb {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \pmb {\theta} ^ {(2)}) \| _ {1} \leq \Delta_ {n} / 6 \Bigr \} \\ = \Pi \left\{\left(\boldsymbol {\theta} ^ {(1) \top}, \mathbf {B} ^ {\top}\right) ^ {\top}: \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {x}} \left| \boldsymbol {\beta} _ {j} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \boldsymbol {\beta} _ {j 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \right| \leq \Delta_ {n} / 6 \right\} \\ \geq \Pi \left\{\left(\boldsymbol {\theta} ^ {(1) \top}, \mathbf {B} ^ {\top}\right) ^ {\top}: \mathbb {E} _ {\mathbf {x}} \mid \boldsymbol {\beta} _ {j} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \beta_ {j 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \mid \leq \frac {\Delta_ {n}}{6 K} \text {f o r} j = 1, \dots , K \right\} \\ = \left[ \Pi \left\{\left(\boldsymbol {\theta} ^ {(1) \top}, \boldsymbol {\beta} _ {1} ^ {\top}\right) ^ {\top}: \mathbb {E} _ {\mathbf {x}} \mid \boldsymbol {\beta} _ {1} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \boldsymbol {\beta} _ {1 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \mid \leq \frac {\Delta_ {n}}{6 K} \right\} \right] ^ {K}, \\ \end{array}
$$
where the last equality holds since all columns of  $\mathbf{B}$  have independent and identical prior.
Taking  $\Delta = \frac{\Delta_n}{6K} = \frac{U_1\epsilon_n^2}{6K}$ , by Proposition 1, we have that for any given nonzero  $\psi_{2}(\mathbf{x};\pmb{\theta}^{(2)})$ ,  $\mathcal{M}_1 > 0$  and  $\mathcal{M}_2 > 0$ ,
$$
\begin{array}{l} \Pi \left\{\boldsymbol {\beta} _ {1}: | \boldsymbol {\beta} _ {1} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \boldsymbol {\beta} _ {1 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) | \leq \Delta \Big | \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \right\} \\ > \exp \left\{- \frac {\left\{\psi_ {2} ^ {\top} (\mathbf {x} ; \boldsymbol {\theta} ^ {(2)}) \boldsymbol {\beta} _ {0} \right\} ^ {2} + \Delta^ {2}}{2 \underline {{B}} \| \psi_ {2} (\mathbf {x} ; \boldsymbol {\theta} ^ {(2)}) \| ^ {2}} \right\} \frac {8 \Delta^ {4}}{\overline {{B}} ^ {2} \| \psi_ {2} (\mathbf {x} ; \boldsymbol {\theta} ^ {(2)}) \| ^ {4}} \\ > \exp \left\{- \mathcal {M} _ {1} n \epsilon_ {n} ^ {2} \cdot \frac {F _ {2} ^ {2} \| \psi_ {2} (\mathbf {x} ; \boldsymbol {\theta} ^ {(2)}) \| ^ {2} + \Delta^ {2}}{2 \mathcal {M} _ {1} n \epsilon_ {n} ^ {2} \cdot S _ {2} ^ {(2)} (\log M ^ {(2)}) ^ {- 1} \| \psi_ {2} (\mathbf {x} ; \boldsymbol {\theta} ^ {(2)}) \| ^ {2}} \right\} \frac {8 \Delta^ {4}}{\left\{S _ {1} ^ {(2)} M ^ {(2) q _ {2}} \right\} ^ {2} \| \psi_ {2} (\mathbf {x} ; \boldsymbol {\theta} ^ {(2)}) \| ^ {4}} \\ > \exp \left\{- \mathcal {M} _ {1} n \epsilon_ {n} ^ {2} \cdot \frac {(F _ {2} ^ {2} + 1) \log M ^ {(2)}}{2 \mathcal {M} _ {1} n \epsilon_ {n} ^ {2} \cdot S _ {2} ^ {(2)}} \right\} \\ \cdot \exp \left[ - \left\{2 \log S _ {1} ^ {(2)} + 2 q _ {2} \log M ^ {(2)} + 4 \log \| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \| - \log (8 \Delta^ {4}) \right\} \right] \\ \geq \exp \left\{- \mathcal {M} _ {1} n \epsilon_ {n} ^ {2} \right\} \cdot \exp \left\{- \mathcal {M} _ {2} n \epsilon_ {n} ^ {2} \right\}, \\ \end{array}
$$
for large enough  $n$ , where the second inequality follows from Condition A.4.2 in Section A.2 and Condition B.2.2 in Section A.3, and the third inequality holds since  $\Delta = \frac{\Delta_n}{6K} = \frac{U_1\epsilon_n^2}{6K}$  by definition and  $\epsilon_{n}\rightarrow 0$  as  $n\to \infty$ . The last inequality holds if  $4\log (1 / \epsilon_n^2)\leq \mathcal{M}_2n\epsilon_n^2$  for large enough  $n$  and  $\log \| \psi_2(\mathbf{x};\pmb {\theta}^{(2)})\| \prec \mathcal{M}_2n\epsilon_n^2$ , where, according to Lemma 1, the latter holds if  $H_{n2}\left(\log E_{n2} + \log \frac{r_{n2}}{H_{n2}}\right)\prec \mathcal{M}_2n\epsilon_n^2$ , which can be guaranteed by Condition A.3.1 in Section A.2. Since the result above holds for any given  $\psi_{2}(\mathbf{x};\pmb{\theta}^{(2)})$ , by summarizing the discussion above, we obtain that
$$
\begin{array}{l} \left. \left(\mathrm {I}\right) \geq \left[ \inf  _ {\boldsymbol {\theta} \in \mathcal {F} _ {n}} \Pi \left\{\boldsymbol {\beta} _ {1}: \mathbb {E} _ {\mathbf {x}} \mid \boldsymbol {\beta} _ {1} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \boldsymbol {\beta} _ {1 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \right| \leq \frac {\Delta_ {n}}{6 K} \mid \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \right\} \right] ^ {K} \\ \geq \exp \{- K (\mathcal {M} _ {1} + \mathcal {M} _ {2}) n \epsilon_ {n} ^ {2} \}. \\ \end{array}
$$
Now we consider the second term (II) in (23), which can be written as
$$
\begin{array}{l} \left. \right.\left(\Pi\right) = \Pi \left\{\boldsymbol {\theta} ^ {(2)}: \mathbb {E} _ {\mathbf {x}} \left\| \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) \right\| _ {1} \leq \Delta_ {n} / 6 \right\} \\ = \Pi \left\{\boldsymbol {\theta} ^ {(2)}: \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {x}} \left| \boldsymbol {\beta} _ {j 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \boldsymbol {\beta} _ {j 0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) \right| \leq \Delta_ {n} / 6 \right\} \\ \geq \Pi \left\{\boldsymbol {\theta} ^ {(2)}: \mathbb {E} _ {\mathbf {x}} \| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) \| _ {1} \leq \frac {\Delta_ {n}}{6 K F _ {2}} \right\}, \\ \end{array}
$$
where the last inequality follows from Condition A.4.2 in Section A.2.
Consider the set
$$
\mathcal {S} ^ {(2)} = \left\{\boldsymbol {\theta} ^ {(2)}: \left| \boldsymbol {\theta} _ {j} ^ {(2)} - \boldsymbol {\theta} _ {j} ^ {(2) *} \right| \leq \eta_ {n} \text {f o r} j \in \boldsymbol {\gamma} ^ {(2) *}; \left| \boldsymbol {\theta} _ {j} ^ {(2)} - \boldsymbol {\theta} _ {j} ^ {(2) *} \right| \leq \eta_ {n} ^ {\prime} \text {f o r} j \notin \boldsymbol {\gamma} ^ {(2) *} \right\}, \tag {24}
$$
where  $\eta_{n}$  and  $\eta_n^\prime$  will be specified later. Let  $r_h$  denote the number of nonzero connections to the  $h$ th hidden layer which includes the bias for the  $h$ th hidden layer and the weights between the  $(h - 1)\mathrm{th}$  and the  $h$ th layer, such that  $\sum r_h = \| \pmb{\gamma}^{(2)*}\| _1\triangleq r_{n2}$ . Then, for any  $\mathbf{x}$  satisfying  $\| \mathbf{x}\|_{\infty}\leq 1$
$$
\begin{array}{l} \left\| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) \right\| _ {1} \\ \leq \eta_ {n} H _ {n 2} \left(E _ {n 2} + \eta_ {n}\right) ^ {H _ {n 2} - 1} \prod_ {h = 1} ^ {H _ {n 2}} r _ {h} + \eta_ {n} ^ {\prime} J _ {n 2} \prod_ {h = 1} ^ {H _ {n 2}} \left(E _ {n 2} + \eta_ {n} + \frac {\eta_ {n} ^ {\prime} L _ {h} ^ {(2)}}{r _ {h}}\right) r _ {h} \\ \leq \eta_ {n} H _ {n 2} \left(U _ {2} E _ {n 2}\right) ^ {H _ {n 2}} \left(\frac {\sum r _ {h}}{H _ {n 2}}\right) ^ {H _ {n 2}} + \eta_ {n} ^ {\prime} J _ {n 2} \left(U _ {2} E _ {n 2}\right) ^ {H _ {n 2}} \left(\frac {\sum r _ {h}}{H _ {n 2}}\right) ^ {H _ {n 2}} \\ \leq 2 U _ {3} \epsilon_ {n} ^ {2}, \\ \end{array}
$$
where the first inequality follows from Lemma 2, and the second and the third inequalities hold if we take  $\eta_{n} = U_{3}\epsilon_{n}^{2} / \{H_{n2}(U_{2}E_{n2})^{H_{n2}}(r_{n2} / H_{n2})^{H_{n2}}\}$  and  $\eta_n' = U_3\epsilon_n^2 /\{J_{n2}(U_2E_{n2})^{H_{n2}}(r_{n2} / H_{n2})^{H_{n2}}\}$  for some constants  $U_{2} > 1$  and  $U_{3} > 0$ . By taking a small enough  $U_{3}$ , we can obtain that  $\mathbb{E}_{\mathbf{x}}\| \psi_2(\mathbf{x};\pmb{\theta}^{(2)}) - \psi_2(\mathbf{x};\pmb{\theta}^{(2)*})\| _1\leq \frac{\Delta_n}{6KF_1}$  for any  $\pmb{\theta}^{(2)}\in S^{(2)}$  defined in (24). Then, it suffices to prove that  $\Pi (S^{(2)}) = \Pi^{(2)}(S^{(2)})\geq \exp (-\mathcal{M}_3n\epsilon_n^2)$  for some positive constant  $\mathcal{M}_3$ , where  $\Pi^{(2)}(\cdot)$  is the prior measure of  $\pmb{\theta}^{(2)}$  as given in Section A.3.
As defined in (5) in Section 3.2, each element of  $\theta^{(2)}$  has an independent continuous prior, denoted  $\pi_{\theta}^{(2)}(\cdot)$ , and we use  $\underline{\pi}_{\theta}^{(2)}$  to denote the minimal density value of  $\pi_{\theta}^{(2)}(\cdot)$  on the interval  $[-E_{n2} - 1, E_{n2} + 1]$ . Then, we obtain that
$$
\begin{array}{l} \Pi^ {(2)} (\mathcal {S} ^ {(2)}) \\ = \prod_ {j \in \mathcal {Y} ^ {(2) *}} \pi_ {\theta} ^ {(2)} \left(\theta_ {j} ^ {(2) *} - \eta_ {n} \leq \theta_ {j} ^ {(2)} \leq \theta_ {j} ^ {(2) *} + \eta_ {n}\right) \prod_ {j \notin \mathcal {Y} ^ {(2) *}} \pi_ {\theta} ^ {(2)} \left(\theta_ {j} ^ {(2) *} - \eta_ {n} ^ {\prime} \leq \theta_ {j} ^ {(2)} \leq \theta_ {j} ^ {(2) *} + \eta_ {n} ^ {\prime}\right) \\ \geq \left(2 \underline {{\pi}} _ {\theta} ^ {(2)} \eta_ {n}\right) ^ {r _ {n 2}} \left\{\pi_ {\theta} ^ {(2)} \left(\theta_ {1} ^ {(2)} \in [ - \eta_ {n} ^ {\prime}, \eta_ {n} ^ {\prime} ]\right) \right\} ^ {J _ {n 2}}, \\ \end{array}
$$
and thus, according to Conditions B.1.1 and B.1.2 in Section A.3, we have that
$$
\begin{array}{l} - \log \Pi^ {(2)} (\mathcal {S} ^ {(2)}) \leq r _ {n 2} \left\{\log \frac {1}{2} + \log \frac {1}{\underline {{\pi}} _ {\theta} ^ {(2)}} + \log \frac {1}{\eta_ {n}} \right\} + J _ {n 2} \log \left\{\frac {1}{\pi_ {\theta} ^ {(2)} \left(\theta_ {1} ^ {(2)} \in [ - \eta_ {n} ^ {\prime} , \eta_ {n} ^ {\prime} ]\right)} \right\} \\ \leq r _ {n 2} \left\{\text {c o n s t a n t} + H _ {n 2} \log n + \log \bar {L} _ {n 2} + \log \left(\frac {1}{\epsilon_ {n} ^ {2}}\right) + \log H _ {n 2} \right. \\ \left. + H _ {n 2} \log \left(U _ {2} E _ {n 2}\right) + H _ {n 2} \log \left(\frac {r _ {n 2}}{H _ {n 2}}\right) \right\} + J _ {n 2} \log \left(\frac {J _ {n 2}}{J _ {n 2} - 1}\right) \\ \asymp r _ {n 2} H _ {n 2} \log n + r _ {n 2} \log \bar {L} _ {n 2}, \\ \end{array}
$$
where the last asymptotic equality follows from Condition A.3 in Section A.2 and the fact that  $\log (1 / \epsilon_n) = O(\log n)$ . Thus,  $(\Pi) \geq \Pi(S^{(2)}) \geq \exp(-\mathcal{M}_3n\epsilon_n^2)$  holds if  $r_{n2}H_{n2}\log n + r_{n2}\log \overline{L}_{n2} \leq U_4n\epsilon_n^2$  for some sufficiently small positive constant  $U_4$ .
Now we consider the third term in (23):
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x}} \| \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}) \| _ {1} = \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {x}} \left| \beta_ {j 0} ^ {\top} \left\{\psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) - \psi_ {2} (\mathbf {x}) \right\} \right| \\ \leq K F _ {2} \mathbb {E} _ {\mathbf {x}} \left| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) - \psi_ {2} (\mathbf {x}) \right| \\ \le K F _ {2} \varpi_ {n 2}, \\ \end{array}
$$
where the second inequality follows from Condition A.4.2 in Section A.2 and the last inequality follows from the definition of the true model given in (14). Thus, we can take  $\epsilon_{n}^{2} = O(\varpi_{n2})$  so that (III)  $\approx 1$  for large enough  $n$ .
Similar discussion can be applied to  $(\mathrm{I})'$ ,  $(\mathrm{II})'$  and  $(\mathrm{III})'$  in (23) and we can obtain that
$$
\begin{array}{l} (I) ^ {\prime} \geq \left[ \inf  _ {\boldsymbol {\theta} \in \mathcal {F} _ {n}} \Pi \left\{\boldsymbol {\alpha} _ {1}: \mathbb {E} _ {\mathbf {x}} \mid \boldsymbol {\alpha} _ {1} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \boldsymbol {\alpha} _ {1 0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) \right| \leq \frac {\Delta_ {n}}{6 K} \left| \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) \right\} \right] ^ {K} \\ \geq \exp \{- K (\mathcal {M} _ {1} ^ {\prime} + \mathcal {M} _ {2} ^ {\prime}) n \epsilon_ {n} ^ {2} \}; \\ \end{array}
$$
$$
\begin{array}{l} (\Pi) ^ {\prime} \geq \Pi (\boldsymbol {\theta} ^ {(1)}: \mathcal {S} ^ {(1)}) \geq \exp (- \mathcal {M} _ {3} ^ {\prime} n \epsilon_ {n} ^ {2}); \\ \left(\mathrm {I I I}\right) ^ {\prime} \approx 1 \text {f o r l a r g e e n o u g h} n, \\ \end{array}
$$
where  $\mathcal{M}_1'$ ,  $\mathcal{M}_2'$  and  $\mathcal{M}_3'$  are constants, and  $S^{(1)}$  can be similarly defined as in (24). Since the discussion above holds for any  $k \in [K]$  and  $r \in [R]$ , by choosing proper  $\mathcal{M}_1$ ,  $\mathcal{M}_2$ ,  $\mathcal{M}_3$ ,  $\mathcal{M}_1'$ ,  $\mathcal{M}_2'$  and  $\mathcal{M}_3'$ , Condition (c) of Lemma 3 can be verified.
B.2.3 Verification of Condition (a) in Lemma 3
Proof. Condition (a) in Lemma 3 can be verified if we can find a parameter set  $\Theta_n \subset \Theta$  such that  $\log N(\epsilon_n / 36, \mathcal{P}_n, d_n) \leq n\epsilon^2$ , where  $\mathcal{P}_n$  represents the set of all densities that can be represented by model (2) in Section 3.1 with parameters in  $\Theta_n$ . Consider the following parameter space:
$$
\Theta_ {n} = \mathcal {S} _ {n 1} \otimes \mathcal {S} _ {n 2} \otimes \mathcal {B} _ {n} \otimes \mathcal {A} _ {n},
$$
$$
\mathcal {S} _ {n 1} = \{\boldsymbol {\theta} ^ {(1)}: | \boldsymbol {\theta} _ {j} ^ {(1)} | \leq I _ {n} ^ {(1)}, | \boldsymbol {\gamma} _ {\theta} ^ {(1)} | = | \{j: | \boldsymbol {\theta} _ {j} ^ {(1)} | \geq \delta_ {n 1} ^ {\prime} \} | \leq k _ {n 1} r _ {n 1}, | \boldsymbol {\gamma} _ {\theta} ^ {(1)} | _ {\mathrm {i n}} \leq k _ {n 1} ^ {\prime} s _ {n 1} \};
$$
$$
S _ {n 2} = \{\boldsymbol {\theta} ^ {(2)}: | \theta_ {j} ^ {(2)} | \leq I _ {n} ^ {(2)}, | \boldsymbol {\gamma} _ {\theta} ^ {(2)} | = | \{j: | \theta_ {j} ^ {(2)} | \geq \delta_ {n 2} ^ {\prime} \} | \leq k _ {n 2} r _ {n 2}, | \boldsymbol {\gamma} _ {\theta} ^ {(2)} | _ {\mathrm {i n}} \leq k _ {n 2} ^ {\prime} s _ {n 2} \};
$$
$$
\mathcal {B} _ {n} = \left\{\mathbf {B} ^ {(k)} = \left(\beta_ {1} ^ {(k)},..., \beta_ {K} ^ {(k)}\right) \text {f o r} k \in [ K ]: | \beta_ {l, j} ^ {(k)} | \leq b _ {n}, \text {f o r} k, l \in [ K ] \text {a n d} j \in [ M ^ {(1)} ] \right\};
$$
$$
\mathcal {A} _ {n} = \left\{\mathbf {A} ^ {(r)} = \left(\boldsymbol {\alpha} _ {1} ^ {(r)}, \dots , \boldsymbol {\alpha} _ {K} ^ {(r)}\right) \text {f o r} r \in [ R ]: | \alpha_ {l, j} ^ {(r)} | \leq a _ {n}, \text {f o r} r \in [ R ], l \in [ K ], \text {a n d} j \in [ M ^ {(2)} ] \right\}.
$$
Here,  $|\gamma_{\theta}^{(1)}|_{\mathrm{in}}$  and  $|\gamma_{\theta}^{(2)}|_{\mathrm{in}}$  denote the input dimensions of the sparse network structures  $\gamma_{\theta}^{(1)}$  and  $\gamma_{\theta}^{(1)}$ , respectively, and  $I_n^{(1)}, I_n^{(2)}, \delta_{n1}', \delta_{n2}', k_{n1} (\leq n / r_{n1}), k_{n2} (\leq n / r_{n2}), k_{n1}' (\leq n / s_{n1}), k_{n2}' (\leq n / s_{n2}), b_n$  and  $a_n$  are positive constants whose values will be described later.
The parameter space  $\Theta_{n}$  can be covered by a set of  $L_{\infty}$  balls of the form  $(\varrho_{j} - \rho_{n},\varrho_{j} + \rho_{n})_{j = 1}^{W_{n}}$  where for the  $j$ th coordinate,  $\varrho_{j}$  is the center which lies inside  $\Theta_{n}$ ,  $\rho_{n}$  is the radius, and  $W_{n} = J_{n1} + J_{n2} + K^{2}M^{(1)} + RKM^{(2)}$  is the number of all involved parameters. It can be verified that the number of balls needed to cover the parameter space is upper bounded by
$$
\begin{array}{l} \mathcal {K} _ {n} \triangleq \left\{\sum_ {j = 1} ^ {k _ {n 1} r _ {n 1}} \chi^ {(1)} (j) \left(\frac {I _ {n} ^ {(1)}}{\rho_ {n}} + 1\right) ^ {j} \right\} \cdot \left\{\sum_ {j = 1} ^ {k _ {n 2} r _ {n 2}} \chi^ {(2)} (j) \left(\frac {I _ {n} ^ {(2)}}{\rho_ {n}} + 1\right) ^ {j} \right\} \\ \cdot \left(\frac {a _ {n}}{\rho_ {n}} + 1\right) ^ {K ^ {2} M ^ {(1)}} \cdot \left(\frac {b _ {n}}{\rho_ {n}} + 1\right) ^ {R K M ^ {(2)}} \\ \triangleq \mathcal {K} _ {n 1} \cdot \mathcal {K} _ {n 2} \cdot \mathcal {K} _ {n 3} \cdot \mathcal {K} _ {n 4}, \tag {25} \\ \end{array}
$$
where for  $l = 1,2$ ,  $\chi^{(l)}(j)$  denotes the number of all valid networks with exactly  $j$  connections and no more than  $k_{nl}^{\prime}s_{nl}$  inputs.
Then we consider the number of  $d_{n}$ -balls needed to cover the set of all densities,  $\mathcal{P}_n$ . Consider two parameters in  $\Theta_n$ :  $\pmb{\theta}_u \triangleq (\pmb{\theta}_u^{(1)\top}, \pmb{\theta}_u^{(2)\top}, \pmb{B}_u^{(1)\top}, \dots, \pmb{B}_u^{(K)\top}, \pmb{A}_u^{(1)\top}, \dots, \pmb{A}_u^{(R)\top})^\top$  and  $\pmb{\theta}_v \triangleq (\pmb{\theta}_v^{(1)\top}, \pmb{\theta}_v^{(2)\top}, \pmb{B}_v^{(1)\top}, \dots, \pmb{B}_v^{(K)\top}, \pmb{A}_v^{(1)\top}, \dots, \pmb{A}_v^{(R)\top})^\top$ , which satisfy that
(i) there exists a network structure  $\gamma^{(1)}$  such that  $|\gamma^{(1)}| \leq k_{n1}r_{n1}$ ,  $|\gamma^{(1)}|_{\mathrm{in}} \leq k_{n1}'s_{n1}$ ,  $|\theta_{u,j}^{(1)} - \theta_{v,j}^{(1)}| \leq \rho_n$  for  $j \in \gamma^{(1)}$ ,  $\theta_{u,j}^{(1)} \leq \delta_{n1}'$  and  $\theta_{v,j}^{(1)} \leq \delta_{n1}'$  for  $j \notin \gamma^{(1)}$ ;
(ii) there exists a network structure  $\gamma^{(2)}$  such that  $|\gamma^{(2)}| \leq k_{n2}r_{n2}, |\gamma^{(2)}|_{\mathrm{in}} \leq k_{n2}'s_{n2}, |\theta_{u,j}^{(2)} - \theta_{v,j}^{(2)}| \leq \rho_n$  for  $j \in \gamma^{(2)}, \theta_{u,j}^{(2)} \leq \delta_{n2}'$  and  $\theta_{v,j}^{(2)} \leq \delta_{n2}'$  for  $j \notin \gamma^{(2)}$ ;
(iii)  $|\beta_{ul,j}^{(k)} - \beta_{vi,j}^{(k)}| \leq \rho_n$  for  $k, l \in [K]$  and  $j \in [M^{(1)}]$ ;
(iv)  $|\alpha_{ul,j}^{(r)} - \alpha_{vl,j}^{(r)}| \leq \rho_n$  for  $r \in [R], l \in [K]$ , and  $j \in [M^{(2)}]$ ,
where  $s_{n1}$  and  $s_{n2}$  are defined in Condition A.3.1 in Appendix A.2, and  $\rho_n$  is a positive constant whose values will be discussed later.
Let  $f_{\pmb{\theta}_u}^{(k)}$  and  $f_{\pmb{\theta}_v}^{(k)}$  denote the corresponding densities in  $\mathcal{P}_n$  for  $k \in [K]$ . By (21) in Result 1 and the fact that  $d(f_1, f_2) \leq \sqrt{K(f_1, f_2)}$  in Appendix B.1.1, we obtain that  $d_n^2(\pmb{\theta}_u, \pmb{\theta}_v) \leq \max_{k \in [K], r \in [R]} U_5 \mathbb{E}_{\mathbf{x}} \| \pmb{\omega}_u^{(k,r)} - \pmb{\omega}_v^{(k,r)} \|_1$  for some positive constant  $U_5$ , where  $\pmb{\omega}_u^{(k,r)} = \mathbf{B}_u^{(k)\top} \psi_2(\mathbf{x}; \pmb{\theta}_u^{(2)}) + \mathbf{A}_u^{(r)\top} \psi_1(\mathbf{x}; \pmb{\theta}_u^{(1)})$  and  $\pmb{\omega}_v^{(k,r)} = \mathbf{B}_v^{(k)\top} \psi_2(\mathbf{x}; \pmb{\theta}_v^{(2)}) + \mathbf{A}_v^{(r)\top} \psi_1(\mathbf{x}; \pmb{\theta}_v^{(1)})$ .
for  $k \in [K]$  and  $r \in [R]$ . For ease of presentation, we omit  $k$  and  $r$  in  $B_u^{(k)}, B_v^{(k)}, A_u^{(r)}, A_v^{(r)}, \omega_u^{(k,r)}$  and  $\omega_v^{(k,r)}$ , and further obtain that
$$
\begin{array}{l} \| \boldsymbol {\omega} _ {u} - \boldsymbol {\omega} _ {v} \| _ {1} \leq \| \mathbf {B} _ {u} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(2)}) - \mathbf {B} _ {v} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \| _ {1} + \| \mathbf {A} _ {u} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(1)}) - \mathbf {A} _ {v} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) \| _ {1} \\ \leq \left\| \mathbf {B} _ {u} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(2)}) - \mathbf {B} _ {u} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \right\| _ {1} + \left\| \mathbf {B} _ {u} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) - \mathbf {B} _ {v} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \right\| _ {1} \\ + \left\| \mathbf {A} _ {u} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(1)}) - \mathbf {A} _ {u} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) \right\| _ {1} + \left\| \mathbf {A} _ {u} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) - \mathbf {A} _ {v} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) \right\| _ {1} \\ = \sum_ {i = j} ^ {K} \left| \boldsymbol {\beta} _ {u j} ^ {\top} \left\{\psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(2)}) - \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \right\} \right| + \sum_ {j = 1} ^ {K} \left| \left(\boldsymbol {\beta} _ {u j} - \boldsymbol {\beta} _ {v j}\right) ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \right| \\ + \sum_ {i = 1} ^ {K} | \boldsymbol {\alpha} _ {u i} ^ {\top} \{\psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(1)}) - \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) \} | + \sum_ {i = 1} ^ {K} | (\boldsymbol {\alpha} _ {u i} - \boldsymbol {\alpha} _ {v i}) ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) | \\ \leq K b _ {n} \| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(2)}) - \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \| _ {1} + K \rho_ {n} \| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(2)}) \| \\ + K a _ {n} \| \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {u} ^ {(1)}) - \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) \| _ {1} + K \rho_ {n} \| \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} _ {v} ^ {(1)}) \|, \\ \end{array}
$$
where the preceding four boundness assumptions for  $\theta_{u}$  and  $\theta_{v}$ , together with the definition of  $\mathcal{A}_n$  and  $\mathcal{B}_n$ , are used.
Similar to the proof in Appendix B.2.2, by using Lemmas 1 and 2, we can further obtain that  $d_n^2 (\pmb {\theta}_u,\pmb {\theta}_v)\leq (\epsilon_n / 36)^2$  if we choose  $\rho_{n},\delta_{n1}^{\prime}$  and  $\delta_{n2}^{\prime}$  in the preceding derivations as follows:
(i)  $\delta_{n1}^{\prime} = U_6\epsilon_n^2 /b_nJ_{n1}(U_6^{\prime}I_n^{(1)})^{H_{n1}}(k_{n1}r_{n1} / H_{n1})^{H_{n1}}$  for some constants  $U_{6} > 0$  and  $U_6^\prime >1$
(ii)  $\delta_{n2}^{\prime} = U_7\epsilon_n^2 /a_nJ_{n2}(U_7'I_n^{(2)})^{H_{n2}}(k_{n2}r_{n2} / H_{n2})^{H_{n2}}$  for some constants  $U_{7} > 0$  and  $U_7^\prime >1$
(iii)  $\rho_{n} = \min \left\{U_{8,1}\epsilon_{n}^{2} / b_{n}H_{n1}(U_{8,1}^{\prime}I_{n}^{(1)})^{H_{n1}}(k_{n1}r_{n1} / H_{n1})^{H_{n1}},\right.$