SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma_ {\infty}, B _ {G}, B _ {G 1}} ^ {\infty}} := \min  _ {f \in \mathcal {F} _ {d, r, B _ {F}, S _ {p, \gamma_ {\infty}, B _ {G}, B _ {G 1}} ^ {\infty}}} \mathcal {L} _ {\mathcal {D}} (f). \tag {458}
$$
We consider the data distribution in Appendix E.4.1 with Assumption E.28, i.e.,  $\mathcal{D}_{\text{parity-uniform}}$  in Appendix E.5. Note that with this dataset, we have  $\| \mathbf{x} \|_{\infty} \leq B_{x\infty} = 1$ . We use the following unbiased initialization:
$$
\text {f o r} i \in \{1, \dots , m \}: \quad \mathbf {a} _ {i} ^ {(0)} \sim \mathcal {N} (0, \sigma_ {a} ^ {2}), \mathbf {w} _ {i} ^ {(0)} \sim \{\pm 1 \} ^ {d}, \mathbf {b} _ {i} = \tilde {b} \leq 1,
$$
$$
\text {f o r} i \in \{m + 1, \dots , 2 m \}: \quad \mathbf {a} _ {i} ^ {(0)} = - \mathbf {a} _ {i - m} ^ {(0)}, \mathbf {w} _ {i} ^ {(0)} = - \mathbf {w} _ {i - m} ^ {(0)}, \mathbf {b} _ {i} = - \mathbf {b} _ {i - m},
$$
$$
\text {f o r} i \in \{2 m + 1, \dots , 4 m \}: \quad \mathbf {a} _ {i} ^ {(0)} = - \mathbf {a} _ {i - 2 m} ^ {(0)}, \mathbf {w} _ {i} ^ {(0)} = \mathbf {w} _ {i - 2 m} ^ {(0)}, \mathbf {b} _ {i} = \mathbf {b} _ {i - 2 m} \tag {459}
$$
Let  $\nabla_{i}$  denote the gradient of the  $i$ -th neuron  $\nabla_{\mathbf{w}_i}\mathcal{L}_{\mathcal{D}}(f_{\Xi^{(0)}})$ . Denote the subset of neurons with nice gradients approximating feature  $(D,s)$  as:
$$
G _ {(D, s), N i c e} ^ {\infty} := \left\{i \in [ 2 m ]: s = \frac {\mathbf {b} _ {i}}{\left| \mathbf {b} _ {i} \right|}, \left\| \frac {\nabla_ {i}}{\left\| \nabla_ {i} \right\|} - D \right\| _ {\infty} \leq \gamma_ {\infty}, \left| \mathbf {a} _ {i} ^ {(0)} \right| B _ {G 1} \geq \| \nabla_ {i} \| _ {2} \geq \left| \mathbf {a} _ {i} ^ {(0)} \right| B _ {G} \right\}.
$$
Lemma E.34 (Existence of Good Networks. Modified Version of Lemma 3.14 Under Uniform Parity Setting). Let  $\lambda^{(1)} = \frac{1}{\eta^{(1)}}$ . For any  $B_{\epsilon} \in (0, B_{b})$ , let  $\sigma_{a} = \Theta \left(\frac{\tilde{b}}{-\ell'(0)\eta^{(1)}B_{G}B_{\epsilon}}\right)$  and  $\delta = 2re^{-\sqrt{mp}} + \frac{1}{d^2}$ . Then, with probability at least  $1 - \delta$  over the initialization, there exists  $\tilde{\mathbf{a}}_i$ 's such that  $f_{(\tilde{\mathbf{a}},\mathbf{W}^{(1)},\mathbf{b})}(\mathbf{x}) = \sum_{i=1}^{4m} \tilde{\mathbf{a}}_i \sigma\left(\left\langle \mathbf{w}_i^{(1)}, \mathbf{x} \right\rangle - \mathbf{b}_i\right)$  satisfies
$$
\mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \leq r B _ {a 1} \left(\frac {B _ {x 1} B _ {G 1} B _ {b}}{\sqrt {m p} B _ {G} B _ {\epsilon}} + \sqrt {2 \log (d) d} \gamma_ {\infty} + B _ {\epsilon}\right) + \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma_ {\infty}, B _ {G}, B _ {G 1}} ^ {\infty}},
$$
Proof of Lemma E.34. Recall  $f^{*}(\mathbf{x}) = \sum_{j=1}^{r} \mathbf{a}_{j}^{*} \sigma(\langle \mathbf{w}_{j}^{*}, \mathbf{x} \rangle - \mathbf{b}_{j}^{*})$ , where  $f^{*} \in \mathcal{F}_{d,r,B_{F},S_{p,\gamma_{\infty}},B_{G},B_{G1}}$  is defined in Definition E.33 and let  $s_{j}^{*} = \frac{\mathbf{b}_{j}^{*}}{|\mathbf{b}_{j}^{*}|}$ . By Lemma D.3, with probability at least  $1 - \delta_{1}$ ,  $\delta_{1} = 2re^{-cmp}$ , for all  $j \in [r]$ , we have  $|G_{(\mathbf{w}_{j}^{*},s_{j}^{*}),Nice}| \geq \frac{mp}{4}$ . Then for all  $i \in G_{(\mathbf{w}_{j}^{*},s_{j}^{*}),Nice}^{\infty} \subseteq [2m]$ , we have  $-\ell'(0)\eta^{(1)}G(\mathbf{w}_{i}^{(0)},\mathbf{b}_{i})\frac{\mathbf{b}_{j}^{*}}{b}$  only depend on  $\mathbf{w}_{i}^{(0)}$  and  $\mathbf{b}_{i}$ , which is independent of  $\mathbf{a}_{i}^{(0)}$ . Given Definition E.32, we have
$$
- \ell^ {\prime} (0) \eta^ {(1)} \| G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right) \| _ {2} \frac {\mathbf {b} _ {j} ^ {*}}{\tilde {b}} \in \left[ \ell^ {\prime} (0) \eta^ {(1)} B _ {x 1} \frac {B _ {b}}{\tilde {b}}, - \ell^ {\prime} (0) \eta^ {(1)} B _ {x 1} \frac {B _ {b}}{\tilde {b}} \right]. \tag {460}
$$
We split  $[r]$  into  $\Gamma = \{j\in [r]:|\mathbf{b}_j^* | < B_\epsilon \}$ ,  $\Gamma_{-} = \{j\in [r]:\mathbf{b}_{j}^{*}\leq -B_{\epsilon}\}$  and  $\Gamma_{+} = \{j\in [r]:$ $\mathbf{b}_j^*\geq B_\epsilon \}$ . Let  $\epsilon_{a} = \frac{B_{G1}B_{b}}{\sqrt{mpB_{G}B_{\epsilon}}}$ . Then we know that for all  $j\in \Gamma_{+}\cup \Gamma_{-}$ , for all  $i\in G_{(\mathbf{w}_j^*,s_j^*)},Nice,$  we have
$$
\Pr_ {\mathbf {a} _ {i} ^ {(0)} \sim \mathcal {N} \left(0, \sigma_ {a} ^ {2}\right)} \left[ \left| - \mathbf {a} _ {i} ^ {(0)} \ell^ {\prime} (0) \eta^ {(1)} \| G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right) \| _ {2} \frac {\left| \mathbf {b} _ {j} ^ {*} \right|}{\tilde {b}} - 1 \right| \leq \epsilon_ {a} \right] \tag {461}
$$
$$
= \Pr_ {\mathbf {a} _ {i} ^ {(0)} \sim \mathcal {N} \left(0, \sigma_ {a} ^ {2}\right)} \left[ 1 - \epsilon_ {a} \leq - \mathbf {a} _ {i} ^ {(0)} \ell^ {\prime} (0) \eta^ {(1)} \| G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right) \| _ {2} \frac {\left| \mathbf {b} _ {j} ^ {*} \right|}{\tilde {b}} \leq 1 + \epsilon_ {a} \right] \tag {462}
$$
$$
= \Pr_ {g \sim \mathcal {N} (0, 1)} \left[ 1 - \epsilon_ {a} \leq g \Theta \left(\frac {\| G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) \| _ {2} \left| \mathbf {b} _ {j} ^ {*} \right|}{B _ {G} B _ {\epsilon}}\right) \leq 1 + \epsilon_ {a} \right] \tag {463}
$$
$$
= \Pr_ {g \sim \mathcal {N} (0, 1)} \left[ (1 - \epsilon_ {a}) \Theta \left(\frac {B _ {G} B _ {\epsilon}}{\| G (\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}) \| _ {2} | \mathbf {b} _ {j} ^ {*} |}\right) \leq g \leq (1 + \epsilon_ {a}) \Theta \left(\frac {B _ {G} B _ {\epsilon}}{\| G (\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}) \| _ {2} | \mathbf {b} _ {j} ^ {*} |}\right) \right]
$$
$$
= \Theta \left(\frac {\epsilon_ {a} B _ {G} B _ {\epsilon}}{\| G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) \| _ {2} \left| \mathbf {b} _ {j} ^ {*} \right|}\right) \tag {464}
$$
$$
\geq \Omega \left(\frac {\epsilon_ {a} B _ {G} B _ {\epsilon}}{B _ {G 1} B _ {b}}\right) \tag {465}
$$
$$
= \Omega \left(\frac {1}{\sqrt {m p}}\right). \tag {466}
$$
$$
\left| - \mathbf {a} _ {i} ^ {(0)} \ell^ {\prime} (0) \eta^ {(1)} \| G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right) \| _ {2} \frac {\left| \mathbf {b} _ {j} ^ {*} \right|}{\tilde {b}} - 1 \right| \leq \epsilon_ {a}, \quad \left| \mathbf {a} _ {i} ^ {(0)} \right| = O \left(\frac {\tilde {b}}{- \ell^ {\prime} (0) \eta^ {(1)} B _ {G} B _ {\epsilon}}\right). \tag {467}
$$
Similarly, for  $j \in \Gamma$ , for all  $i \in G_{(\mathbf{w}_j^*, s_j^*), Nice}^\infty$ , with probability  $\Omega\left(\frac{1}{\sqrt{mp}}\right)$  over  $\mathbf{a}_i^{(0)}$ , we have
$$
\left| - \mathbf {a} _ {i} ^ {(0)} \ell^ {\prime} (0) \eta^ {(1)} \| G (\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}) \| _ {2} \frac {B _ {\epsilon}}{\tilde {b}} - 1 \right| \leq \epsilon_ {a}, \quad \left| \mathbf {a} _ {i} ^ {(0)} \right| = O \left(\frac {\tilde {b}}{- \ell^ {\prime} (0) \eta^ {(1)} B _ {G} B _ {\epsilon}}\right). \tag {468}
$$
For all  $j \in [r]$ , let  $\Lambda_j \subseteq G_{(\mathbf{w}_j^*, s_j^*), Nice}^\infty$  be the set of  $i$ 's such that condition Equation (467) or Equation (468) are satisfied. By Chernoff bound and union bound, with probability at least  $1 - \delta_2$ ,  $\delta_2 = r e^{-\sqrt{mp}}$ , for all  $j \in [r]$  we have  $|\Lambda_j| \geq \Omega(\sqrt{mp})$ . We have for  $\forall j \in \Gamma_+ \cup \Gamma_-, \forall i \in \Lambda_j$ ,
$$
\begin{array}{l} \left| \frac {\left| \mathbf {b} _ {j} ^ {*} \right|}{\tilde {b}} \left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle \right| (469) \\ \leq \left| \left\langle - \mathbf {a} _ {i} ^ {(0)} \ell^ {\prime} (0) \eta^ {(1)} \| G (\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}) \| _ {2} \frac {| \mathbf {b} _ {j} ^ {*} |}{\tilde {b}} \frac {\mathbf {w} _ {i} ^ {(1)}}{\| \mathbf {w} _ {i} ^ {(1)} \| _ {2}} - \frac {\mathbf {w} _ {i} ^ {(1)}}{\| \mathbf {w} _ {i} ^ {(1)} \| _ {2}}, \mathbf {x} \right\rangle + \left\langle \frac {\mathbf {w} _ {i} ^ {(1)}}{\| \mathbf {w} _ {i} ^ {(1)} \| _ {2}} - \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle \right| \\ \leq \epsilon_ {a} \| \mathbf {x} \| _ {2} + \sqrt {2 \log (d) d} \gamma_ {\infty}. (470) \\ \end{array}
$$
With probability  $1 - \frac{1}{d^2}$  by Hoeffding's inequality. Similarly, for  $\forall j\in \Gamma ,\forall i\in \Lambda_{j}$
$$
\left| \frac {B _ {\epsilon}}{\tilde {b}} \left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle \right| \leq \epsilon_ {a} \| \mathbf {x} \| _ {2} + \sqrt {2 \log (d) d} \gamma_ {\infty}. \tag {471}
$$
If  $i\in \Lambda_j$ $j\in \Gamma_{+}\cup \Gamma_{-}$  , set  $\tilde{\mathbf{a}}_i = \mathbf{a}_j^*\frac{|\mathbf{b}_j^*|}{|\Lambda_j|\tilde{b}}$  , if  $i\in \Lambda_j$ $j\in \Gamma$  , set  $\tilde{\mathbf{a}}_i = \mathbf{a}_j^*\frac{B_\epsilon}{|\Lambda_j|\tilde{b}}$  otherwise set  $\tilde{\mathbf{a}}_i = 0$  we have  $\| \tilde{\mathbf{a}}\| _0 = O\left(r(mp)^{\frac{1}{2}}\right),\| \tilde{\mathbf{a}}\| _2 = O\left(\frac{B_{a2}B_b}{\tilde{b}(mp)^{\frac{1}{4}}}\right),\| \tilde{\mathbf{a}}\|_{\infty} = O\left(\frac{B_{a1}B_b}{\tilde{b}(mp)^{\frac{1}{2}}}\right)$
$$
\mathcal {L} _ {\mathcal {D}} \left(f _ {(\hat {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \tag {472}
$$
$$
= \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) - \mathcal {L} _ {\mathcal {D}} \left(f ^ {*}\right) + \mathcal {L} _ {\mathcal {D}} \left(f ^ {*}\right) \tag {473}
$$
$$
\leq \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})} (\mathbf {x}) - f ^ {*} (\mathbf {x}) \right| \right] + \mathcal {L} _ {\mathcal {D}} \left(f ^ {*}\right) \tag {474}
$$
$$
\leq \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {i = 1} ^ {m} \tilde {\mathbf {a}} _ {i} \sigma \left(\left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \tilde {b}\right) + \sum_ {i = m + 1} ^ {2 m} \tilde {\mathbf {a}} _ {i} \sigma \left(\left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle + \tilde {b}\right) - \sum_ {j = 1} ^ {r} \mathbf {a} _ {j} ^ {*} \sigma \left(\left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle - \mathbf {b} _ {j} ^ {*}\right) \right| \right]
$$
$$
+ \mathcal {L} _ {\mathcal {D}} (f ^ {*}) \tag {475}
$$
$$
\leq \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {j \in \Gamma_ {+}} \sum_ {i \in \Lambda_ {j}} \mathbf {a} _ {j} ^ {*} \frac {1}{| \Lambda_ {j} |} \left| \frac {| \mathbf {b} _ {j} ^ {*} |}{\tilde {b}} \sigma \left(\left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \tilde {b}\right) - \sigma \left(\left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle - \mathbf {b} _ {j} ^ {*}\right) \right| \right] \right. \tag {476}
$$
$$
+ \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {j \in \Gamma_ {-}} \sum_ {i \in \Lambda_ {j}} \mathbf {a} _ {j} ^ {*} \frac {1}{| \Lambda_ {j} |} \left| \frac {| \mathbf {b} _ {j} ^ {*} |}{\tilde {b}} \sigma \left(\left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle + \tilde {b}\right) - \sigma \left(\left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle - \mathbf {b} _ {j} ^ {*}\right) \right| \right| \right] \tag {477}
$$
$$
+ \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {j \in \Gamma} \sum_ {i \in \Lambda_ {j}} \mathbf {a} _ {j} ^ {*} \frac {1}{| \Lambda_ {j} |} \left| \frac {B _ {\epsilon}}{\tilde {b}} \sigma \left(\left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \tilde {b}\right) - \sigma \left(\left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle - \mathbf {b} _ {j} ^ {*}\right) \right| \right| \right] + \mathcal {L} _ {\mathcal {D}} (f ^ {*}) \tag {478}
$$
$$
\leq \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {j \in \Gamma_ {+}} \sum_ {i \in \Lambda_ {j}} \mathbf {a} _ {j} ^ {*} \frac {1}{| \Lambda_ {j} |} \left| \frac {| \mathbf {b} _ {j} ^ {*} |}{\tilde {b}} \left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle \right| \right| \right] \tag {479}
$$
$$
+ \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {j \in \Gamma_ {-}} \sum_ {i \in \Lambda_ {j}} \mathbf {a} _ {j} ^ {*} \frac {1}{| \Lambda_ {j} |} \left| \frac {| \mathbf {b} _ {j} ^ {*} |}{\tilde {b}} \left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle - \left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle \right| \right| \right] \tag {480}
$$
$$
+ \mathbb {E} _ {(\mathbf {x}, y)} \left[ \left| \sum_ {j \in \Gamma} \sum_ {i \in \Lambda_ {j}} \mathbf {a} _ {j} ^ {*} \frac {1}{| \Lambda_ {j} |} \left| \frac {B _ {\epsilon}}{\tilde {b}} \left\langle \mathbf {w} _ {i} ^ {(1)}, \mathbf {x} \right\rangle + B _ {\epsilon} - \left\langle \mathbf {w} _ {j} ^ {*}, \mathbf {x} \right\rangle \right| \right| \right] + \mathcal {L} _ {\mathcal {D}} (f ^ {*}) \tag {481}
$$
$$
\leq r \| \mathbf {a} ^ {*} \| _ {\infty} \left(\epsilon_ {a} \mathbb {E} _ {(\mathbf {x}, y)} \| \mathbf {x} \| _ {2} + \sqrt {2 \log (d) d} \gamma_ {\infty}\right) + | \Gamma | \| \mathbf {a} ^ {*} \| _ {\infty} B _ {\epsilon} + \mathcal {L} _ {\mathcal {D}} \left(f ^ {*}\right) \tag {482}
$$
$$
\leq r B _ {a 1} \left(\epsilon_ {a} B _ {x 1} + \sqrt {2 \log (d)} d \gamma_ {\infty}\right) + | \Gamma | B _ {a 1} B _ {\epsilon} + \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}, B _ {G 1}} ^ {\infty}}. \tag {483}
$$
We finish the proof by union bound and  $\delta \geq \delta_{1} + \delta_{2} + \frac{1}{d^{2}}$ .
Lemma E.35 (Empirical Gradient Concentration Bound for Single Coordinate). For  $i \in [m]$ , when  $n \geq (\log(d))^6$ , with probability at least  $1 - O\left(\exp\left(-n^{\frac{1}{3}}\right)\right)$  over training samples, we have
$$
\left| \frac {\partial \widetilde {\mathcal {L}} _ {\mathcal {Z}} (f _ {\Xi})}{\partial \mathbf {w} _ {i , j}} - \frac {\partial \mathcal {L} _ {\mathcal {D}} (f _ {\Xi})}{\partial \mathbf {w} _ {i , j}} \right| \leq O \left(\frac {| \mathbf {a} _ {i} | B _ {x \infty}}{n ^ {\frac {1}{3}}}\right), \quad \forall j \in [ d ]. \tag {484}
$$
Proof of Lemma E.35. First, we define,
$$
z _ {i, j} ^ {(l)} = \ell^ {\prime} \left(y ^ {(l)} f _ {\Xi} \left(\mathbf {x} ^ {(l)}\right)\right) y ^ {(l)} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i}, \mathbf {x} ^ {(l)} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} _ {j} ^ {(l)} \right] \tag {485}
$$
$$
- \mathbb {E} _ {(\mathbf {x}, y)} \left[ \ell^ {\prime} \left(y f _ {\Xi} (\mathbf {x})\right) y \left[ \sigma^ {\prime} \left(\langle \mathbf {w} _ {i}, \mathbf {x} \rangle - \mathbf {b} _ {i}\right) \right] \mathbf {x} _ {j} \right]. \tag {486}
$$
As  $|\ell'(z)| \leq 1, |y| \leq 1, |\sigma'(z)| \leq 1$ , we have  $z_{i,j}^{(l)}$  is zero-mean random variable with  $|z_{i,j}^{(l)}| \leq 2B_{x\infty}$  as well as  $\mathbb{E}\left[\left|z_{i,j}^{(l)}\right|_2^2\right] \leq 4B_{x\infty}^2$ . Then by Bernstein Inequality, for  $0 < z < 2B_{x\infty}$ , we have
$$
\begin{array}{l} \Pr \left(\left| \frac {\partial \widetilde {\mathcal {L}} _ {\mathcal {Z}} \left(f _ {\Xi}\right)}{\partial \mathbf {w} _ {i , j}} - \frac {\partial \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi}\right)}{\partial \mathbf {w} _ {i , j}} \right| \geq | \mathbf {a} _ {i} | z\right) = \Pr \left(\left| \frac {1}{n} \sum_ {l \in [ n ]} z _ {i, j} ^ {(l)} \right| \geq z\right) (487) \\ \leq \exp \left(- n \cdot \frac {z ^ {2}}{8 B _ {x \infty}}\right). (488) \\ \end{array}
$$
Thus, for some  $i \in [m]$ , when  $n \geq (\log(d))^6$ , with probability at least  $1 - O\left(\exp \Theta\left(-n^{\frac{1}{3}}\right)\right)$ , from a union bound over  $j \in [d]$ , we have, for  $\forall j \in [d]$ ,
$$
\left| \frac {\partial \widetilde {\mathcal {L}} _ {\mathcal {Z}} (f _ {\Xi})}{\partial \mathbf {w} _ {i , j}} - \frac {\partial \mathcal {L} _ {\mathcal {D}} (f _ {\Xi})}{\partial \mathbf {w} _ {i , j}} \right| \leq O \left(\frac {| \mathbf {a} _ {i} | B _ {x \infty}}{n ^ {\frac {1}{3}}}\right). \tag {489}
$$
Lemma E.36 (Existence of Good Networks under Empirical Risk. Modified version of Lemma D.13 Under Uniform Parity Setting). Suppose  $n > \Omega \left(\left(\frac{B_x}{\sqrt{B_{x2}}} + \log \frac{1}{p} + \frac{B_{x\infty}}{B_G|\ell'(0)|} + \frac{B_{x\infty}}{B_{G1}|\ell'(0)|}\right)^3 + (\log(d))^6\right)$ . Let  $\lambda^{(1)} = \frac{1}{\eta^{(1)}}$ . For any  $B_{\epsilon} \in (0, B_b)$ , let  $\sigma_a = \Theta\left(\frac{\tilde{b}}{-|\ell'(0)|\eta^{(1)}B_B B_{\epsilon}}\right)$  and  $\delta = 2re^{-\sqrt{\frac{mp}{2}}} + \frac{1}{d^2}$ . Then, with probability at least  $1 - \delta$  over the initialization and training samples, there exists  $\tilde{\mathbf{a}}_i$ 's such that  $f_{(\tilde{\mathbf{a}},\mathbf{W}^{(1)},\mathbf{b})}(\mathbf{x}) = \sum_{i=1}^{4m} \tilde{\mathbf{a}}_i \sigma\left(\left\langle \mathbf{w}_i^{(1)},\mathbf{x} \right\rangle - \mathbf{b}_i\right)$  satisfies
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \tag {490} \\ \leq r B _ {a 1} \left(\frac {2 B _ {x 1} B _ {G 1} B _ {b}}{\sqrt {m p} B _ {G} B _ {\epsilon}} + \sqrt {2 \log (d) d} \left(\gamma_ {\infty} + O \left(\frac {B _ {x \infty}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{3}}}\right)\right) + B _ {\epsilon}\right) + \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}, B _ {G 1}} ^ {\infty}}, \\ \end{array}
$$
$$
a n d \| \tilde {\mathbf {a}} \| _ {0} = O \left(r (m p) ^ {\frac {1}{2}}\right), \| \tilde {\mathbf {a}} \| _ {2} = O \left(\frac {B _ {a 2} B _ {b}}{\tilde {b} (m p) ^ {\frac {1}{4}}}\right), \| \tilde {\mathbf {a}} \| _ {\infty} = O \left(\frac {B _ {a 1} B _ {b}}{\tilde {b} (m p) ^ {\frac {1}{2}}}\right).
$$
Proof of Lemma E.36. Denote  $\rho = O\left(\exp \Theta \left(-n^{\frac{1}{3}}\right)\right)$  and  $\beta = O\left(\frac{B_{x\infty}}{n^{\frac{1}{3}}}\right)$ . Note that by symmetric initialization, we have  $\ell'(yf_{\Xi^{(0)}}(\mathbf{x})) = |\ell'(0)|$  for any  $\mathbf{x} \in \mathcal{X}$ , so that, by Lemma E.35, we have  $\left|\widetilde{G}(\mathbf{w}_i^{(0)}, \mathbf{b}_i)_j - G(\mathbf{w}_i^{(0)}, \mathbf{b}_i)_j\right| \leq \frac{\beta}{|\ell'(0)|}$  with probability at least  $1 - \rho$ . Thus, by union bound, we can see that  $S_{p,\gamma_{\infty}, B_G, B_{G1}}^{\infty} \subseteq \widetilde{S}_{p - \rho, \gamma_{\infty} + \frac{\beta}{B_G|\ell'(0)|}, B_G - \frac{\beta}{|\ell'(0)|}, B_{G1} + \frac{\beta}{|\ell'(0)|}}$ . Consequently, we have  $\mathrm{OPT}_{d,r,B_F,\widetilde{S}_{p - \rho,\gamma_{\infty} + \frac{\beta}{B_G|\ell'(0)|}, B_G - \frac{\beta}{|\ell'(0)|}, B_{G1} + \frac{\beta}{|\ell'(0)|}} \leq \mathrm{OPT}_{d,r,B_F,S_{p,\gamma_{\infty}, B_G, B_{G1}}^{\infty}}$ . Exactly follow the proof in Lemma D.4 by replacing  $S_{p,\gamma_{\infty}, B_G, B_{G1}}^{\infty}$  to  $\widetilde{S}_{p - \rho,\gamma_{\infty} + \frac{\beta}{B_G|\ell'(0)|}, B_G - \frac{\beta}{|\ell'(0)|}, B_{G1} + \frac{\beta}{|\ell'(0)|}}$ . Then, we finish the proof by  $\rho \leq \frac{p}{2}$ ,  $\frac{\beta}{|\ell'(0)|} \leq (1 - 1/\sqrt{2})B_G$ ,  $\frac{\beta}{|\ell'(0)|} \leq (\sqrt{2} - 1)B_{G1}$ .
Theorem E.37 (Online Convex Optimization under Empirical Risk. Modified version of Theorem D.17 Under Uniform Parity Setting). Consider training by Algorithm 1, and any  $\delta \in (0,1)$ . Assume  $d \geq \log m$ ,  $\delta \leq O\left(\frac{1}{d^2}\right)$ . Set
$$
\begin{array}{l} L e t \quad 0 \qquad <   \qquad T \eta B _ {x 1} \qquad \leq \qquad o (1), \quad m \qquad = \qquad \Omega \left(\frac {1}{\sqrt {\delta}} + \frac {1}{p} \left(\log \left(\frac {r}{\delta}\right)\right) ^ {2}\right) \quad a n d \quad n \qquad > \\ \Omega \left(\left(\frac {B _ {x}}{\sqrt {B _ {x 2}}} + \log \frac {T m}{p \delta} + \left(1 + \frac {1}{B _ {G}} + \frac {1}{B _ {G 1}}\right) \frac {B _ {x \infty}}{| \ell^ {\prime} (0) |}\right) ^ {3}\right). \quad W i t h p r o b a b i l i t y a t l e a s t 1 - \delta o v e r t h e r e f e r e n c e. \end{array}
$$
initialization and training samples, there exists  $t \in [T]$  such that
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) (491) \\ \leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} \left(\frac {2 \sqrt {2} \sqrt {B _ {x 1} B _ {G 1}}}{(m p) ^ {\frac {1}{4}}} \sqrt {\frac {B _ {b}}{B _ {G}}} + \sqrt {2 \log (d) d} \left(\gamma_ {\infty} + O \left(\frac {B _ {x \infty}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{3}}}\right)\right)\right) \\ + \eta \left(\sqrt {r} B _ {a 2} B _ {b} T \eta B _ {x 1} ^ {2} + m \tilde {b}\right) O \left(\frac {\sqrt {\log m} B _ {x 1} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + 1\right) + O \left(\frac {B _ {a 2} ^ {2} B _ {b} ^ {2}}{\eta T \tilde {b} ^ {2} (m p) ^ {\frac {1}{2}}}\right) (492) \\ + \frac {1}{n ^ {\frac {1}{3}}} O \left( \right.\left(\frac {r B _ {a 1} B _ {b}}{\tilde {b}} + m \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + \frac {\tilde {b}}{B _ {x 1}}\right)\right) (493) \\ \left. \cdot \left(\left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + T \eta^ {2} B _ {x 1} \tilde {b}\right) B _ {x} + \tilde {b}\right) + 2\right) (494) \\ + \frac {1}{n ^ {\frac {1}{3}}} O \left(m \eta \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + T \eta^ {2} B _ {x 1} \tilde {b}\right) \sqrt {B _ {x 2}}\right). (495) \\ \end{array}
$$
$$
\tilde {b} = \Theta \left(\frac {B _ {G} ^ {\frac {1}{4}} B _ {a 2} B _ {b} ^ {\frac {3}{4}}}{\sqrt {r B _ {a 1}}}\right), \quad m = \Omega \left(\frac {1}{p \epsilon^ {4}} \left(r B _ {a 1} \sqrt {B _ {x 1} B _ {G 1}} \sqrt {\frac {B _ {b}}{B _ {G}}}\right) ^ {4} + \frac {1}{\sqrt {\delta}} + \frac {1}{p} \left(\log \left(\frac {r}{\delta}\right)\right) ^ {2}\right),
$$
$$
n = \Omega \left(\left(\frac {m B _ {x} B _ {a 2} ^ {2} \sqrt {B _ {b}} (m p) ^ {\frac {1}{2}} \log m}{\epsilon r B _ {a 1} \sqrt {B _ {G}}}\right) ^ {3} + \left(\frac {B _ {x}}{\sqrt {B _ {x 2}}} + \log \frac {T m}{p \delta} + (1 + \frac {1}{B _ {G}} + \frac {1}{B _ {G 1}}) \frac {B _ {x \infty}}{| \ell^ {\prime} (0) |}\right) ^ {3}\right),
$$
$$
\Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}}\right) (\mathbf {x}) \neq y \right] \leq \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) \tag {496}
$$
$$
\leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma_ {\infty}, B _ {G}, B _ {G 1}} ^ {\infty}} + r B _ {a 1} \sqrt {2 \log (d) d} \left(\gamma_ {\infty} + O \left(\frac {B _ {x \infty}}{B _ {G} \left| \ell^ {\prime} (0) \right| n ^ {\frac {1}{3}}}\right)\right) + \epsilon . \tag {497}
$$
Proof of Theorem E.37. Proof of the theorem and parameter choices remain the same as Theorem D.17 except for setting  $B_{\epsilon} = \frac{\sqrt{B_{x1}B_{G1}}}{(mp)^{\frac{1}{4}}}\sqrt{\frac{B_b}{B_G}}$  and apply Lemma E.36.
E.6.2 Feature Learning of Uniform Parity Functions
$$
g _ {i, j} = \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i} \right] \mathbf {x} _ {j} \right] \tag {498}
$$
$$
\xi_ {k} = (- 1) ^ {\frac {k - 1}{2}} \frac {\binom {n - 1} {\frac {2}{k - 1}}}{\binom {n - 1} {k - 1}} \cdot 2 ^ {- (n - 1)} \binom {n - 1} {\frac {n - 1}{2}}. \tag {499}
$$
Lemma E.38 (Uniform Parity Functions: Gradient Feature Learning. Corollary of Lemma 3 in [15]). Assume that  $n \geq 2(k + 1)^2$ . Then, the following holds:
If  $j\in A$  ,then
$$
g _ {i, j} = \xi_ {k - 1} \prod_ {l \in A \backslash \{j \}} \left(\mathbf {w} _ {i, l} ^ {(0)}\right). \tag {500}
$$
If  $i\notin A$  ,then
$$
g _ {i, j} = \xi_ {k - 1} \prod_ {l \in A \cup \{j \}} \left(\mathbf {w} _ {i, l} ^ {(0)}\right). \tag {501}
$$
Lemma E.39 (Uniform Parity Functions: Existence of Good Networks (Alternative)). Assume the same condition as in Lemma E.38. Define
$$
D = \frac {\sum_ {l \in A} \mathbf {M} _ {l}}{\| \sum_ {l \in A} \mathbf {M} _ {l} \| _ {2}} \tag {502}
$$