SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr_ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ \left| \sum_ {l \in A} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle \right| \geq t \right] = \Pr_ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ \left| \sum_ {l \in A} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \right\rangle \phi_ {l} \right| \geq t \right] (408) \\ \leq 2 \exp \left(- \frac {t ^ {2}}{2 \sum_ {l \in A} \left\langle \mathbf {w} _ {i} ^ {(0)} , \mathbf {M} _ {l} \right\rangle^ {2}}\right) (409) \\ \leq \exp \left(- \frac {t ^ {2}}{\Theta (r k \sigma_ {w} ^ {2})}\right). (410) \\ \end{array}
$$
In the proof of the third to the last statement, we need  $\tilde{b} = C_b\sqrt{\tau rk\log d}\sigma_w$ , where  $C_b$  is a large enough universal constant.
For the third statement, when  $\mathbf{x} \sim \mathcal{X}_U$ , by union bound and previous statements,
$$
\begin{array}{l} \Pr_ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ \sum_ {l \in [ d ]} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle - \mathbf {b} _ {i} \geq 0 \right] (411) \\ \leq \Pr_ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ \sum_ {l \in A} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle \geq \frac {\mathbf {b} _ {i}}{2} \right] + \Pr_ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ \sum_ {l \in A ^ {\perp}} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle \geq \frac {\mathbf {b} _ {i}}{2} \right] (412) \\ \leq O \left(\frac {1}{d ^ {\tau}}\right). (413) \\ \end{array}
$$
For the forth statement, when  $\mathbf{x} \sim \mathcal{X}_{j, + }$ , by  $C_{\text{Sure}, 1} \geq \frac{3}{2}$  and previous statements,
$$
\begin{array}{l} \Pr_ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \sum_ {l \in [ d ]} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle - \mathbf {b} _ {i} \geq 0 \right] (414) \\ = \Pr_ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \sum_ {l \in A _ {j}} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle + \sum_ {l \in A \backslash A _ {j}} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle + \sum_ {l \in A ^ {\perp}} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle \geq \mathbf {b} _ {i} \right] (415) \\ \geq \Pr_ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \sum_ {l \in A ^ {\perp}} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle \geq \left(1 - C _ {S u r e, 1}\right) \mathbf {b} _ {i} \right] (416) \\ \geq \Pr_ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \sum_ {l \in A ^ {\perp}} \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {M} _ {l} \phi_ {l} \right\rangle \geq - \frac {\mathbf {b} _ {i}}{2} \right] (417) \\ \geq 1 - O \left(\frac {1}{d ^ {\tau}}\right). (418) \\ \end{array}
$$
For the last statement, we prove similarly by  $0 < C_{\text{Sure},2} \leq \frac{1}{2}$ .
Then, Lemma E.26 gives gradients of neurons in  $S_{D_j, \text{Sure}}$ . It shows that these gradients are highly aligned with  $D_j$ .
Lemma E.26 (Parity Functions: Feature Emergence). Assume the same conditions as in Lemma E.23, for all  $j \in [r]$ ,  $i \in S_{D_j, \text{Sure}}$ , we have the following holds:
(1) For all  $l \in A_j$ , we have
$$
p _ {A} - O \left(\frac {1}{d ^ {\tau}}\right) \leq \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \leq p _ {A} + O \left(\frac {1}{d ^ {\tau}}\right). \tag {419}
$$
(2) For all  $l \in A_j$ , any  $j' \neq j, j' \in [r]$ , we have
$$
\left| \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \right| \leq O \left(\frac {1}{d ^ {\tau}}\right). \tag {420}
$$
(3) For all  $l \in A^{\perp}$ , we have
$$
\left| \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \right| \leq O \left(\frac {1}{d ^ {\tau}}\right). \tag {421}
$$
Proof of Lemma E.26. For all  $l \in [d]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] (422) \\ = p _ {A} \sum_ {l \in [ r ]} \left(\mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {l, +}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {l, -}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right]\right) (423) \\ + (1 - 2 r p _ {A}) \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right]. (424) \\ \end{array}
$$
For the first statement, for all  $l \in A_j$ , by Lemma E.25 (3) and (4), we have
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] (425) \\ = p _ {A} \left(\mathbb {E} _ {\mathbf {x} \sim \chi_ {j, +}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \right] + \mathbb {E} _ {\mathbf {x} \sim \chi_ {j, -}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \right]\right) (426) \\ + (1 - 2 r p _ {A}) \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] (427) \\ \geq p _ {A} \left(1 - O \left(\frac {1}{d ^ {\tau}}\right)\right) - O \left(\frac {1}{d ^ {\tau}}\right) (428) \\ \geq p _ {A} - O \left(\frac {1}{d ^ {\tau}}\right), (429) \\ \end{array}
$$
and we also have
$$
\mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \tag {430}
$$
$$
= p _ {A} \left(\mathbb {E} _ {\mathbf {x} \sim \chi_ {j, +}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \right] + \mathbb {E} _ {\mathbf {x} \sim \chi_ {j, -}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \right]\right) \tag {431}
$$
$$
+ (1 - 2 r p _ {A}) \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {U}} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \tag {432}
$$
$$
\leq p _ {A} + O \left(\frac {1}{d ^ {\tau}}\right). \tag {433}
$$
Similarly, for the second statement, for all  $l \in A_{j'}$ , any  $j' \neq j, j' \in [r]$ , by Lemma E.25 (3) and (5), we have
$$
\left| \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \right| \tag {434}
$$
$$
\leq \left| p _ {A} \left(\mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j ^ {\prime}, +}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \right] + \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j ^ {\prime}, -}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \right]\right) \right| + O \left(\frac {1}{d ^ {\tau}}\right)
$$
$$
\leq O \left(\frac {1}{d ^ {\tau}}\right). \tag {435}
$$
For the third statement, for all  $l \in A^{\perp}$ , by Lemma E.25 (3), (4), (5), we have
$$
\left| \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \right| \tag {436}
$$
$$
\leq p _ {A} \sum_ {l \in [ r ]} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {l, +}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {l, -}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \right| + O \left(\frac {1}{d ^ {\tau}}\right)
$$
$$
\leq p _ {A} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, -}} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \phi_ {l} \right] \right| + O \left(\frac {1}{d ^ {\tau}}\right)
$$
$$
\leq p _ {A} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \left(1 - \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right)\right) \phi_ {l} \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, -}} \left[ \left(1 - \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right)\right) \phi_ {l} \right] \right|
$$
$$
+ p _ {A} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, +}} [ \phi_ {l} ] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, -}} [ \phi_ {l} ] \right| + O \left(\frac {1}{d ^ {\tau}}\right) \tag {437}
$$
$$
= p _ {A} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, +}} \left[ \left(1 - \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right)\right) \phi_ {l} \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {X} _ {j, -}} \left[ \left(1 - \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right)\right) \phi_ {l} \right] \right|
$$
$$
+ O \left(\frac {1}{d ^ {\tau}}\right) \tag {438}
$$
$$
\leq O \left(\frac {1}{d ^ {\tau}}\right), \tag {439}
$$
where the second inequality follows  $2rp_{A} \leq 1$  and the third inequality follows the triangle inequality.
E.4.3 Parity Functions: Final Guarantee
Lemma E.27 (Parity Functions: Existence of Good Networks. Part statement of Lemma 4.7). Assume the same conditions as in Lemma E.23. Define
$$
f ^ {*} (\mathbf {x}) = \sum_ {j = 1} ^ {r} \sum_ {i = 0} ^ {k} (- 1) ^ {i + 1} \sqrt {k} \tag {440}
$$
$$
\cdot \left[ \sigma \left(\langle D _ {j}, \mathbf {x} \rangle - \frac {2 i - k - 1}{\sqrt {k}}\right) - 2 \sigma \left(\langle D _ {j}, \mathbf {x} \rangle - \frac {2 i - k}{\sqrt {k}}\right) + \sigma \left(\langle D _ {j}, \mathbf {x} \rangle - \frac {2 i - k + 1}{\sqrt {k}}\right) \right].
$$
For  $\mathcal{D}_{\text{parity}}$  setting, we have  $f^{*} \in \mathcal{F}_{d,3r(k + 1),B_{F},S_{p,\gamma},B_{G}}$ , where  $B_{F} = (B_{a1},B_{a2},B_{b}) =$
$$
\left(2 \sqrt {k}, 2 \sqrt {r k (k + 1)}, \frac {k + 1}{\sqrt {k}}\right), p = \Theta \left(\frac {1}{\sqrt {\tau r \log d} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau r / 8\right)}}\right), \gamma = \frac {1}{d ^ {\tau - 2}}, B _ {G} = \sqrt {k} p _ {A} - O \left(\frac {\sqrt {k}}{d ^ {\tau}}\right)
$$
and  $B_{x1} = \sqrt{d}, B_{x2} = d$ . We also have  $\mathrm{OPT}_{d,3r(k + 1),B_F,S_p,\gamma,B_G} = 0$ .
Proof of Lemma E.27. We can check  $B_{x1} = \sqrt{d}, B_{x2} = d$  by direct calculation. By Lemma E.23, we have  $f^{*} \in \mathcal{F}_{d,3r(k + 1),B_{F},S_{p,\gamma ,B_{G}}}$ . We note that
$$
\sigma \left(\langle D _ {j}, \mathbf {x} \rangle - \frac {2 i - k - 1}{\sqrt {k}}\right) - 2 \sigma \left(\langle D _ {j}, \mathbf {x} \rangle - \frac {2 i - k}{\sqrt {k}}\right) + \sigma \left(\langle D _ {j}, \mathbf {x} \rangle - \frac {2 i - k + 1}{\sqrt {k}}\right) \tag {441}
$$
is a bump function for  $\langle D_j, \mathbf{x} \rangle$  at  $\frac{2i - k}{\sqrt{k}}$ . We can check that  $yf^{*}(\mathbf{x}) \geq 1$ . Thus, we have
$$
\begin{array}{l} \mathrm {O P T} _ {d, 3 r (k + 1), B _ {F}, S _ {p, \gamma , B _ {G}}} \leq \mathcal {L} _ {\mathcal {D} _ {\text {p a r i t y}}} \left(f ^ {*}\right) (442) \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {p a r i t y}}} \mathcal {L} _ {(\mathbf {x}, y)} \left(f ^ {*}\right) (443) \\ = 0. (444) \\ \end{array}
$$
Theorem 4.8 (Parity Functions: Main Result). Assume Assumption 4.5. For any  $\epsilon, \delta \in (0,1)$ , when Algorithm 1 uses hinge loss with
$$
\Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}} (\mathbf {x})\right) \neq y \right] \leq \frac {3 r \sqrt {k}}{d ^ {(\tau - 3) / 2}} + \epsilon .
$$
Proof of Theorem 4.8. Let  $\tilde{b} = C_b\sqrt{\tau r k \log d}\sigma_w$ , where  $C_b$  is a large enough universal constant. By Lemma E.27, we have  $f^* \in \mathcal{F}_{d,3r(k + 1),B_F,S_{p,\gamma},B_G}$ , where  $B_F = (B_{a1},B_{a2},B_b) =$
$$
\left(2 \sqrt {k}, 2 \sqrt {r k (k + 1)}, \frac {k + 1}{\sqrt {k}}\right), p = \Theta \left(\frac {1}{\sqrt {\tau r \log d \cdot d ^ {(9 C _ {b} ^ {2} \tau r / 8)}}}\right), \gamma = \frac {1}{d ^ {\tau - 2}}, B _ {G} = \sqrt {k} p _ {A} - O \left(\frac {\sqrt {k}}{d ^ {\tau}}\right)
$$
and  $B_{x1} = \sqrt{d}, B_{x2} = d$ . We also have  $\mathrm{OPT}_{d,3r(k + 1),B_F,S_{p,\gamma},B_G} = 0$ .
Adjust  $\sigma_w$  such that  $\tilde{b} = C_b\sqrt{\tau rk\log d}\sigma_w = \Theta \left(\frac{B_G^{\frac{1}{4}}B_{a2}B_b^{\frac{3}{4}}}{\sqrt{rB_{a1}}}\right)$ . Injecting above parameters into Theorem 3.12, we have with probability at least  $1 - \delta$  over the initialization, with proper hyperparameters, there exists  $t\in [T]$  such that
$$
\Pr [ \mathrm {s i g n} (f _ {\Xi^ {(t)}} (\mathbf {x})) \neq y ] \leq \frac {2 \sqrt {2} r \sqrt {k}}{d ^ {(\tau - 3) / 2}} + O \left(\frac {r B _ {a 1} B _ {x 1} B _ {x 2} ^ {\frac {1}{4}} (\log n) ^ {\frac {1}{4}}}{\sqrt {B _ {G}} n ^ {\frac {1}{4}}}\right) + \epsilon / 2 \leq \frac {3 r \sqrt {k}}{d ^ {(\tau - 3) / 2}} + \epsilon .
$$
E.5 Uniform Parity Functions
We consider the sparse parity problem over the uniform data distribution studied in [15]. We use the properties of the problem to prove the key lemma (i.e., the existence of good networks) in our framework and then derive the final guarantee from our theorem of the simple setting (Theorem 3.4). We provide Theorem E.31 as (1) use it as a warm-up and (2) follow the original analysis in [15] to give a comparison. We will provide Theorem E.40 as an alternative version that trains both layers.
Consider the same data distribution in Appendix E.4.1 and Definition E.22 with the following assumptions.
Assumption E.28 (Uniform Parity Functions). We follow the data distribution in Appendix E.4.1. Let  $r = 1, p_A = 0, p_o = \frac{1}{2}$ ,  $\mathbf{M} = I_{d \times d}$  and  $d \geq 2k^2$ , and  $k$  is an even number.
We denote this data distribution as  $\mathcal{D}_{\text{parity-uniform}}$  setting.
To apply our framework, again we only need to compute the parameters in the Gradient Feature set and the corresponding optimal approximation loss. To this end, we first define the gradient features: let
$$
D = \frac {\sum_ {l \in A} \mathbf {M} _ {l}}{\| \sum_ {l \in A} \mathbf {M} _ {l} \| _ {2}}. \tag {445}
$$
We follow the initialization and training dynamic in [15].
Initialization and Loss. We use hinge loss and we have unbiased initialization, for all  $i \in [m]$ ,
$$
\mathbf {a} _ {i} ^ {(0)} \sim \operatorname {U n i f} (\{\pm 1 \}), \mathbf {w} _ {i} ^ {(0)} \sim \operatorname {U n i f} (\{\pm 1 \} ^ {d}), \mathbf {b} _ {i} = \operatorname {U n i f} (\{- 1 + 1 / k, \dots , 1 - 1 / k \}). \tag {446}
$$
Training Process. We use the following one-step training algorithm for this specific data distribution.
Algorithm 4 Network Training via Gradient Descent [15]. Special case of Algorithm 2
Initialize  $(\mathbf{a}^{(0)},\mathbf{W}^{(0)},\mathbf{b})$  as in Equation (8) and Equation (446); Sample  $\mathcal{Z}\sim \mathcal{D}_{\text{parity-uniform}}^n$ $\mathbf{W}^{(1)} = \mathbf{W}^{(0)} - \eta^{(1)}(\nabla_{\mathbf{W}}\widetilde{\mathcal{L}}_{\mathcal{Z}}(f_{\Xi^{(0)}}) + \lambda^{(1)}\mathbf{W}^{(0)})$ $\mathbf{a}^{(1)} = \mathbf{a}^{(0)} - \eta^{(1)}(\nabla_{\mathbf{a}}\widetilde{\mathcal{L}}_{\mathcal{Z}}(f_{\Xi^{(0)}}) + \lambda^{(1)}\mathbf{a}^{(0)})$   
for  $t = 2$  to  $T$  do  
 $\mathbf{a}^{(t)} = \mathbf{a}^{(t - 1)} - \eta^{(t)}\nabla_{\mathbf{a}}\widetilde{\mathcal{L}}_{\mathcal{Z}}(f_{\Xi^{(t - 1)}})$   
end for
Use the notation in Section 5.3 of [92], for every  $S \in [n]$ , s.t.  $|S| = k$ , we define
$$
\xi_ {k} := \widehat {\mathbf {M a j}} (S) = (- 1) ^ {\frac {k - 1}{2}} \frac {\binom {\frac {d - 1}{2}} {\frac {d - 1}{2}}}{\binom {d - 1} {k - 1}} \cdot 2 ^ {- (d - 1)} \binom {d - 1} {\frac {d - 1}{2}}. \tag {447}
$$
Lemma E.29 (Uniform Parity Functions: Existence of Good Networks. Rephrase of Lemma 5 in [15]). For every  $\epsilon, \delta \in (0,1/2)$ , denoting  $\tau = \frac{|\xi_{k-1}|}{16k\sqrt{2d\log(32k^3d/\epsilon)}}$ , let  $\eta^{(1)} = \frac{1}{k|\xi_{k-1}|}$ ,  $\lambda^{(1)} = \frac{1}{\eta^{(1)}}$ ,  $m \geq k \cdot 2^k\log(k/\delta)$ ,  $n \geq \frac{2}{\tau^2}\log(4dm/\delta)$  and  $d \geq \Omega$  ( $k^4\log(kd/\epsilon)$ ), w.p. at least  $1 - 2\delta$  over the initialization and the training samples, there exists  $\tilde{\mathbf{a}} \in \mathbb{R}^m$  with  $\|\tilde{\mathbf{a}}\|_{\infty} \leq 8k$  and  $\|\tilde{\mathbf{a}}\|_2 \leq 8k\sqrt{k}$  such that  $f_{(\tilde{\mathbf{a}},\mathbf{W}^{(1)},\mathbf{b})}$  satisfies
$$
\mathcal {L} _ {\mathcal {D} _ {\text {p a r i t y - u n i f o r m}}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \leq \epsilon . \tag {448}
$$
Additionally, it holds that  $\| \sigma (\mathbf{W}^{(1)}^{\top}\mathbf{x} - \mathbf{b})\|_{\infty}\leq d + 1$
Remark E.30. In [15], they update the bias term in the first gradient step. However, if we check the proof carefully, we can see that the fixed bias still goes through all their analysis.
E.5.1 Uniform Parity Functions: Final Guarantee
Considering training by Algorithm 4, we have the following results.
Theorem E.31 (Uniform Parity Functions: Main Result). Fix  $\epsilon \in (0,1/2)$  and let  $m \geq \Omega(k \cdot 2^k \log(k/\epsilon))$ ,  $n \geq \Omega\left(k^{7/6} d\binom{d}{k-1} \log(kd/\epsilon) \log(dm/\epsilon) + \frac{k^3 md^2}{\epsilon^2}\right)$ ,  $d \geq \Omega\left(k^4 \log(kd/\epsilon)\right)$ . Let  $\eta^{(1)} = \frac{1}{k|\xi_{k-1}|}$ ,  $\lambda^{(1)} = \frac{1}{\eta^{(1)}}$ , and  $\eta = \eta^{(t)} = \Theta\left(\frac{1}{d^2 m}\right)$ , for all  $t \in \{2,3,\ldots,T\}$ . If  $T \geq \Omega\left(\frac{k^3 md^2}{\epsilon}\right)$ , then training by Algorithm 4 with hinge loss, w.h.p. over the initialization and the training samples, there exists  $t \in [T]$  such that
$$
\Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}}\right) (\mathbf {x}) \neq y \right] \leq \mathcal {L} _ {\mathcal {D} _ {\text {p a r i t y - u n i f o r m}}} f _ {\left(\mathbf {a} ^ {(t)}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)} \leq \epsilon . \tag {449}
$$
Proof of Theorem E.31. By Lemma E.29, w.h.p., we have for properly chosen hyper-parameters,
$$
\operatorname {O P T} _ {\mathbf {W} ^ {(1)}, \mathbf {b}, B _ {a 2}} \leq \mathcal {L} _ {\mathcal {D} _ {\text {p a r i t y - u n i f o r m}}} \left(f _ {(\hat {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \leq \frac {\epsilon}{3}. \tag {450}
$$
We compute the  $L$ -smooth constant of  $\widetilde{\mathcal{L}}_{\Xi}\left(f_{(\mathbf{a},\mathbf{W}^{(1)},\mathbf{b})}\right)$  to a.
$$
\begin{array}{l} \left\| \nabla_ {\mathbf {a}} \widetilde {\mathcal {L}} _ {\mathcal {Z}} \left(f _ {\left(\mathbf {a} _ {1}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)}\right) - \nabla_ {\mathbf {a}} \widetilde {\mathcal {L}} _ {\mathcal {Z}} \left(f _ {\left(\mathbf {a} _ {2}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)}\right) \right\| _ {2} (451) \\ = \left\| \frac {1}{n} \sum_ {\mathbf {x} \in \mathcal {Z}} \left[ \left(\ell^ {\prime} \left(y f _ {\left(\mathbf {a} _ {1}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)} (\mathbf {x})\right) - \ell^ {\prime} \left(y f _ {\left(\mathbf {a} _ {2}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)} (\mathbf {x})\right)\right) \sigma \left(\mathbf {W} ^ {(1) ^ {\top}} \mathbf {x} - \mathbf {b}\right) \right] \right\| _ {2} (452) \\ \leq \left\| \frac {1}{n} \sum_ {\mathbf {x} \in \mathcal {Z}} \left[ \left| f _ {\left(\mathbf {a} _ {1}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)} (\mathbf {x}) - f _ {\left(\mathbf {a} _ {2}, \mathbf {W} ^ {(1)}, \mathbf {b}\right)} (\mathbf {x}) \right| \sigma \left(\mathbf {W} ^ {(1) \top} \mathbf {x} - \mathbf {b}\right) \right] \right\| _ {2} (453) \\ \leq \frac {1}{n} \sum_ {\mathbf {x} \in \mathcal {Z}} \left[ \| \mathbf {a} _ {1} - \mathbf {a} _ {2} \| _ {2} \left\| \sigma \left(\mathbf {W} ^ {(1) \top} \mathbf {x} - \mathbf {b}\right) \right\| _ {2} ^ {2} \right]. (454) \\ \end{array}
$$
By the Lemma E.29, we have  $\| \sigma (\mathbf{W}^{(1)}^{\top}\mathbf{x} - \mathbf{b})\|_{\infty}\leq d + 1$ . Thus, we have,
$$
\begin{array}{l} L = O \left(\frac {1}{n} \sum_ {\mathbf {x} \in \mathcal {Z}} \left\| \sigma \left(\mathbf {W} ^ {(1) \top} \mathbf {x} - \mathbf {b}\right) \right\| _ {2} ^ {2}\right) (455) \\ \leq O \left(d ^ {2} m\right). (456) \\ \end{array}
$$
This means that we can let  $\eta = \Theta\left(\frac{1}{d^2m}\right)$  and we will get our convergence result. Note that we have  $\mathbf{a}^{(1)} = \mathbf{0}$  and  $\| \tilde{\mathbf{a}} \|_2 = O\left(k\sqrt{k}\right)$ . So, if we choose  $T \geq \Omega\left(\frac{k^3}{\epsilon\eta}\right)$ , there exists  $t \in [T]$  such that  $\widetilde{\mathcal{L}}_{\mathcal{Z}}\left(f_{(\mathbf{a}^{(t)},\mathbf{W}^{(1)},\mathbf{b})}\right) - \widetilde{\mathcal{L}}_{\mathcal{Z}}\left(f_{(\tilde{\mathbf{a}},\mathbf{W}^{(1)},\mathbf{b})}\right) \leq O\left(\frac{L\|\mathbf{a}^{(1)} - \tilde{\mathbf{a}}\|_2^2}{T}\right) \leq \epsilon / 3$ .
We also have  $\sqrt{\frac{\|\tilde{\mathbf{a}}\|_2^2(\|\mathbf{W}^{(1)}\|_F^2B_x^2 + \|\mathbf{b}\|_2^2)}{n}} \leq \frac{\epsilon}{3}$ . Then our theorem gets proved by Theorem 3.4.
E.6 Uniform Parity Functions: Alternative Analysis
It is also possible to unify [15] into our general Gradient Feature Learning Framework by mildly modifying the framework in Theorem 3.12. In order to do that, we first need to use a different metric in the definition of gradient features.
E.6.1 Modified General Feature Learning Framework for Uniform Parity Functions
Definition E.32 (Gradient Feature with Infinity Norm). For a unit vector  $D \in \mathbb{R}^d$  with  $\| D \|_2 = 1$ , and a  $\gamma_\infty \in (0,1)$ , a direction neighborhood (cone)  $\mathcal{C}_{D,\gamma_\infty}^\infty$  is defined as:  $\mathcal{C}_{D,\gamma_\infty}^\infty := \left\{\mathbf{w} \Big| \left\| \frac{\mathbf{w}}{\|\mathbf{w}\|} - D \right\|_\infty < \gamma_\infty \right\}$ . Let  $\mathbf{w} \in \mathbb{R}^d$ ,  $b \in \mathbb{R}$  be random variables drawn from some distribution  $\mathcal{W}$ ,  $\mathcal{B}$ . A Gradient Feature set with parameters  $p, \gamma_\infty, B_G, B_{G1}$  is defined as:
$$
S_{p,\gamma_{\infty},B_{G},B_{G1}}^{\infty}(\mathcal{W},\mathcal{B}):= \Bigg\{(D,s)\bigg|\Pr_{\mathbf{w},b}\left[G(\mathbf{w},b)\in \mathcal{C}_{D,\gamma_{\infty}}^{\infty}  ,  B_{G1}\geq \| G(\mathbf{w},b)\|_{2}\geq B_{G}  ,  s = \frac{b}{|b|}\right]\geq p\Bigg\}.
$$
When clear from context, write it as  $S_{p,\gamma_\infty,B_G,B_{G1}}^\infty$ .
Definition E.33 (Optimal Approximation via Gradient Features with Infinity Norm). The Optimal Approximation network and loss using gradient feature induced networks  $\mathcal{F}_{d,r,B_F,S_{p,\gamma_\infty,B_G,B_{G1}}^\infty}$  are defined as:
$$
f ^ {*} := \quad \operatorname {a r g m i n} \quad \mathcal {L} _ {\mathcal {D}} (f), \tag {457}
$$
$$
f \in \mathcal {F} _ {d, r, B _ {F}, S _ {p, \gamma_ {\infty}}, B _ {G}} ^ {\infty}
$$