SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} p _ {r - 1} = \Pr \left[ \hat {\zeta} _ {r _ {2}} \left| \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}}, \hat {\zeta} _ {r _ {1} + 1}, \dots , \hat {\zeta} _ {r _ {2}} \right. \right] \geq \Pr \left[ \hat {\zeta} _ {r _ {2}} \right] \geq \Pr \left[ \hat {\zeta} _ {r _ {2}} \right] (257) \\ \geq 1 - \Theta \left(\frac {\| \tilde {\mu} _ {r - 1} \| _ {2}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / (8 \| \tilde {\mu} _ {r _ {2}} \| _ {2} ^ {2})\right)}}\right). (258) \\ \end{array}
$$
On the other hand, if  $X$  is a  $\chi^2(k)$  random variable. Then we have
$$
\Pr (X \geq k + 2 \sqrt {k x} + 2 x) \leq e ^ {- x}. \tag {259}
$$
Therefore, by assumption  $B_{\mu 1}\geq \Omega \left(\sigma_{B + }\sqrt{\frac{\tau\log d}{d}}\right)$  , we have
$$
\begin{array}{l} \Pr \left(\frac {1}{\sigma_ {w} ^ {2}} \left\| \mathbf {w} _ {i} ^ {(0)} \right\| _ {2} ^ {2} \geq d + 2 \sqrt {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right) + 2\right) d \log d}\right) (260) \\ \left. + 2 \left(9 C _ {b} ^ {2} \tau \sigma_ {B +} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right) + 2\right) \log d\right) (261) \\ \leq O \left(\frac {1}{d ^ {2} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}}\right). (262) \\ \end{array}
$$
Recall  $B_{\mu 1} = \min_{j\in [r]}\| \tilde{\mu}_j\| _2$ ,  $B_{\mu 2} = \max_{j\in [r]}\| \tilde{\mu}_j\| _2$ . Thus, by union bound, we have
$$
\begin{array}{l} \Pr \left[ i \in S _ {D _ {j}, S u r e} \right] (263) \\ \geq \Pi_ {l \in [ r ]} p _ {l} - O \left(\frac {1}{d ^ {2} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}}\right) (264) \\ \geq \Theta \left(\frac {B _ {\mu 1}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}} \cdot \left(1 - \frac {r B _ {\mu 2}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(8 B _ {\mu 2} ^ {2}\right)\right)}}\right)\right) (265) \\ - O \left(\frac {1}{d ^ {2} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}}\right) (266) \\ \geq \Theta \left(\frac {B _ {\mu 1}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}}\right). (267) \\ \end{array}
$$
In Lemma E.9, we compute the activation pattern for the neurons in  $S_{D_j, Sure}$ .
Lemma E.9 (Mixture of Gaussians: Activation Pattern). Assume the same conditions as in Lemma E.7, for all  $j \in [r]$ ,  $i \in S_{D_j, S_{\text{ure}}}$ , we have
(1) When  $\mathbf{x} \sim \mathcal{N}_j(\mu_j, \sigma_j I_{d \times d})$ , the activation probability satisfies,
$$
\Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i} \geq 0 \right] \geq 1 - O \left(\frac {1}{d ^ {\tau}}\right). \tag {268}
$$
(2) For all  $j^{\prime}\neq j,j^{\prime}\in [r]$ , when  $\mathbf{x}\sim \mathcal{N}_{j^{\prime}}(\mu_{j^{\prime}},\Sigma_{j^{\prime}})$ , the activation probability satisfies,
$$
\Pr_ {\mathbf {x} \sim \mathcal {N} _ {j ^ {\prime}} \left(\mu_ {j ^ {\prime}}, \sigma_ {j ^ {\prime}} I _ {d \times d}\right)} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i} \geq 0 \right] \leq O \left(\frac {1}{d ^ {\tau}}\right). \tag {269}
$$
Proof of Lemma E.9. In the proof, we need  $\tilde{b} = C_b\sqrt{\tau d\log d}\sigma_w\sigma_{B+}$ , where  $C_b$  is a large enough universal constant. For the first statement, when  $\mathbf{x} \sim \mathcal{N}_j(\mu_j,\sigma_jI_{d\times d})$ , by  $C_{Sure,1} \geq \frac{3}{2}$ , we have
$$
\begin{array}{l} \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i} \geq 0 \right] \geq \Pr_ {\mathbf {x} \sim \mathcal {N} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle \geq (1 - C _ {S u r e, 1}) \mathbf {b} _ {i} \right] (270) \\ \geq \Pr_ {\mathbf {x} \sim \mathcal {N} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle \geq - \frac {\mathbf {b} _ {i}}{2} \right] (271) \\ = 1 - \Pr_ {\mathbf {x} \sim \mathcal {N} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle \leq - \frac {\mathbf {b} _ {i}}{2} \right] (272) \\ \geq 1 - \exp \left(- \frac {\mathbf {b} _ {i} ^ {2}}{\Theta (d \sigma_ {w} ^ {2} \sigma_ {j} ^ {2})}\right) (273) \\ \geq 1 - O \left(\frac {1}{d ^ {\tau}}\right), (274) \\ \end{array}
$$
where the third inequality follows the Chernoff bound and symmetry of the Gaussian vector.
For the second statement, we prove similarly by  $0 < C_{\text{Sure},2} \leq \frac{1}{2}$ .
Then, Lemma E.10 gives gradients of neurons in  $S_{D_j, \text{Sure}}$ . It shows that these gradients are highly aligned with  $D_j$ .
Lemma E.10 (Mixture of Gaussians: Feature Emergence). Assume the same conditions as in Lemma E.7, for all  $j \in [r]$ ,  $i \in S_{D_j, S_{\text{ure}}}$ , we have
$$
\begin{array}{l} \left\langle \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} \right], y _ {(j)} D _ {j} \right\rangle (275) \\ \geq p _ {j} B _ {\mu 1} \sqrt {d} \left(1 - O \left(\frac {1}{d ^ {\tau}}\right)\right) - B _ {\mu 2} O \left(\frac {1}{d ^ {\tau - \frac {1}{2}}}\right) - \sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right). (276) \\ \end{array}
$$
For any unit vector  $D_j^\perp$  which is orthogonal with  $D_j$ , we have
$$
\left| \left\langle \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} \right], D _ {j} ^ {\perp} \right\rangle \right| \leq B _ {\mu 2} O \left(\frac {1}{d ^ {\tau - \frac {1}{2}}}\right) + \sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right). \tag {277}
$$
Proof of Lemma E.10. For all  $j \in [r]$ ,  $i \in S_{D_j, Sure}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} \right] (278) \\ = \sum_ {l \in [ r ]} p _ {l} \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} _ {l} (\mathbf {x})} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} \right] (279) \\ = \sum_ {l \in [ r ]} p _ {l} y _ {(l)} \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} (0, \sigma_ {l} I _ {d \times d})} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} + \mu_ {l} \right\rangle - \mathbf {b} _ {i}\right) (\mathbf {x} + \mu_ {l}) \right]. (280) \\ \end{array}
$$
Thus, by Lemma F.7 and Lemma E.9,
For any unit vector  $D_j^\perp$  which is orthogonal with  $D_j$ , similarly, we have
$$
\begin{array}{l} \left| \left\langle \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} \right], D _ {j} ^ {\perp} \right\rangle \right| (294) \\ \leq p _ {j} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} (0, \sigma_ {j} I)} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} + \mu_ {j} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} ^ {\top} D _ {j} ^ {\perp} \right] \right| (295) \\ + \sum_ {l \in [ r ], l \neq j} p _ {l} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} (0, \sigma_ {l} I)} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} + \mu_ {l} \right\rangle - \mathbf {b} _ {i}\right) (\mathbf {x} + \mu_ {l}) ^ {\top} D _ {j} ^ {\perp} \right] \right| (296) \\ \leq B _ {\mu 2} O \left(\frac {1}{d ^ {\tau - \frac {1}{2}}}\right) + p _ {j} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} (0, \sigma_ {j} I)} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} + \mu_ {j} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} ^ {\top} D _ {j} ^ {\perp} \right] \right| (297) \\ + \sum_ {l \in [ r ], l \neq j} p _ {l} \left| \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} (0, \sigma_ {l} I)} \left[ \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} + \mu_ {l} \right\rangle - \mathbf {b} _ {i}\right) \mathbf {x} ^ {\top} D _ {j} ^ {\perp} \right] \right| (298) \\ \leq B _ {\mu 2} O \left(\frac {1}{d ^ {\tau - \frac {1}{2}}}\right) + \sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right). (299) \\ \end{array}
$$
E.2.3 Mixture of Gaussians: Final Guarantee
Lemma E.11 (Mixture of Gaussians: Existence of Good Networks. Part statement of Lemma 4.3). Assume the same conditions as in Lemma E.7. Define
$$
f ^ {*} (\mathbf {x}) = \sum_ {j = 1} ^ {r} \frac {y _ {(j)}}{\sqrt {\tau \log d} \sigma_ {B +}} \left[ \sigma \left(\left\langle D _ {j}, \mathbf {x} \right\rangle - 2 \sqrt {\tau \log d} \sigma_ {B +}\right) \right]. \tag {300}
$$
For  $\mathcal{D}_{\text{mixture}}$  setting, we have  $f^{*} \in \mathcal{F}_{d,r,B_{F},S_{p,\gamma},B_{G}}$ , where  $B_{F} = (B_{a1},B_{a2},B_{b}) =$
$$
\left(\frac {1}{\sqrt {\tau \log d \sigma_ {B +}}}, \frac {\sqrt {r}}{\sqrt {\tau \log d \sigma_ {B +}}}, 2 \sqrt {\tau \log d} \sigma_ {B +}\right), p = \Theta \left(\frac {B _ {\mu 1}}{\sqrt {\tau \log d \sigma_ {B +} \cdot d ^ {(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / (2 B _ {\mu 1} ^ {2})}}}\right), \gamma =
$$
$\frac{1}{d^{0.9\tau - 1.5}}$ ,  $B_{G} = p_{B}B_{\mu 1}\sqrt{d} - O\left(\frac{\sigma_{B + }}{d^{0.9\tau}}\right)$  and  $B_{x1} = (B_{\mu 2} + \sigma_{B + })\sqrt{d}, B_{x2} = (B_{\mu 2} + \sigma_{B + })^2 d.$  We also have  $\mathrm{OPT}_{d,r,B_F,S_{p,\gamma},B_G}\leq \frac{3}{d^\tau} +\frac{4}{d^{0.9\tau - 1}\sqrt{\tau\log d}}$
Proof of Lemma E.11. We can check  $B_{x1} = (B_{\mu 2} + \sigma_{B+})\sqrt{d}, B_{x2} = (B_{\mu 2} + \sigma_{B+})^2 d$  by direct calculation. By Lemma E.7, we have  $f^{*} \in \mathcal{F}_{d,r,B_{F},S_{p,\gamma},B_{G}}$ .
For any  $j\in [r]$ , by  $B_{\mu 1}\geq \Omega \left(\sigma_{B + }\sqrt{\frac{\tau\log d}{d}}\right)\geq 4\sigma_{B + }\sqrt{\frac{\tau\log d}{d}}$ , we have
$$
\begin{array}{l} \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {j}, \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +} \geq \sqrt {\tau \log d} \sigma_ {B +} \right] (301) \\ = \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle D _ {j}, \mathbf {x} \right\rangle + \| \mu_ {j} \| _ {2} - 2 \sqrt {\tau \log d} \sigma_ {B +} \geq \sqrt {\tau \log d} \sigma_ {B +} \right] (302) \\ \geq \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \left\langle D _ {j}, \mathbf {x} \right\rangle + \sqrt {d} B _ {\mu 1} - 2 \sqrt {\tau \log d} \sigma_ {B +} \geq \sqrt {\tau \log d} \sigma_ {B +} \right] (303) \\ \geq \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {j}, \mathbf {x} \rangle \geq - \sqrt {\tau \log d} \sigma_ {B +} \right] (304) \\ \geq 1 - \frac {1}{d ^ {\tau}}, (305) \\ \end{array}
$$
where the last inequality follows Chernoff bound.
For any  $l \neq j, l \in [r]$ , by  $\theta \leq \frac{\sigma_{B+}}{B_{\mu 2}} \sqrt{\frac{\tau \log d}{d}}$ , we have
$$
\begin{array}{l} \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {l}, \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +} \geq 0 \right] (306) \\ \leq \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {l}, \mathbf {x} \rangle + \theta B _ {\mu 2} \sqrt {d} - 2 \sqrt {\tau \log d} \sigma_ {B +} \geq 0 \right] (307) \\ \leq \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(0, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {l}, \mathbf {x} \rangle \geq \sqrt {\tau \log d} \sigma_ {B +} \right] (308) \\ \leq \frac {1}{d ^ {\tau}}. (309) \\ \end{array}
$$
$$
\begin{array}{l} \Pr_ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {m i x t u r e}}} [ y f ^ {*} (\mathbf {x}) > 1 ] (310) \\ \geq \sum_ {j \in [ r ]} p _ {j} \left(\Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {j}, \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +} \geq \sqrt {\tau \log d} \sigma_ {B +} \right]\right) (311) \\ - \sum_ {j \in [ r ]} p _ {j} \left(\sum_ {l \neq j, l \in [ r ]} \Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {l}, \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +} <   0 \right]\right) (312) \\ \end{array}
$$
$$
\geq 1 - \frac {2}{d ^ {\tau}}. \tag {313}
$$
$$
\mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {m i x t u r e}}} [ \mathbb {I} [ y f ^ {*} (\mathbf {x}) \leq 1 ] | y f ^ {*} (\mathbf {x}) ] ] \tag {314}
$$
$$
\begin{array}{l} \leq \sum_ {j \in [ r ]} p _ {j} \left(\Pr_ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \langle D _ {j}, \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +} <   \sqrt {\tau \log d} \sigma_ {B +} \right] \frac {y _ {(j)} ^ {2} \sqrt {\tau \log d} \sigma_ {B +}}{\sqrt {\tau \log d} \sigma_ {B +}}\right) \\ + \sum_ {j \in [ r ]} p _ {j} \left(\sum_ {l \neq j, l \in [ r ]} \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} _ {j} \left(\mu_ {j}, \sigma_ {j} I _ {d \times d}\right)} \left[ \sigma^ {\prime} \left[ \langle D _ {l}, \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +} > 0 \right] \frac {\langle D _ {l} , \mathbf {x} \rangle - 2 \sqrt {\tau \log d} \sigma_ {B +}}{\sqrt {\tau \log d} \sigma_ {B +}} \right]\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq \frac {1}{d ^ {\tau}} + \sum_ {j \in [ r ]} p _ {j} \left(\sum_ {l \neq j, l \in [ r ]} \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} _ {j} (0, \sigma_ {j} I _ {d \times d})} \left[ \sigma^ {\prime} \left[ \langle D _ {l}, \mathbf {x} \rangle > \sqrt {\tau \log d} \sigma_ {B +} \right] \frac {\langle D _ {l} , \mathbf {x} \rangle - \sqrt {\tau \log d} \sigma_ {B +}}{\sqrt {\tau \log d} \sigma_ {B +}} \right]\right) \\ \leq \frac {1}{d ^ {\tau}} + \frac {1}{\sqrt {\tau \log d}} \sum_ {j \in [ r ]} p _ {j} \left(\sum_ {l \neq j, l \in [ r ]} \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} _ {j} \left(0, I _ {d \times d}\right)} \left[ \sigma^ {\prime} \left[ \langle D _ {l}, \mathbf {x} \rangle > \sqrt {\tau \log d} \right] \langle D _ {l}, \mathbf {x} \rangle \right]\right) (315) \\ \leq \frac {1}{d ^ {\tau}} + \frac {4}{d ^ {0 . 9 \tau - 1} \sqrt {\tau \log d}}, (316) \\ \end{array}
$$
where the second last inequality follows Lemma F.7 and  $r \leq 2d$ . Thus, we have
$$
\begin{array}{l} \operatorname {O P T} _ {d, r, B _ {F}, S _ {p, \gamma}, B _ {G}} \leq \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {m i x t u r e}}} [ \ell (y f ^ {*} (\mathbf {x})) ] (317) \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {m i x t u r e}}} [ \mathbb {I} [ y f ^ {*} (\mathbf {x}) \leq 1 ] (1 - y f ^ {*} (\mathbf {x})) ] (318) \\ \leq \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {m i x t u r e}}} [ \mathbb {I} [ y f ^ {*} (\mathbf {x}) \leq 1 ] | y f ^ {*} (\mathbf {x}) | ] + \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {m i x t u r e}}} [ \mathbb {I} [ y f ^ {*} (\mathbf {x}) \leq 1 ] ] \\ \leq \frac {3}{d ^ {\tau}} + \frac {4}{d ^ {0 . 9 \tau - 1} \sqrt {\tau \log d}}. (319) \\ \end{array}
$$
Theorem 4.4 (Mixtures of Gaussians: Main Result). Assume Assumption 4.1. For any  $\epsilon, \delta \in (0,1)$  when Algorithm 1 uses hinge loss with
$$
m = \operatorname {p o l y} \left(\frac {1}{\delta}, \frac {1}{\epsilon}, d ^ {\Theta \left(\tau \sigma_ {B +} {} ^ {2} / B _ {\mu 1} ^ {2}\right)}, r, \frac {1}{p _ {B}}\right) \leq e ^ {d}, \quad T = \operatorname {p o l y} (m), \quad n = \operatorname {p o l y} (m)
$$
$$
\Pr [ \operatorname {s i g n} (f _ {\Xi^ {(t)}} (\mathbf {x})) \neq y ] \leq \frac {\sqrt {2} r}{d ^ {0 . 4 \tau - 0 . 8}} + \epsilon .
$$
Proof of Theorem 4.4. Let  $\tilde{b} = C_b\sqrt{\tau d\log d}\sigma_w\sigma_{B+}$ , where  $C_b$  is a large enough universal constant. By Lemma E.11, we have  $f^* \in \mathcal{F}_{d,r,B_F,S_{p,\gamma},B_G}$ , where  $B_F = (B_{a1},B_{a2},B_b) = \left(\frac{1}{\sqrt{\tau\log d}\sigma_{B+}},\frac{\sqrt{r}}{\sqrt{\tau\log d}\sigma_{B+}},2\sqrt{\tau\log d}\sigma_{B+}\right)$ ,  $p = \Theta\left(\frac{B_{\mu 1}}{\sqrt{\tau\log d}\sigma_{B+}.d\left(9C_b^2\tau\sigma_{B+}^2/(2B_{\mu 1}^2)\right)}\right)$ ,  $\gamma = \frac{1}{d^{0.9\tau-1.5}}$ ,  $B_G = p_B B_{\mu 1}\sqrt{d} - O\left(\frac{\sigma_{B+}}{d^{0.9\tau}}\right)$  and  $B_{x1} = (B_{\mu 2} + \sigma_{B+})\sqrt{d}$ ,  $B_{x2} = (B_{\mu 2} + \sigma_{B+})^2 d$ . We also have  $\mathrm{OPT}_{d,r,B_F,S_{p,\gamma},B_G} \leq \frac{3}{d^\tau} + \frac{4}{d^{0.9\tau-1}\sqrt{\tau\log d}}$ .
Adjust  $\sigma_w$  such that  $\tilde{b} = C_b\sqrt{\tau d\log d}\sigma_w\sigma_{B+} = \Theta\left(\frac{B_G^{\frac{1}{4}}B_{a2}B_b^{\frac{3}{4}}}{\sqrt{rB_a1}}\right)$ . Injecting above parameters into Theorem 3.12, we have with probability at least  $1 - \delta$  over the initialization, with proper hyper-parameters, there exists  $t \in [T]$  such that
$$
\begin{array}{l} \Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}} (\mathbf {x})\right) \neq y \right] (320) \\ \leq \frac {3}{d ^ {\tau}} + \frac {4}{d ^ {0 . 9 \tau - 1} \sqrt {\tau \log d}} + \frac {\sqrt {2} r B _ {\mu 2}}{d ^ {(0 . 9 \tau - 1 . 5) / 2} \sqrt {\tau \log d} \sigma_ {B +}} + O \left(\frac {r B _ {a 1} B _ {x 1} B _ {x 2} ^ {\frac {1}{4}} (\log n) ^ {\frac {1}{4}}}{\sqrt {B _ {G}} n ^ {\frac {1}{4}}}\right) + \epsilon / 2 \\ \leq \frac {\sqrt {2} r}{d ^ {0 . 4 \tau - 0 . 8}} + \epsilon . (321) \\ \end{array}
$$
E.3 Mixture of Gaussians - XOR
We consider a special Mixture of Gaussians distribution studied in [99]. Consider the same data distribution in Appendix E.2.1 and Definition E.6 with the following assumptions.
Assumption E.12 (Mixture of Gaussians in [99]). Assume four Gaussians cluster with XOR-like pattern, for any  $\tau > 0$ ,
-  $r = 4$  and  $p_1 = p_2 = p_3 = p_4 = \frac{1}{4}$ .
-  $\mu_{1} = -\mu_{2}, \mu_{3} = -\mu_{4}$  and  $\|\mu_1\|_2 = \|\mu_2\|_2 = \|\mu_3\|_2 = \|\mu_4\|_2 = \sqrt{d}$  and  $\langle \mu_1, \mu_3 \rangle = 0$ .
- For all  $j \in [4]$ ,  $\Sigma_{j} = \sigma_{B} I_{d \times d}$  and  $1 \leq \sigma_{B} \leq \sqrt{\frac{d}{\tau \log \log d}}$ .
-  $y_{(1)} = y_{(2)} = 1$  and  $y_{(3)} = y_{(4)} = -1$ .
We denote this data distribution as  $\mathcal{D}_{\text{mixture - xor}}$  setting.
E.3.1 Mixture of Gaussians - XOR: Feature Learning
Lemma E.13 (Mixture of Gaussians in [99]: Gradient Feature Set). Let  $C_{\text{Sure}, 1} = \frac{6}{5}$ ,  $C_{\text{Sure}, 2} = \frac{\sqrt{2}}{\sqrt{\tau \log \log d}}$ ,  $\tilde{b} = \sqrt{\tau d \log \log d} \sigma_w \sigma_B$  and  $d$  is large enough. For  $\mathcal{D}_{\text{mixture - xor}}$  setting, we have  $(D_j, +1) \in S_{p,\gamma, B_G}$  for all  $j \in [4]$ , where
$$
p = \Theta \left(\frac {1}{\sqrt {\tau \log \log d} \sigma_ {B} \cdot (\log d) ^ {\frac {1 8 \tau \sigma_ {B} ^ {2}}{2 5}}}\right), \quad \gamma = \frac {\sigma_ {B}}{\sqrt {d}}, \tag {322}
$$
$$
B _ {G} = \frac {\sqrt {d}}{4} \left(1 - O \left(\frac {1}{\left(\log d\right) ^ {\frac {\tau}{5 0}}}\right)\right) - \sigma_ {B} O \left(\frac {1}{\left(\log d\right) ^ {0 . 0 1 8 \tau}}\right). \tag {323}
$$
Proof of Lemma E.13. For all  $j \in [r]$ , by Lemma E.16, for all  $i \in S_{D_j, Sure}$
$$
1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right|}{\left\| G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right) \right\| _ {2}} \tag {324}
$$
$$
\leq 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right|}{\sqrt {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right| ^ {2} + \max  _ {D _ {j} ^ {\top} D _ {j} ^ {\perp} = 0 , \| D _ {j} ^ {\perp} \| _ {2} = 1} \left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} ^ {\perp} \right\rangle \right| ^ {2}}} \tag {325}
$$
$$
\leq 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right|}{\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right| + \max  _ {D _ {j} ^ {\top} D _ {j} ^ {\perp} = 0, \| D _ {j} ^ {\perp} \| _ {2} = 1} \left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} ^ {\perp} \right\rangle \right|} \tag {326}
$$
$$
\leq 1 - \frac {1}{1 + \frac {\sigma_ {B} O \left(\frac {1}{(\log d) ^ {0 . 0 1 8 \tau}}\right)}{\frac {1}{4} \sqrt {d} \left(1 - O \left(\frac {1}{(\log d) ^ {\frac {\tau}{5 0}}}\right)\right) - \sigma_ {B} O \left(\frac {1}{(\log d) ^ {0 . 0 1 8 \tau}}\right)}} \tag {327}
$$
$$
\leq \frac {\sigma_ {B} O \left(\frac {1}{(\log d) ^ {0 . 0 1 8 \tau}}\right)}{\frac {1}{4} \sqrt {d} \left(1 - O \left(\frac {1}{(\log d) ^ {\frac {\tau}{5 0}}}\right)\right) - \sigma_ {B} O \left(\frac {1}{(\log d) ^ {0 . 0 1 8 \tau}}\right)} \tag {328}
$$
$$
<   \frac {\sigma_ {B}}{\sqrt {d}} = \gamma . \tag {329}
$$
Thus, we have  $G(\mathbf{w}_i^{(0)}, \mathbf{b}_i) \in \mathcal{C}_{D_j,\gamma}$  and  $\left|\left\langle G(\mathbf{w}_i^{(0)}, \mathbf{b}_i), D_j \right\rangle \right| \leq \| G(\mathbf{w}_i^{(0)}, \mathbf{b}_i) \|_2 \leq B_{x1}, \frac{\mathbf{b}_i}{|\mathbf{b}_i|} = +1$ . Thus, by Lemma E.14, we have
$$
\Pr_ {\mathbf {w}, b} \left[ G (\mathbf {w}, b) \in \mathcal {C} _ {D _ {j}, \gamma} \text {a n d} \| G (\mathbf {w}, b) \| _ {2} \geq B _ {G} \text {a n d} \frac {b}{| b |} = + 1 \right] \tag {330}
$$
$$
\geq \Pr [ i \in S _ {D _ {j}, S u r e} ] \tag {331}
$$
$$
\geq p. \tag {332}
$$
Lemma E.14 (Mixture of Gaussians in [99]: Geometry at Initialization). Assume the same conditions as in Lemma E.13. Recall for all  $i \in [m]$ ,  $\mathbf{w}_i^{(0)} \sim \mathcal{N}(0, \sigma_w^2 I_{d \times d})$ , over the random initialization, we have for all  $i \in [m], j \in [4]$
$$
\Pr \left[ i \in S _ {D _ {j}, S u r e} \right] \geq \Theta \left(\frac {1}{\sqrt {\tau \log \log d} \sigma_ {B} \cdot (\log d) ^ {\frac {1 8 \tau \sigma_ {B} ^ {2}}{2 5}}}\right). \tag {333}
$$
Proof of Lemma E.14. WLOG, let  $j = 1$ . By Assumption E.12, for the first condition in Definition E.6, we have,
$$
\Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {1} \right\rangle \geq C _ {\text {S u r e}, 1} \mathbf {b} _ {i} \right] \geq \Theta \left(\frac {1}{\sqrt {\tau \log \log d} \sigma_ {B} \cdot (\log d) ^ {\frac {1 8 \tau \sigma_ {B} ^ {2}}{2 5}}}\right), \tag {334}
$$
where the last inequality follows Lemma F.6.
$$
\Pr \left[ \left| \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {2} \right\rangle \right| \leq C _ {S u r e, 2} \mathbf {b} _ {i} \right] \geq 1 - \frac {1}{2 \sqrt {\pi}} \frac {1}{\sigma_ {B} \cdot e ^ {\sigma_ {B} ^ {2}}}, \tag {335}
$$
On the other hand, if  $X$  is a  $\chi^2 (k)$  random variable. Then we have
$$
\Pr (X \geq k + 2 \sqrt {k x} + 2 x) \leq e ^ {- x}. \tag {336}
$$
$$
\begin{array}{l} \Pr \left(\frac {1}{\sigma_ {w} ^ {2}} \left\| \mathbf {w} _ {i} ^ {(0)} \right\| _ {2} ^ {2} \geq d + 2 \sqrt {\left(\frac {1 8 \tau \sigma_ {B} ^ {2}}{2 5} + 2\right) d \log \log d} + 2 \left(\frac {1 8 \tau \sigma_ {B} ^ {2}}{2 5} + 2\right) \log \log d\right) (337) \\ \leq O \left(\frac {1}{\left(\log d\right) ^ {2} \cdot \left(\log d\right) ^ {\frac {1 8 \tau \sigma_ {B} ^ {2}}{2 5}}}\right). (338) \\ \end{array}
$$
Thus, by union bound, we have