SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
+ \frac {\sqrt {\log n}}{n ^ {\frac {1}{2}}} O \left(m \eta \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + T \eta^ {2} B _ {x 1} \tilde {b}\right) \sqrt {B _ {x 2}}\right). \tag {197}
$$
and note that  $B_G \leq B_{x_1} \leq B_x$  and  $\sqrt{B_{x_2}} \leq B_x$  naturally, we have
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) \tag {198}
$$
$$
\leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} \left(\frac {2 \sqrt {2} B _ {x 1}}{(m p) ^ {\frac {1}{4}}} \sqrt {\frac {B _ {b}}{B _ {G}}} + B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{2}}}\right)}\right) \tag {199}
$$
$$
+ \frac {\epsilon}{2} + O \left(\frac {B _ {x 1} B _ {a 2} ^ {2} B _ {b} ^ {2}}{\tilde {b} ^ {2} (m p) ^ {\frac {1}{4}}}\right) + \frac {\sqrt {\log n}}{n ^ {\frac {1}{2}}} O \left(\frac {m B _ {x} B _ {a 2} ^ {2} \sqrt {B _ {b}} (m p) ^ {\frac {1}{2}} \log m}{r B _ {a 1} \sqrt {B _ {G}}}\right) \tag {200}
$$
$$
+ \frac {\sqrt {\log n}}{n ^ {\frac {1}{2}}} O \left(\frac {\epsilon \sqrt {B _ {x 2} \log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}}\right) \tag {201}
$$
$$
\leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{2}}}\right)} + \epsilon . \tag {202}
$$
E Applications in Special Cases
We present the case study of linear data in Appendix E.1, mixtures of Gaussians in Appendix E.2 and Appendix E.3, parity functions in Appendix E.4, Appendix E.5 and Appendix E.6, and multiple-index models in Appendix E.7.
In special case applications, we consider binary classification with hinge loss, e.g.,  $\ell(z) = \max\{1 - z, 0\}$ . Let  $\mathcal{X} = \mathbb{R}^d$  be the input space, and  $\mathcal{Y} = \{\pm 1\}$  be the label space.
Remark E.1 (Hinge Loss and Logistic Loss). Both hinge loss and logistic loss can be used in special cases and general cases. For convenience, we use hinge loss in special cases, where we can directly get the ground-truth NN close form of the optimal solution which has zero loss. For logistic loss, there is no zero-loss solution. We can still show that the OPT value has an exponentially small upper bound at the cost of more computation.
E.1 Linear Data
Data Distributions. Suppose two labels are equiprobable, i.e.,  $\mathbb{E}[y = -1] = \mathbb{E}[y = +1] = \frac{1}{2}$ . The input data are linearly separable and there is a ground truth direction  $\mathbf{w}^*$ , where  $\| \mathbf{w}^* \|_2 = 1$ , such that  $y \langle \mathbf{w}^*, \mathbf{x} \rangle > 0$ . We also assume  $\mathbb{E}[y P_{\mathbf{w}^{*} \perp} \mathbf{x}] = 0$ , where  $P_{\mathbf{w}^{*} \perp}$  is the projection operator on the complementary space of the ground truth, i.e., the components of input data being orthogonal with the ground truth are independent of the label  $y$ . We define the input data signal level as  $\rho := \mathbb{E}[y \langle \mathbf{w}^*, \mathbf{x} \rangle] > 0$  and the margin as  $\beta := \min_{(\mathbf{x}, y)} y \langle \mathbf{w}^*, \mathbf{x} \rangle > 0$ .
We call this data distribution  $\mathcal{D}_{\text{linear}}$ .
Lemma E.2 (Linear Data: Gradient Feature Set). Let  $\tilde{b} = d^{\tau}B_{x1}\sigma_{w}$ , where  $\tau$  is any number large enough to satisfy  $d^{\tau /2 - \frac{1}{4}} > \Omega\left(\frac{\sqrt{B_{x2}}}{\rho}\right)$ . For  $\mathcal{D}_{linear}$  setting, we have  $(\mathbf{w}^{*}, - 1)\in S_{p,\gamma ,B_{G}}$  where
$$
p = \frac {1}{2}, \quad \gamma = \Theta \left(\frac {\sqrt {B _ {x 2}}}{\rho d ^ {\tau / 2 - \frac {1}{4}}}\right), \quad B _ {G} = \rho - \Theta \left(\frac {\sqrt {B _ {x 2}}}{d ^ {\tau / 2 - \frac {1}{4}}}\right). \tag {203}
$$
Proof of Lemma E.2. By data distribution, we have
$$
\mathbb {E} _ {(\mathbf {x}, y)} [ y \mathbf {x} ] = \rho \mathbf {w} ^ {*}. \tag {204}
$$
Define  $S_{Sure}:\{i\in [m]:\| \mathbf{w}_i^{(0)}\| _2\leq 2\sqrt{d}\sigma_w\}$ . For all  $i\in [m]$ , we have
$$
\Pr [ i \in S _ {S u r e} ] = \Pr [ \| \mathbf {w} _ {i} ^ {(0)} \| _ {2} \leq 2 \sqrt {d} \sigma_ {w} ] \geq \frac {1}{2}. \tag {205}
$$
For all  $i \in S_{\text{Sure}}$ , by Markov's inequality and considering neuron  $i + m$ , we have
$$
\begin{array}{l} \Pr_ {\mathbf {x}} \left[ \left\langle \mathbf {w} _ {i + m} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i + m} <   0 \right] = \Pr_ {\mathbf {x}} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle + \mathbf {b} _ {i} <   0 \right] (206) \\ \leq \Pr_ {\mathbf {x}} \left[ \| \mathbf {w} _ {i} ^ {(0)} \| _ {2} \| \mathbf {x} \| _ {2} \geq \mathbf {b} _ {i} \right] (207) \\ \leq \Pr_ {\mathbf {x}} \left[ \| \mathbf {x} \| _ {2} \geq \frac {d ^ {\tau - \frac {1}{2}} B _ {x 1}}{2} \right] (208) \\ \leq \Theta \left(\frac {1}{d ^ {\tau - \frac {1}{2}}}\right). (209) \\ \end{array}
$$
For all  $i \in S_{\text{Sure}}$ , by Hölder's inequality, we have
$$
\begin{array}{l} \left\| \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \left(1 - \sigma^ {\prime} \left[ \left\langle \mathbf {w} _ {i + m} ^ {(0)}, \mathbf {x} \right\rangle - \mathbf {b} _ {i + m} \right]\right) \mathbf {x} \right] \right\| _ {2} (210) \\ = \left\| \mathbb {E} _ {(\mathbf {x}, y)} \left[ y \left(1 - \sigma^ {\prime} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle + \mathbf {b} _ {i} \right]\right) \mathbf {x} \right] \right\| _ {2} (211) \\ \leq \sqrt {\mathbb {E} [ \| \mathbf {x} \| _ {2} ^ {2} ] \mathbb {E} \left[ \left(1 - \sigma^ {\prime} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mathbf {x} \right\rangle + \mathbf {b} _ {i} \right]\right) ^ {2} \right]} (212) \\ \leq \Theta \left(\frac {\sqrt {B _ {x 2}}}{d ^ {\tau / 2 - \frac {1}{4}}}\right). (213) \\ \end{array}
$$
$$
\begin{array}{l} 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i + m} ^ {(0)} , \mathbf {b} _ {i + m}\right) , \mathbf {w} ^ {*} \right\rangle \right|}{\| G \left(\mathbf {w} _ {i + m} ^ {(0)} , \mathbf {b} _ {i + m}\right) \| _ {2}} = 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , - \mathbf {b} _ {i}\right) , \mathbf {w} ^ {*} \right\rangle \right|}{\| G \left(\mathbf {w} _ {i} ^ {(0)} , - \mathbf {b} _ {i}\right) \| _ {2}} (214) \\ \leq 1 - \frac {\rho - \Theta \left(\frac {\sqrt {B _ {x 2}}}{d ^ {\tau / 2} - \frac {1}{4}}\right)}{\rho + \Theta \left(\frac {\sqrt {B _ {x 2}}}{d ^ {\tau / 2} - \frac {1}{4}}\right)} (215) \\ = \Theta \left(\frac {\sqrt {B _ {x 2}}}{\rho d ^ {\tau / 2 - \frac {1}{4}}}\right) = \gamma . (216) \\ \end{array}
$$
We finish the proof by  $\frac{\mathbf{b}_{i + m}}{|\mathbf{b}_{i + m}|} = -1$ .
Lemma E.3 (Linear Data: Existence of Good Networks). Assume the same conditions as in Lemma E.2. Define
$$
f ^ {*} (\mathbf {x}) = \frac {1}{\beta} \sigma \left(\left\langle \mathbf {w} ^ {*}, \mathbf {x} \right\rangle\right) - \frac {1}{\beta} \sigma \left(\left\langle - \mathbf {w} ^ {*}, \mathbf {x} \right\rangle\right). \tag {217}
$$
For  $\mathcal{D}_{linear}$  setting, we have  $f^{*} \in \mathcal{F}_{d,r,B_{F},S_{p,\gamma ,B_{G}}}$ , where  $r = 2$ ,  $B_{F} = (B_{a1},B_{a2},B_{b}) = \left(\frac{1}{\beta},\frac{\sqrt{2}}{\beta},\frac{1}{B_{x1}^{2}}\right)$ ,  $p = \frac{1}{2}$ ,  $\gamma = \Theta \left(\frac{\sqrt{B_{x2}}}{\rho d^{\tau / 2 - \frac{1}{4}}}\right)$ ,  $B_{G} = \rho -\Theta \left(\frac{\sqrt{B_{x2}}}{d^{\tau / 2 - \frac{1}{4}}}\right)$ . We also have  $\mathrm{OPT}_{d,r,B_F,S_{p,\gamma ,B_G}} = 0$ .
Proof of Lemma E.3. By Lemma E.2 and Lemma F.3, we have  $f^{*}\in \mathcal{F}_{d,r,B_{F},S_{p,\gamma},B_{G}}$ . We also have
$$
\begin{array}{l} \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} \leq \mathcal {L} _ {\mathcal {D} _ {\text {l i n e a r}}} (f ^ {*}) (218) \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {\text {l i n e a r}}} \mathcal {L} _ {(\mathbf {x}, y)} \left(f ^ {*}\right) (219) \\ = 0. (220) \\ \end{array}
$$
Theorem E.4 (Linear Data: Main Result). For  $\mathcal{D}_{\text{linear}}$  setting, for any  $\delta \in (0,1)$  and for any  $\epsilon \in (0,1)$  when
$$
m = \operatorname {p o l y} \left(\frac {1}{\delta}, \frac {1}{\epsilon}, \frac {1}{\beta}, \frac {1}{\rho}\right) \leq e ^ {d}, \quad T = \operatorname {p o l y} (m, B _ {x 1}), \quad n = \operatorname {p o l y} \left(m, B _ {x}, \frac {1}{\delta}, \frac {1}{\epsilon}, \frac {1}{\beta}, \frac {1}{\rho}\right), \tag {221}
$$
trained by Algorithm 1 with hinge loss, with probability at least  $1 - \delta$  over the initialization, with proper hyper-parameters, there exists  $t\in [T]$  such that
$$
\Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}} (\mathbf {x})\right) \neq y \right] \leq \epsilon . \tag {222}
$$
Proof of Theorem E.4. Let  $\tilde{b} = d^{\tau}B_{x1}\sigma_{w}$ , where  $\tau$  is a number large enough to satisfy  $d^{\tau /2 - \frac{1}{4}} > \Omega\left(\frac{\sqrt{B_{x2}}}{\rho}\right)$  and  $O\left(\frac{B_{x1}B_{x2}^{\frac{1}{4}}}{\beta\sqrt{\rho}d^{\tau /4 - \frac{1}{8}}}\right) \leq \frac{\epsilon}{2}$ . By Lemma E.3, we have  $f^{*} \in \mathcal{F}_{d,r,B_{F},S_{p,\gamma ,B_{G}}}$ , where  $r = 2, B_{F} = (B_{a1},B_{a2},B_{b}) = \left(\frac{1}{\beta},\frac{\sqrt{2}}{\beta},\frac{1}{B_{x1}^{2}}\right), p = \frac{1}{2}, \gamma = \Theta\left(\frac{\sqrt{B_{x2}}}{\rho d^{\tau / 2 - \frac{1}{4}}}\right), B_{G} = \rho -\Theta\left(\frac{\sqrt{B_{x2}}}{d^{\tau / 2 - \frac{1}{4}}}\right)$ . We also have  $\mathrm{OPT}_{d,r,B_F,S_p,\gamma ,B_G} = 0$ .
Adjust  $\sigma_w$  such that  $\tilde{b} = d^{\tau}B_{x1}\sigma_w = \Theta \left(\frac{B_G^{\frac{1}{4}}B_{a2}B_b^{\frac{3}{4}}}{\sqrt{rB_{a1}}}\right)$ . Injecting above parameters into Theorem 3.12, we have with probability at least  $1 - \delta$  over the initialization, with proper hyper-parameters, there exists  $t\in [T]$  such that
$$
\Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}} (\mathbf {x})\right) \neq y \right] \leq O \left(\frac {B _ {x 1} B _ {x 2} ^ {\frac {1}{4}}}{\beta \sqrt {\rho} d ^ {\tau / 4 - \frac {1}{8}}}\right) + O \left(\frac {B _ {x 1} B _ {x 2} ^ {\frac {1}{4}} (\log n) ^ {\frac {1}{4}}}{\beta \sqrt {\rho} n ^ {\frac {1}{4}}}\right) + \epsilon / 2 \leq \epsilon . \tag {223}
$$
E.2 Mixture of Gaussians
We recap the problem setup in Section 4.1 for readers' convenience.
E.2.1 Problem Setup
Data Distributions. We follow the notations from [99]. The data are from a mixture of  $r$  high-dimensional Gaussians, and each Gaussian is assigned to one of two possible labels in  $\mathcal{V} = \{\pm 1\}$ . Let  $S(y) \subseteq [r]$  denote the set of indices of the Gaussians associated with the label  $y$ . The data distribution is then:
$$
q (\mathbf {x}, y) = q (y) q (\mathbf {x} | y), \quad q (\mathbf {x} | y) = \sum_ {j \in \mathcal {S} (y)} p _ {j} \mathcal {N} _ {j} (\mathbf {x}), \tag {224}
$$
where  $\mathcal{N}_j(\mathbf{x})$  is a multivariate normal distribution with mean  $\mu_{j}$  and covariance  $\Sigma_{j}$ , and  $p_j$  are chosen such that  $q(\mathbf{x},y)$  is correctly normalized.
We call this data distribution  $\mathcal{D}_{\text{mixture}}$ .
We will make some assumptions about the Gaussians, for which we first introduce some notations. For all  $j \in [r]$ , let  $y_{(j)} \in \{+1, -1\}$  be the label for  $\mathcal{N}_j(\mathbf{x})$ .
$$
D _ {j} := \frac {\mu_ {j}}{\| \mu_ {j} \| _ {2}}, \tilde {\mu} _ {j} := \mu_ {j} / \sqrt {d}, B _ {\mu 1} := \min  _ {j \in [ r ]} \| \tilde {\mu} _ {j} \| _ {2}, B _ {\mu 2} := \max  _ {j \in [ r ]} \| \tilde {\mu} _ {j} \| _ {2}, p _ {B} := \min  _ {j \in [ r ]} p _ {j}.
$$
Assumption E.5 (Mixture of Gaussians. Recap of Assumption 4.1). Let  $8 \leq \tau \leq d$  be a parameter that will control our final error guarantee. Assume
- Equiprobable labels:  $q(-1) = q(+1) = 1/2$ .
- For all  $j \in [r]$ ,  $\Sigma_{j} = \sigma_{j} I_{d \times d}$ . Let  $\sigma_{B} := \max_{j \in [r]} \sigma_{j}$  and  $\sigma_{B+} := \max \{ \sigma_{B}, B_{\mu 2} \}$ .
-  $r \leq 2d$ ,  $p_B \geq \frac{1}{2d}$ ,  $\Omega\left(\frac{1}{d} + \sqrt{\frac{\tau \sigma_{B+2} \log d}{d}}\right) \leq B_{\mu 1} \leq B_{\mu 2} \leq d$ .
- The Gaussians are well-separated: for all  $i \neq j \in [r]$ , we have  $-1 \leq \langle D_i, D_j \rangle \leq \theta$ , where
$$
0 \leq \theta \leq \min \left\{\frac {1}{2 r}, \frac {\sigma_ {B +}}{B _ {\mu 2}} \sqrt {\frac {\tau \log d}{d}} \right\}.
$$
Below, we define a sufficient condition that randomly initialized weights will fall in nice gradients set after the first gradient step update.
Definition E.6 (Mixture of Gaussians: Subset of Nice Gradients Set). Recall  $\mathbf{w}_i^{(0)}$  is the weight for the  $i$ -th neuron at initialization. For all  $j \in [r]$ , let  $S_{D_j, Sure} \subseteq [m]$  be those neurons that satisfy
$\left\langle \mathbf{w}_i^{(0)},\mu_j\right\rangle \geq C_{Sure,1}\mathbf{b}_i,$
$\left\langle \mathbf{w}_i^{(0)},\mu_{j'}\right\rangle \leq C_{Sure,2}\mathbf{b}_i,\text{for all} j'\neq j,j'\in [r].$
$\left\| \mathbf{w}_i^{(0)}\right\| _2\leq \Theta (\sqrt{d}\sigma_w).$
E.2.2 Mixture of Gaussians: Feature Learning
We show the important Lemma E.7 first and defer other Lemmas after it.
Lemma E.7 (Mixture of Gaussians: Gradient Feature Set. Part statement of Lemma 4.3). Let  $C_{Sure,1} = \frac{3}{2}$ ,  $C_{Sure,2} = \frac{1}{2}$ ,  $\tilde{b} = C_b\sqrt{\tau d\log d}\sigma_w\sigma_{B+}$ , where  $C_b$  is a large enough universal constant. For  $\mathcal{D}_{mixture}$  setting, we have  $(D_j, +1) \in S_{p,\gamma,B_G}$  for all  $j \in [r]$ , where
$$
p = \Theta \left(\frac {B _ {\mu 1}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}}\right), \quad \gamma = \frac {1}{d ^ {0 . 9 \tau - 1 . 5}}, \tag {225}
$$
$$
B _ {G} = p _ {B} B _ {\mu 1} \sqrt {d} - O \left(\frac {\sigma_ {B +}}{d ^ {0 . 9 \tau}}\right). \tag {226}
$$
Proof of Lemma E.7. For all  $j \in [r]$ , by Lemma E.10, for all  $i \in S_{D_j, Sure}$
$$
\begin{array}{l} 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right|}{\left\| G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) \right\| _ {2}} (227) \\ \leq 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right|}{\sqrt {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right| ^ {2} + \max  _ {D _ {j} ^ {\top} D _ {j} ^ {\perp} = 0 , \| D _ {j} ^ {\perp} \| _ {2} = 1} \left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} ^ {\perp} \right\rangle \right| ^ {2}}} (228) \\ \leq 1 - \frac {\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)} , \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right|}{\left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right) , D _ {j} \right\rangle \right| + \max  _ {D _ {j} ^ {\top} D _ {j} ^ {\perp} = 0, \| D _ {j} ^ {\perp} \| _ {2} = 1} \left| \left\langle G \left(\mathbf {w} _ {i} ^ {(0)}, \mathbf {b} _ {i}\right), D _ {j} ^ {\perp} \right\rangle \right|} (229) \\ \leq 1 - \frac {1}{1 + \frac {B _ {\mu 2} O \left(\frac {1}{d ^ {\tau} - \frac {1}{2}}\right) + \sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right)}{p _ {j} B _ {\mu 1} \sqrt {d} \left(1 - O \left(\frac {1}{d ^ {\tau}}\right)\right) - B _ {\mu 2} O \left(\frac {1}{d ^ {\tau} - \frac {1}{2}}\right) - \sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right)}} (230) \\ \leq \frac {\sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right)}{p _ {j} B _ {\mu 1} \sqrt {d} - \sigma_ {B +} O \left(\frac {1}{d ^ {0 . 9 \tau}}\right)} (231) \\ <   \frac {1}{d ^ {0 . 9 \tau - 1 . 5}} = \gamma , (232) \\ \end{array}
$$
where the last inequality follows  $B_{\mu 1}\geq \Omega \left(\sigma_{B + }\sqrt{\frac{\tau\log d}{d}}\right)$
Thus, we have  $G(\mathbf{w}_i^{(0)}, \mathbf{b}_i) \in \mathcal{C}_{D_j,\gamma}$  and  $\left|\left\langle G(\mathbf{w}_i^{(0)}, \mathbf{b}_i), D_j \right\rangle \right| \leq \| G(\mathbf{w}_i^{(0)}, \mathbf{b}_i) \|_2 \leq B_{x1}, \frac{\mathbf{b}_i}{|\mathbf{b}_i|} = +1$ . Thus, by Lemma E.8, we have
$$
\Pr_ {\mathbf {w}, b} \left[ G (\mathbf {w}, b) \in \mathcal {C} _ {D _ {j}, \gamma} \text {a n d} \| G (\mathbf {w}, b) \| _ {2} \geq B _ {G} \text {a n d} \frac {b}{| b |} = + 1 \right] \tag {233}
$$
$$
\geq \Pr [ i \in S _ {D _ {j}, S u r e} ] \tag {234}
$$
$$
\geq p. \tag {235}
$$
Thus,  $(D_j, +1) \in S_{p,\gamma, B_G}$ . We finish the proof.
Below are Lemmas used in the proof of Lemma E.7. In Lemma E.8, we calculate  $p$  used in  $S_{p,\gamma,B_G}$ . Lemma E.8 (Mixture of Gaussians: Geometry at Initialization. Lemma B.2 in [7]). Assume the same conditions as in Lemma E.7, recall for all  $i \in [m]$ ,  $\mathbf{w}_i^{(0)} \sim \mathcal{N}(0,\sigma_w^2 I_{d\times d})$ , over the random initialization, we have for all  $i \in [m], j \in [r]$ ,
$$
\Pr \left[ i \in S _ {D _ {j}, S u r e} \right] \geq \Theta \left(\frac {B _ {\mu 1}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / \left(2 B _ {\mu 1} ^ {2}\right)\right)}}\right). \tag {236}
$$
Proof of Lemma E.8. Recall for all  $l \in [r]$ ,  $\tilde{\mu}_l = \mu_l / \sqrt{d}$ .
WLOG, let  $j = r$ . For all  $l \in [r - 1]$ . We define  $Z_{1} = \{l \in [r - 1] : \langle D_{l}, D_{r} \rangle \geq -\theta\}$  and  $Z_{2} = \{l \in [r - 1] : -1 < \langle D_{l}, D_{r} \rangle < -\theta\}$ . WLOG, let  $Z_{1} = [r_{1}]$ ,  $Z_{2} = \{r_{1} + 1, \dots, r_{2}\}$ , where  $0 \leq r_{1} \leq r_{2} \leq r - 1$ . We define the following events
$$
\zeta_ {l} = \left\{\left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {l} \right\rangle \leq C _ {\text {S u r e}, 2} \mathbf {b} _ {i} \right\}, \hat {\zeta} _ {l} = \left\{\left| \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {l} \right\rangle \right| \leq C _ {\text {S u r e}, 2} \mathbf {b} _ {i} \right\}. \tag {237}
$$
We define space  $A = \operatorname{span}(\mu_1, \ldots, \mu_{r_1})$  and  $\hat{\mu}_r = P_{A^\perp} \mu_r$ , where  $P_{A^\perp}$  is the projection operator on the complementary space of  $A$ . For  $l \in Z_2$ , we also define  $\dot{\mu}_l = \mu_l - \frac{\langle \mu_l, \mu_r \rangle \mu_r}{\|\mu_r\|_2^2}$ , and the event
$$
\dot {\zeta} _ {l} = \left\{\left\langle \mathbf {w} _ {i} ^ {(0)}, \dot {\mu} _ {l} \right\rangle \leq C _ {S u r e, 2} \mathbf {b} _ {i} \right\}, \hat {\zeta} _ {l} = \left\{\left| \left\langle \mathbf {w} _ {i} ^ {(0)}, \dot {\mu} _ {l} \right\rangle \right| \leq C _ {S u r e, 2} \mathbf {b} _ {i} \right\}. \tag {238}
$$
For  $l \in Z_2$ , we have  $\mu_l = \dot{\mu}_l - \rho \mu_r$ , where  $\rho \geq 0$ . So  $\langle \mathbf{w}, \mu_l \rangle = \langle \mathbf{w}, \dot{\mu}_l \rangle - \rho \langle \mathbf{w}, \mu_r \rangle \leq \langle \mathbf{w}, \dot{\mu}_l \rangle$  when  $\langle \mathbf{w}, \mu_r \rangle \geq 0$ . As a result, we have
$$
\dot {\zeta} _ {l} \cap \left\{\left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \right\} \subseteq \zeta_ {l} \cap \left\{\left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \right\}. \tag {239}
$$
By Assumption 4.1, we have
$$
\frac {1}{2} \leq 1 - r \theta \leq 1 - r _ {1} \theta \leq \frac {\left\| \hat {\mu} _ {r} \right\| _ {2}}{\left\| \mu_ {r} \right\| _ {2}} \leq 1. \tag {240}
$$
$$
\begin{array}{l} \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {\text {S u r e}, 1} \mathbf {b} _ {i}, \zeta_ {1}, \dots , \zeta_ {r - 1} \right] (241) \\ = \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {\text {S u r e}, 1} \mathbf {b} _ {i}, \zeta_ {1}, \dots , \zeta_ {r _ {2}} \right] (242) \\ \geq \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i}, \zeta_ {1}, \dots , \zeta_ {r _ {1}}, \dot {\zeta} _ {r _ {1} + 1}, \dots , \dot {\zeta} _ {r _ {2}} \right] (243) \\ \geq \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i}, \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}}, \hat {\zeta} _ {r _ {1} + 1}, \dots , \hat {\zeta} _ {r _ {2}} \right] (244) \\ = \underbrace {\operatorname* {P r} \left[ \left\langle \mathbf {w} _ {i} ^ {(0)} , \mu_ {r} \right\rangle \geq C _ {S u r e , 1} \mathbf {b} _ {i} \Big | \hat {\zeta} _ {1} , \ldots , \hat {\zeta} _ {r _ {1}} , \hat {\zeta} _ {r _ {1} + 1} , \ldots , \hat {\zeta} _ {r _ {2}} \right]} _ {p _ {r}} \underbrace {\operatorname* {P r} \left[ \hat {\zeta} _ {1} , \ldots , \hat {\zeta} _ {r _ {1}} , \hat {\zeta} _ {r _ {1} + 1} , \ldots , \hat {\zeta} _ {r _ {2}} \right]} _ {\Pi_ {l \in [ r _ {2} ]} p _ {l}}. \\ \end{array}
$$
For the first condition in Definition E.6, we have,
$$
\begin{array}{l} p _ {r} = \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} \right\rangle \geq C _ {\text {S u r e}, 1} \mathbf {b} _ {i} \mid \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}}, \hat {\bar {\zeta}} _ {r _ {1} + 1}, \dots , \hat {\bar {\zeta}} _ {r _ {2}} \right] (245) \\ = \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \hat {\mu} _ {r} + \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \mid \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}} \right] (246) \\ \geq \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \hat {\mu} _ {r} + \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i}, \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq 0 | \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}} \right] (247) \\ = \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \hat {\mu} _ {r} + \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \mid \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq 0, \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}} \right] (248) \\ \cdot \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq 0 | \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}} \right] (249) \\ = \frac {1}{2} \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \hat {\mu} _ {r} + \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \mid \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq 0, \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}} \right] (250) \\ \geq \frac {1}{2} \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \hat {\mu} _ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \mid \left\langle \mathbf {w} _ {i} ^ {(0)}, \mu_ {r} - \hat {\mu} _ {r} \right\rangle \geq 0, \hat {\zeta} _ {1}, \dots , \hat {\zeta} _ {r _ {1}} \right] (251) \\ = \frac {1}{2} \Pr \left[ \left\langle \mathbf {w} _ {i} ^ {(0)}, \hat {\mu} _ {r} \right\rangle \geq C _ {S u r e, 1} \mathbf {b} _ {i} \right] (252) \\ \geq \Theta \left(\frac {\| \tilde {\mu} _ {r} \| _ {2}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(9 C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / (2 \| \tilde {\mu} _ {r} \| _ {2} ^ {2})\right)}}\right), (253) \\ \end{array}
$$
where the last equality following that  $\hat{\mu}_r$  is orthogonal with  $\mu_1, \ldots, \mu_{r_1}$  and the property of the standard Gaussian vector, and the last inequality follows Lemma F.6.
For the second condition in Definition E.6, by Lemma F.6, we have,
$$
p _ {1} = \Pr \left[ \hat {\zeta} _ {1} \right] = 1 - \Theta \left(\frac {\| \tilde {\mu} _ {1} \| _ {2}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / (8 \| \tilde {\mu} _ {1} \| _ {2} ^ {2})\right)}}\right) \tag {254}
$$
$$
p _ {2} = \Pr \left[ \hat {\zeta} _ {2} \mid \hat {\zeta} _ {1} \right] \geq \Pr \left[ \hat {\zeta} _ {2} \right] \geq 1 - \Theta \left(\frac {\| \tilde {\mu} _ {2} \| _ {2}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\left(C _ {b} ^ {2} \tau \sigma_ {B +} {} ^ {2} / (8 \| \tilde {\mu} _ {2} \| _ {2} ^ {2})\right)}}\right) \tag {255}
$$
$$
\vdots \tag {256}
$$