SlowGuess/NeurIPS_2023_OCR huggingface dataset:
iterations, where  $C_4$  can be taken as  $2/3$ . Therefore, we know that  $F_{-1}(\mathbf{W}^{(t,b)},\mathbf{x}_i),F_{+1}(\mathbf{W}^{(t,b)},\mathbf{x}_i) = O(1)$  in  $(0,0)\leq (t,b)\leq (T_2,0)$ . Thus, there exists a positive constant  $C$  such that  $-\ell_i^{\prime (t,b)}\geq C$  for  $0\leq t\leq T_2$ .
Recall that we denote  $\{i\in [n]|y_i = y\}$  as  $S_{y}$ , and we have the update rule
$$
\begin{array}{l} \gamma_ {j, r} ^ {(t, b + 1)} = \gamma_ {j, r} ^ {(t, b)} - \frac {\eta}{B m} \cdot \left[ \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {+}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \cdot \boldsymbol {\mu} \rangle) \right. \\ \left. - \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {-}} \ell_ {i} ^ {\prime (t, b)} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \cdot \boldsymbol {\mu} \right\rangle\right) \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}. \\ \end{array}
$$
For the growth of  $\gamma_{j,r}^{(t,b)}$ , if  $\langle \mathbf{w}_{j,r}^{(t,b)}, \boldsymbol{\mu} \rangle \geq 0$ , we have
$$
\begin{array}{l} \gamma_ {j, r} ^ {(t, b + 1)} = \gamma_ {j, r} ^ {(t, b)} - \frac {\eta}{B m} \cdot \left[ \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {+} \cap S _ {1}} \ell_ {i} ^ {\prime (t)} - \sum_ {i \in \mathcal {I} _ {t, b} \cap S _ {-} \cap S _ {1}} \ell_ {i} ^ {\prime (t)} \right] \| \boldsymbol {\mu} \| _ {2} ^ {2} \\ \geq \gamma_ {j, r} ^ {(t, b)} + \frac {\eta}{B m} \cdot \left[ C | \mathcal {I} _ {t, b} \cap S _ {+} \cap S _ {1} | - | \mathcal {I} _ {t, b} \cap S _ {-} \cap S _ {- 1} | \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}. \\ \end{array}
$$
Similarly, if  $\langle \mathbf{w}_{j,r}^{(t,b)},\pmb {\mu}\rangle <  0,$
$$
\gamma_ {j, r} ^ {(t, b + 1)} \geq \gamma_ {j, r} ^ {(t, b)} + \frac {\eta}{B m} \cdot \left[ C | \mathcal {I} _ {t, b} \cap S _ {+} \cap S _ {- 1} | - | \mathcal {I} _ {t, b} \cap S _ {-} \cap S _ {1} | \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}.
$$
Therefore, for  $t \in [T_2, T_1]$ , we have
$$
\begin{array}{l} \gamma_ {j, r} ^ {(t, 0)} \geq \sum_ {\left(t ^ {\prime}, b ^ {\prime}\right) <   (t, 0)} \frac {\eta}{B m} \left[ C \min  \left\{\left| \mathcal {I} _ {t ^ {\prime}, b ^ {\prime}} \cap S _ {+} \cap S _ {- 1} \right|, \left| \mathcal {I} _ {t ^ {\prime}, b ^ {\prime}} \cap S _ {+} \cap S _ {1} \right| \right\} - \left| \mathcal {I} _ {t, b} \cap S _ {-} \right| \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} \\ \geq \frac {\eta}{B m} \left(c _ {3} t c _ {4} H C \frac {B}{4} - T _ {1} n q\right) \| \boldsymbol {\mu} \| _ {2} ^ {2} \\ = \frac {\eta}{B m} (c _ {3} c _ {4} t C \frac {n}{4} - T _ {1} n q) \| \mu \| _ {2} ^ {2} \\ \geq \frac {\eta c _ {3} c _ {4} C t n \| \mu \| _ {2} ^ {2}}{8 B m} \tag {46} \\ \geq \frac {c _ {3} c _ {4} C C _ {4} n \| \mu \| _ {2} ^ {2}}{(P - 1) ^ {2} \sigma_ {p} ^ {2} d} = \Theta (n \cdot \mathrm {S N R} ^ {2}) = \Theta (\widehat {\gamma}), \\ \end{array}
$$
where the second inequality is due to Lemma B.6, the third inequality is due to  $q < \frac{C_4 C c_3 c_4}{8 C_3}$  in Condition 3.1.
And it follows directly from (45) that
$$
\overline {{\rho}} _ {j, r, i} ^ {(T _ {1}, 0)} \leq \frac {3 \eta (P - 1) ^ {2} \sigma_ {p} ^ {2} d T _ {1} H}{2 B m} = \frac {3 C _ {3}}{2}, \overline {{\rho}} _ {j, r, i} ^ {(T _ {1}, 0)} = O (1),
$$
C.2.2 Second Stage
By the signal-noise decomposition, at the end of the first stage, we have
$$
\mathbf {w} _ {j, r} ^ {(t, b)} = \mathbf {w} _ {j, r} ^ {(0, 0)} + j \gamma_ {j, r} ^ {(t, b)} \| \boldsymbol {\mu} \| _ {2} ^ {- 2} \boldsymbol {\mu} + \frac {1}{P - 1} \sum_ {i = 1} ^ {n} \overline {{\rho}} _ {j, r, i} ^ {(t, b)} \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {- 2} \boldsymbol {\xi} _ {i} + \frac {1}{P - 1} \sum_ {i = 1} ^ {n} \underline {{\rho}} _ {j, r, i} ^ {(t, b)} \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {- 2} \boldsymbol {\xi} _ {i}.
$$
for  $j \in [\pm 1]$  and  $r \in [m]$ . By the results we get in the first stage, we know that at the beginning of this stage, we have the following property holds:
-  $\overline{\rho}_{j,r^*,i}^{(T_1,0)} \geq 2$  for any  $r^* \in S_i^{(0,0)} = \{r \in [m] : \langle \mathbf{w}_{y_i,r}^{(0,0)}, \pmb{\xi}_i \rangle > \sigma_0 \sigma_p \sqrt{d} / \sqrt{2}\}$ ,  $j \in \{\pm 1\}$  and  $i \in [n]$  with  $y_i = j$ .
$\max_{j,r,i}|\underline{\rho}_{j,r,i}^{(T_1,0)}| = \max \{\beta ,O\big(n\sqrt{\log(n / \delta)}\log (T^*) / \sqrt{d}\big)\} .$
-  $\gamma_{j,r}^{(T_1,0)} = \Theta (\widehat{\gamma})$  for any  $j\in \{\pm 1\},r\in [m]$
where  $\widehat{\gamma} = n\cdot \mathrm{SNR}^2$  . Now we choose  $\mathbf{W}^{*}$  as follows
$$
\mathbf {w} _ {j, r} ^ {*} = \mathbf {w} _ {j, r} ^ {(0, 0)} + \frac {2 0 \log (2 / \epsilon)}{P - 1} \left[ \sum_ {i = 1} ^ {n} \mathbb {1} (j = y _ {i}) \cdot \frac {\boldsymbol {\xi} _ {i}}{\| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}} \right].
$$
Lemma C.10. Under the same conditions as Theorem 3.2, we have that  $\| \mathbf{W}^{(T_1,0)} - \mathbf{W}^*\| _F\leq$ $\widetilde{O} (m^{1 / 2}n^{1 / 2}(P - 1)^{-1}\sigma_p^{-1}d^{-1 / 2}(1 + \max \{\beta ,n\sqrt{\log(n / \delta)}\log (T^*) / \sqrt{d}\}))$
$$
\begin{array}{l} \| \mathbf {W} ^ {(T _ {1}, 0)} - \mathbf {W} ^ {*} \| _ {F} \\ \leq \left\| \mathbf {W} ^ {(T _ {1}, 0)} - \mathbf {W} ^ {(0, 0)} \right\| _ {F} + \left\| \mathbf {W} ^ {*} - \mathbf {W} ^ {(0, 0)} \right\| _ {F} \\ \leq O (\sqrt {m}) \max  _ {j, r} \gamma_ {j, r} ^ {(T _ {1}, 0)} \| \boldsymbol {\mu} \| _ {2} ^ {- 1} + \frac {1}{P - 1} O (\sqrt {m}) \max  _ {j, r} \left\| \sum_ {i = 1} ^ {n} \bar {\rho} _ {j, r, i} ^ {(T _ {1}, 0)} \cdot \frac {\boldsymbol {\xi} _ {i}}{\| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}} + \sum_ {i = 1} ^ {n} \underline {{\rho}} _ {j, r, i} ^ {(T _ {1}, 0)} \cdot \frac {\boldsymbol {\xi} _ {i}}{\| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2}} \right\| _ {2} \\ + O (m ^ {1 / 2} n ^ {1 / 2} \log (1 / \epsilon) (P - 1) ^ {- 1} \sigma_ {p} ^ {- 1} d ^ {- 1 / 2}) \\ = O \left(m ^ {1 / 2} \widehat {\gamma} \| \boldsymbol {\mu} \| _ {2} ^ {- 1}\right) + \widetilde {O} \left(m ^ {1 / 2} n ^ {1 / 2} (P - 1) ^ {- 1} \sigma_ {p} ^ {- 1} d ^ {- 1 / 2} \left(1 + \max  \left\{\beta , n \sqrt {\log (n / \delta)} \log \left(T ^ {*}\right) / \sqrt {d} \right\}\right)\right) \\ + O (m ^ {1 / 2} n ^ {1 / 2} \log (1 / \epsilon) (P - 1) ^ {- 1} \sigma_ {p} ^ {- 1} d ^ {- 1 / 2}) \\ = O \left(m ^ {1 / 2} n \cdot \operatorname {S N R} \cdot (P - 1) ^ {- 1} \sigma_ {p} ^ {- 1} d ^ {- 1 / 2} \left(1 + \max  \left\{\beta , n \sqrt {\log (n / \delta)} \log \left(T ^ {*}\right) / \sqrt {d} \right\}\right)\right) \\ + \widetilde {O} \left(m ^ {1 / 2} n ^ {1 / 2} \log (1 / \epsilon) \left(P - 1\right) ^ {- 1} \sigma_ {p} ^ {- 1} d ^ {- 1 / 2}\right) \\ = \widetilde {O} (m ^ {1 / 2} n ^ {1 / 2} (P - 1) ^ {- 1} \sigma_ {p} ^ {- 1} d ^ {- 1 / 2} (1 + \max \{\beta , n \sqrt {\log (n / \delta)} \log (T ^ {*}) / \sqrt {d} \})), \\ \end{array}
$$
where the first inequality is by triangle inequality, the second inequality and the first equality are by our decomposition of  $\mathbf{W}^{(T_1,0)}$ ,  $\mathbf{W}^*$  and Lemma B.1; the second equality is by  $n \cdot \mathrm{SNR}^2 = \Theta(\widehat{\gamma})$  and  $\mathrm{SNR} = \|\pmb{\mu}\| / (P - 1)\sigma_p d^{1/2}$ ; the third equality is by  $n^{1/2} \cdot \mathrm{SNR} = O(1)$ .
Lemma C.11. Under the same conditions as Theorem 3.2, we have that
$$
y _ {i} \langle \nabla f (\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}), \mathbf {W} ^ {*} \rangle \geq \log (2 / \epsilon)
$$
for all  $(T_1,0)\leq (t,b)\leq (T^*,0)$
Proof of Lemma C.11. Recall that  $f(\mathbf{W}^{(t,b)}) = (1 / m)\sum_{j,r}j \cdot [\sigma (\langle \mathbf{w}_{j,r}^{(t,b)},\widehat{y}_i\pmb {\mu}\rangle) + (P - 1)\sigma (\langle \mathbf{w}_{j,r}^{(t,b)},\pmb {\xi}_i\rangle)]$ , thus we have
$$
\begin{array}{l} y _ {i} \left\langle \nabla f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right), \mathbf {W} ^ {*} \right\rangle \\ = \frac {1}{m} \sum_ {j, r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \boldsymbol {\mu} \right\rangle\right) \left\langle y _ {i} \widehat {y} _ {i} \boldsymbol {\mu}, j \mathbf {w} _ {j, r} ^ {*} \right\rangle + \frac {P - 1}{m} \sum_ {j, r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle\right) \left\langle y _ {i} \boldsymbol {\xi} _ {i}, j \mathbf {w} _ {j, r} ^ {*} \right\rangle \\ = \frac {1}{m} \sum_ {j, r} \sum_ {i ^ {\prime} = 1} ^ {n} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle\right) 2 0 \log (2 / \epsilon) \mathbb {1} \left(j = y _ {i ^ {\prime}}\right) \cdot \frac {\left\langle \boldsymbol {\xi} _ {i ^ {\prime}} , \boldsymbol {\xi} _ {i} \right\rangle}{\left\| \boldsymbol {\xi} _ {i ^ {\prime}} \right\| _ {2} ^ {2}} \\ + \frac {1}{m} \sum_ {j, r} \sum_ {i ^ {\prime} = 1} ^ {n} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \boldsymbol {\mu} \rangle) 2 0 \log (2 / \epsilon) \mathbb {1} (j = y _ {i ^ {\prime}}) \cdot \frac {\langle \widehat {y} _ {i} \boldsymbol {\mu} , \boldsymbol {\xi} _ {i ^ {\prime}} \rangle}{\| \boldsymbol {\xi} _ {i ^ {\prime}} \| _ {2} ^ {2}} \\ + \frac {1}{m} \sum_ {j, r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \boldsymbol {\mu} \right\rangle\right) \left\langle y _ {i} \widehat {y} _ {i} \boldsymbol {\mu}, j \mathbf {w} _ {j, r} ^ {(0, 0)} \right\rangle + \frac {P - 1}{m} \sum_ {j, r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle\right) \left\langle y _ {i} \boldsymbol {\xi} _ {i}, j \mathbf {w} _ {j, r} ^ {(0, 0)} \right\rangle \\ \geq \frac {1}{m} \sum_ {j = y _ {i}, r} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle) 2 0 \log (2 / \epsilon) - \frac {1}{m} \sum_ {j, r} \sum_ {i ^ {\prime} \neq i} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle) 2 0 \log (2 / \epsilon) \cdot \frac {| \langle \boldsymbol {\xi} _ {i ^ {\prime}} , \boldsymbol {\xi} _ {i} \rangle |}{\| \boldsymbol {\xi} _ {i ^ {\prime}} \| _ {2} ^ {2}} \\ - \frac {1}{m} \sum_ {j, r} \sum_ {i ^ {\prime} = 1} ^ {n} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \boldsymbol {\mu} \rangle) 2 0 \log (2 / \epsilon) \cdot \frac {| \langle \widehat {y} _ {i} \boldsymbol {\mu} , \boldsymbol {\xi} _ {i ^ {\prime}} \rangle |}{\| \boldsymbol {\xi} _ {i ^ {\prime}} \| _ {2} ^ {2}} - \frac {1}{m} \sum_ {j, r} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \boldsymbol {\mu} \rangle) \beta \\ \end{array}
$$
$$
\begin{array}{l} \geq \underbrace{\frac{1}{m}\sum_{j = y_{i},r}\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t,b)},\boldsymbol{\xi}_{i}\rangle)20\log(2 / \epsilon)}_{\mathrm{I}_{9}} - \underbrace{\frac{1}{m}\sum_{j,r}\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t,b)},\boldsymbol{\xi}_{i}\rangle)20\log(2 / \epsilon)O\big(n\sqrt{\log(n / \delta)} / \sqrt{d}\big)}_{\mathrm{I}_{10}} \\ - \underbrace {\frac {1}{m} \sum_ {j , r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j , r} ^ {(t , b)} , \widehat {y} _ {i} \boldsymbol {\mu} \right\rangle\right) O \left(n \sqrt {\log (n / \delta)} \cdot \operatorname {S N R} \cdot d ^ {- 1 / 2}\right)} _ {\mathrm {I} _ {1 1}} - \underbrace {\frac {1}{m} \sum_ {j , r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j , r} ^ {(t , b)} , y _ {i} \boldsymbol {\mu} \right\rangle\right) \beta} _ {\mathrm {I} _ {1 2}}, \tag {47} \\ \end{array}
$$
where the first inequality is by Lemma B.2 and the last inequality is by Lemma B.1. Then, we will bound each term in (47) respectively.
For  $\mathrm{I}_{10},\mathrm{I}_{11},\mathrm{I}_{12},\mathrm{I}_{14}$  , we have that
$$
\left| \mathrm {I} _ {1 0} \right| \leq O \left(n \sqrt {\log (n / \delta)} / \sqrt {d}\right), \left| \mathrm {I} _ {1 1} \right| \leq O \left(n \sqrt {\log (n / \delta)} \cdot \operatorname {S N R} \cdot d ^ {- 1 / 2}\right), \tag {48}
$$
$$
\left| \mathrm {I} _ {1 2} \right| \leq O (\beta),
$$
For  $j = y_{i}$  and  $r\in S_i^{(0)}$  , according to Lemma C.3, we have
$$
\begin{array}{l} (P - 1) \left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle \geq (P - 1) \left\langle \mathbf {w} _ {j, r} ^ {(0, 0)}, \boldsymbol {\xi} _ {i} \right\rangle + \overline {{\rho}} _ {j, r, i} ^ {(t, b)} - 5 n \sqrt {\frac {\log \left(4 n ^ {2} / \delta\right)}{d}} \alpha \\ \geq 2 - \beta - 5 n \sqrt {\frac {\log (4 n ^ {2} / \delta)}{d}} \alpha \\ \geq 1. 5 - \beta > 0, \\ \end{array}
$$
where the first inequality is by Lemma C.3; the second inequality is by  $5n\sqrt{\frac{\log(4n^2 / \delta)}{d}}\leq 0.5$ ; and the last inequality is by  $\beta < 1.5$ . Therefore, for  $\mathrm{I}_9$ , according to the fourth statement of Proposition C.8, we have
$$
\mathrm {I} _ {9} \geq \frac {1}{m} \left| \widetilde {S} _ {i} ^ {(t, b)} \right| 2 0 \log (2 / \epsilon) \geq 2 \log (2 / \epsilon). \tag {49}
$$
By plugging (48) and (49) into (47) and according to triangle inequality we have
$$
y _ {i} \left\langle \nabla f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right), \mathbf {W} ^ {*} \right\rangle \geq \mathrm {I} _ {9} - | \mathrm {I} _ {1 0} | - | \mathrm {I} _ {1 1} | - | \mathrm {I} _ {1 2} | - | \mathrm {I} _ {1 4} | \geq \log (2 / \epsilon),
$$
Lemma C.12. Under Assumption 3.1, for  $(0,0) \leq (t,b) \leq (T^{*},0)$ , the following result holds.
$$
\left\| \nabla L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) \right\| _ {F} ^ {2} \leq O \left(\max  \left\{\left\| \boldsymbol {\mu} \right\| _ {2} ^ {2}, (P - 1) ^ {2} \sigma_ {p} ^ {2} d \right\}\right) L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right).
$$
Proof. We first prove that
$$
\left\| \nabla f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right) \right\| _ {F} = O \left(\max  \{\| \boldsymbol {\mu} \| _ {2}, (P - 1) \sigma_ {p} \sqrt {d} \}\right). \tag {50}
$$
Without loss of generality, we suppose that  $\widehat{y}_i = 1$ . Then we have that
$$
\begin{array}{l} \| \nabla f (\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}) \| _ {F} \leq \frac {1}{m} \sum_ {j, r} \left\| \left[ \sigma^ {\prime} \left(\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\mu} \rangle\right) \boldsymbol {\mu} + (P - 1) \sigma^ {\prime} \left(\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \rangle\right) \boldsymbol {\xi} _ {i} \right] \right\| _ {2} \\ \leq \frac {1}{m} \sum_ {j, r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\mu} \right\rangle\right) \| \boldsymbol {\mu} \| _ {2} + \frac {P - 1}{m} \sum_ {j, r} \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle\right) \| \boldsymbol {\xi} _ {i} \| _ {2} \\ \leq 4 \max  \{\| \boldsymbol {\mu} \| _ {2}, 2 (P - 1) \sigma_ {p} \sqrt {d} \}, \\ \end{array}
$$
where the first and second inequalities are by triangle inequality, the third inequality is by Lemma B.1.
Then we upper bound the gradient norm  $\| \nabla L_S(\mathbf{W}^{(t,b)})\| _F$  as:
$$
\| \nabla L _ {\mathcal {I} _ {t, b}} (\mathbf {W} ^ {(t, b)}) \| _ {F} ^ {2} \leq \left[ \frac {1}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \ell^ {\prime} \big (y _ {i} f (\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}) \big) \| \nabla f (\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}) \| _ {F} \right] ^ {2}
$$
$$
\begin{array}{l} \leq \left[ \frac {1}{B} \sum_ {i \in \mathcal {I} _ {t, b}} O (\max  \{\| \boldsymbol {\mu} \| _ {2}, (P - 1) \sigma_ {p} \sqrt {d} \}) \big (- \ell^ {\prime} \big (y _ {i} f (\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i})) \big) \right] ^ {2} \\ \leq O \left(\max  \left\{\| \boldsymbol {\mu} \| _ {2} ^ {2}, (P - 1) ^ {2} \sigma_ {p} ^ {2} d \right\}\right) \cdot \frac {1}{B} \sum_ {i \in \mathcal {I} _ {t, b}} - \ell^ {\prime} \left(y _ {i} f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right)\right) \\ \leq O \left(\max  \left\{\| \boldsymbol {\mu} \| _ {2} ^ {2}, (P - 1) \sigma_ {p} ^ {2} d _ {I} ^ {2} \right\} L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right)\right), \\ \end{array}
$$
where the first inequality is by triangle inequality, the second inequality is by (50), the third inequality is by Cauchy-Schwartz inequality and the last inequality is due to the property of the cross entropy loss  $-\ell' \leq \ell$ .
Lemma C.13. Under the same conditions as Theorem 3.2, we have that
$$
\left\| \mathbf {W} ^ {(t, b)} - \mathbf {W} ^ {*} \right\| _ {F} ^ {2} - \left\| \mathbf {W} ^ {(t + 1, b)} - \mathbf {W} ^ {*} \right\| _ {F} ^ {2} \geq \eta L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) - \eta \epsilon
$$
Proof of Lemma C.13. We have
$$
\begin{array}{l} \| \mathbf {W} ^ {(t, b)} - \mathbf {W} ^ {*} \| _ {F} ^ {2} - \| \mathbf {W} ^ {(t + 1, b)} - \mathbf {W} ^ {*} \| _ {F} ^ {2} \\ = 2 \eta \left\langle \nabla L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right), \mathbf {W} ^ {(t, b)} - \mathbf {W} ^ {*} \right\rangle - \eta^ {2} \| \nabla L _ {S} \left(\mathbf {W} ^ {(t, b)}\right) \| _ {F} ^ {2} \\ = \frac {2 \eta}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b)} \left[ y _ {i} f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right) - \left\langle \nabla f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right), \mathbf {W} ^ {*} \right\rangle \right] - \eta^ {2} \| \nabla L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) \| _ {F} ^ {2} \\ \geq \frac {2 \eta}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b)} \left[ y _ {i} f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right) - \log (2 / \epsilon) \right] - \eta^ {2} \| \nabla L _ {S} \left(\mathbf {W} ^ {(t, b)}\right) \| _ {F} ^ {2} \\ \geq \frac {2 \eta}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \left[ \ell \left(y _ {i} f \left(\mathbf {W} ^ {(t, b)}, \mathbf {x} _ {i}\right)\right) - \epsilon / 2 \right] - \eta^ {2} \| \nabla L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) \| _ {F} ^ {2} \\ \geq \eta L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) - \eta \epsilon , \\ \end{array}
$$
where the first inequality is by Lemma C.11; the second inequality is due to the convexity of the cross-entropy function; the last inequality is due to Lemma C.12.
Lemma C.14. Under the same conditions as Theorem 3.2, with probability  $1 - \delta$ , we have that
$$
\left| L _ {\mathcal {I} ^ {(t, b)}} \left(\mathbf {W} ^ {(t, b)}\right) - L _ {\mathcal {I} ^ {(t, b)}} \left(\mathbf {W} ^ {(t, 0)}\right) \right| \leq \epsilon ,
$$
$$
\begin{array}{l} \left| L _ {\mathcal {I} ^ {(t, b)}} \left(\mathbf {W} ^ {(t, b)}\right) - L _ {\mathcal {I} ^ {(t, b)}} \left(\mathbf {W} ^ {(t, 0)}\right) \right| \\ \leq \frac {1}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \left| \ell \left(y _ {i} f \left(\mathbf {W} ^ {(t, b)}, x _ {i}\right)\right) - \ell \left(y _ {i} f \left(\mathbf {W} ^ {(t, 0)}, x _ {i}\right)\right) \right| \\ \leq \frac {1}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \left| y _ {i} f \left(\mathbf {W} ^ {(t, b)}, x _ {i}\right) - y _ {i} f \left(\mathbf {W} ^ {(t, 0)}, x _ {i}\right) \right| \\ \leq \frac {1}{B} \sum_ {i \in \mathcal {I} _ {t, b}} \frac {1}{m} \sum_ {j, r} \left(\left| \left\langle \mathbf {w} _ {j, r} ^ {(t, b)} - \mathbf {w} _ {j, r} ^ {(t, 0)}, \boldsymbol {\mu} \right\rangle \right| + (P - 1) \left| \left\langle \mathbf {w} _ {j, r} ^ {(t, b)} - \mathbf {w} _ {j, r} ^ {(t, 0)}, \boldsymbol {\xi} _ {i} \right\rangle \right|\right) \\ \leq \frac {H \eta (P - 1)}{B m} \| \mu \| _ {2} \sigma_ {p} \sqrt {2 \log (6 n / \delta)} + \frac {H \eta (P - 1) ^ {2}}{B m} 2 \sigma_ {p} ^ {2} \sqrt {d \log (6 n ^ {2} / \delta)} \\ \leq \epsilon , \\ \end{array}
$$
where the first inequality is due to triangle inequality, the second inequality is due to  $|\ell_i^{\prime}|\leq 1$ , the third inequality is due to triangle inequality and the definition of neural networks, the forth inequality is due to parameter update rule (17) and Lemma B.1, and the fifth inequality is due to Condition 3.1.
Lemma C.15. Under the same conditions as Theorem 3.2, for all  $T_{1} \leq t \leq T^{*}$ , we have  $\max_{j,r,i} |\underline{\rho}_{j,r,i}^{(t,b)}| = \max \left\{O\left(\sqrt{\log(mn / \delta)} \cdot \sigma_{0}(P - 1)\sigma_{p}\sqrt{d}\right), O\left(n\sqrt{\log(n / \delta)}\log (T^{*}) / \sqrt{d}\right)\right\}$ . Besides,
$$
\frac {1}{(s - T _ {1}) H} \sum_ {(T _ {1}, 0) \leq (t, b) <   (s, 0)} L _ {\mathcal {I} _ {t, b}} (\mathbf {W} ^ {(t, b)}) \leq \frac {\| \mathbf {W} ^ {(T _ {1} , 0)} - \mathbf {W} ^ {*} \| _ {F} ^ {2}}{\eta (s - T _ {1}) H} + \epsilon
$$
for all  $T_{1} \leq t \leq T^{*}$ . Therefore, we can find an iterate with training loss smaller than  $2\epsilon$  within  $T = T_{1} + \left[\| \mathbf{W}^{(T_{1})} - \mathbf{W}^{*}\|_{F}^{2} / (\eta \epsilon)\right] = T_{1} + \widetilde{O}(\eta^{-1}\epsilon^{-1}mnd^{-1}\sigma_{p}^{-2})$  iterations.
Proof of Lemma C.15. Note that  $\max_{j,r,i}|\underline{\rho}_{j,r,i}^{(t)}| = \max \left\{O\left(\sqrt{\log(mn / \delta)}\right)\cdot \sigma_0(P - 1)\sigma_p\sqrt{d}\right.$ ,  $O\left(n\sqrt{\log(n / \delta)}\log (T^*) / \sqrt{d}\right)\}$  can be proved in the same way as Lemma C.9.
For any  $T_1 \leq s \leq T^*$ , by taking a summation of the inequality in Lemma C.13 and dividing  $(s - T_1)H$  on both sides, we obtain that
$$
\frac {1}{(s - T _ {1}) H} \sum_ {(T _ {1}, 0) \leq (t, b) <   (s, 0)} L _ {\mathcal {I} _ {t, b}} (\mathbf {W} ^ {(t, b)}) \leq \frac {\| \mathbf {W} ^ {(T _ {1} , 0)} - \mathbf {W} ^ {*} \| _ {F} ^ {2}}{\eta (s - T _ {1}) H} + \epsilon .
$$
According to the definition of  $T$ , we have
$$
\frac{1}{(T - T_{1})H}\sum_{(T_{1},0)\leq (t,b) <   (T,0)}L_{\mathcal{I}_{t,b}}(\mathbf{W}^{(t,b)})\leq 2\epsilon .
$$
Then there exists an epoch  $T_{1} \leq t \leq T^{*}$  such that
$$
\frac {1}{H} \sum_ {b = 0} ^ {H - 1} L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) \leq 2 \epsilon .
$$
Thus, according to Lemma C.14, we have
$$
L _ {S} (\mathbf {W} ^ {(t, 0)}) \leq 3 \epsilon .
$$
Lemma C.16. Under the same conditions as Theorem 3.2, we have
$$
\sum_ {i = 1} ^ {n} \bar {\rho} _ {j, r, i} ^ {(t, b)} / \gamma_ {j ^ {\prime}, r ^ {\prime}} ^ {(t, b)} = \Theta (\mathrm {S N R} ^ {- 2}) \tag {51}
$$
for all  $j,j^{\prime}\in \{\pm 1\}$ $r,r^{\prime}\in [m]$  and  $(T_{2},0)\leq (t,b)\leq (T^{*},0)$
Proof. Now suppose that there exists  $(0,0) < (\widetilde{T},0) \leq (T^{*},0)$  such that  $\sum_{i=1}^{n} \overline{\rho}_{j,r,i}^{(t,0)} / \gamma_{j',r'}^{(t,b)} = \Theta(\mathrm{SNR}^{-2})$  for all  $(0,0) < (t,0) < (\widetilde{T},0)$ . Then for  $\overline{\rho}_{j,r,i}^{(t,b)}$ , according to Lemma C.1, we have
$$
\begin{array}{l} \gamma_ {j, r} ^ {(t + 1, 0)} = \gamma_ {j, r} ^ {(t, b)} - \frac {\eta}{B m} \cdot \sum_ {b <   H} \left[ \sum_ {i \in S _ {+} \cap \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b _ {i} ^ {(t)})} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b _ {i} ^ {(t)})}, \widehat {y} _ {i} \cdot \boldsymbol {\mu} \rangle) \right. \\ \left. - \sum_ {i \in S _ {-} \cap \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b _ {i} ^ {(t)})} \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b _ {i} ^ {(t)})}, \widehat {y} _ {i} \cdot \boldsymbol {\mu} \rangle) \right] \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}, \\ \end{array}
$$
$$
\overline {{\rho}} _ {j, r, i} ^ {(t + 1, 0)} = \overline {{\rho}} _ {j, r, i} ^ {(t, 0)} - \frac {\eta (P - 1) ^ {2}}{B m} \cdot \ell_ {i} ^ {\prime (t, b _ {i} ^ {(t)})} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t, b _ {i} ^ {(t)})}, \boldsymbol {\xi} _ {i} \rangle) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = j),
$$
$$
\sum_ {i = 1} ^ {n} \bar {\rho} _ {j, r, i} ^ {(\widetilde {T}, 0)}
$$