SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \nabla_ {\ell} f \left(\boldsymbol {W} ^ {(t, m)}\right) = \nabla_ {\ell} f \left(\boldsymbol {W} ^ {(t, m)}\right) - \nabla_ {\ell} f \left(\boldsymbol {W} ^ {\star}\right) \\ = \int_ {0} ^ {1} \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {(t, m)} + u \cdot \left(\boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star}\right)\right) d u \cdot \left(\boldsymbol {W} _ {\ell} ^ {(t, m)} - \boldsymbol {W} _ {\ell} ^ {\star}\right). \tag {48} \\ \end{array}
$$
For notational convenience, we use  $\pmb{H}$  to denote the integration as
$$
\boldsymbol {H} := \int_ {0} ^ {1} \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {(t, m)} + u \cdot \left(\boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star}\right)\right) d u. \tag {49}
$$
$$
\begin{array}{l} \left[ \begin{array}{c} \boldsymbol {W} ^ {(t, m + 1)} - \boldsymbol {W} ^ {\star} \\ \boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star} \end{array} \right] = \left[ \begin{array}{c c} \boldsymbol {I} - \eta \boldsymbol {H} & \beta \boldsymbol {I} \\ \boldsymbol {I} & \boldsymbol {0} \end{array} \right] \left[ \begin{array}{c} \boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star} \\ \boldsymbol {W} ^ {(t, m - 1)} - \boldsymbol {W} ^ {\star} \end{array} \right] \\ + \eta \left[ \begin{array}{c} \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) - g _ {t} ^ {(m)} (\boldsymbol {W} _ {\ell}; \boldsymbol {W} ^ {(t, m)}) \\ \mathbf {0} \end{array} \right]. \tag {50} \\ \end{array}
$$
Let  $\pmb{H} = \pmb{S}\pmb{\Lambda}\pmb{S}^T$  be the eigen-decomposition of  $\pmb{H}$ . Then, we define
$$
\boldsymbol {A} (\beta) := \left[ \begin{array}{c c} \boldsymbol {S} ^ {\top} & \boldsymbol {0} \\ \boldsymbol {0} & \boldsymbol {S} ^ {\top} \end{array} \right] \boldsymbol {A} (\beta) \left[ \begin{array}{c c} \boldsymbol {S} & \boldsymbol {0} \\ \boldsymbol {0} & \boldsymbol {S} \end{array} \right] = \left[ \begin{array}{c c} \boldsymbol {I} - \eta \boldsymbol {\Lambda} + \beta \boldsymbol {I} & \beta \boldsymbol {I} \\ \boldsymbol {I} & 0 \end{array} \right]. \tag {51}
$$
Since  $\left[ \begin{array}{ll}S & 0\\ 0 & S \end{array} \right]\left[ \begin{array}{ll}S^{\top} & 0\\ 0 & S^{\top} \end{array} \right] = \left[ \begin{array}{ll}I & 0\\ 0 & I \end{array} \right],$  we know  $A(\beta)$  and  $\left[ \begin{array}{cc}I - \eta \Lambda +\beta \mathbf{I} & \beta I\\ I & 0 \end{array} \right]$  share the same
eigenvalues. Let  $\lambda_i^{(\mathbf{A})}$  be the  $i$ -th eigenvalue of  $\pmb{H}_t^{(\ell)}$ , then the corresponding  $i$ -th eigenvalue of (51), denoted by  $\lambda_i^{(\mathbf{A})}$ , satisfies
$$
\left(\lambda_ {i} ^ {(\mathbf {A})} (\beta)\right) ^ {2} - \left(1 - \eta \lambda_ {i} ^ {(\mathbf {A})} + \beta\right) \lambda_ {i} ^ {(\mathbf {A})} (\beta) + \beta = 0. \tag {52}
$$
By simple calculation, we have
$$
\left| \lambda_ {i} ^ {(\mathbf {A})} (\beta) \right| = \left\{ \begin{array}{l} \sqrt {\beta}, \quad \text {i f} \quad \beta \geq \left(1 - \sqrt {\eta \lambda_ {i} ^ {(\mathbf {A})}}\right) ^ {2}, \\ \frac {1}{2} \left| (1 - \eta \lambda_ {i} ^ {(\mathbf {A})} + \beta) + \sqrt {(1 - \eta \lambda_ {i} ^ {(\mathbf {A})} + \beta) ^ {2} - 4 \beta} \right|, \text {o t h e r w i s e .} \end{array} \right. \tag {53}
$$
Specifically, we have
$$
\lambda_ {i} ^ {(\mathbf {A})} (0) > \lambda_ {i} ^ {(\mathbf {A})} (\beta), \quad \text {f o r} \quad \forall \beta \in \left(0, (1 - \eta \lambda_ {i} ^ {(\mathbf {\Lambda})}) ^ {2}\right), \tag {54}
$$
and  $\lambda_i^{(\mathbf{A})}$  achieves the minimum  $\lambda_i^{(\mathbf{A})\star} = \left|1 - \sqrt{\eta\lambda_i^{(\mathbf{A})}}\right|$  when  $\beta^{\star} = \left(1 - \sqrt{\eta\lambda_i^{(\mathbf{A})}}\right)^2$ . From Lemma 2, for any  $\pmb{a} \in \mathbb{R}^d$  with  $\| \pmb{a}\|_2 = 1$ , we have
$$
\begin{array}{l} \boldsymbol {a} ^ {\top} \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) \boldsymbol {a} = \int_ {0} ^ {1} \boldsymbol {a} ^ {\top} \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {(t, m)} + u \cdot \left(\boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star}\right)\right) \boldsymbol {a} \cdot d u \\ \leq \int_ {0} ^ {1} \lambda_ {\max } \| \boldsymbol {a} \| _ {2} ^ {2} d u = \lambda_ {\max }, \tag {55} \\ \end{array}
$$
$$
\begin{array}{l} \boldsymbol {a} ^ {\top} \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) \boldsymbol {a} = \int_ {0} ^ {1} \boldsymbol {a} ^ {\top} \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {(t, m)} + u \cdot (\boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star})\right) \boldsymbol {a} \cdot d u \\ \geq \int_ {0} ^ {1} \lambda_ {\min } \| \boldsymbol {a} \| _ {2} ^ {2} d u = \lambda_ {\min }, \\ \end{array}
$$
where  $\lambda_{\mathrm{max}}\approx \frac{1}{K}$ , and  $\lambda_{\mathrm{min}}\approx \frac{\rho}{K^2}$ . Therefore, we have
$$
\lambda_ {\min } ^ {(\mathbf {A})} \approx \frac {(1 - c _ {I}) \rho}{K ^ {2}}, \quad \text {a n d} \quad \lambda_ {\max } ^ {(\mathbf {A})} \approx \frac {1}{K}. \tag {56}
$$
Thus, when  $\eta \leq \frac{1}{2\lambda_{\max}^{(\Lambda)}} \lesssim K$ ,  $\| \mathbf{A}(\beta^{\star}) \|_2$  can be bounded by
$$
\left\| \boldsymbol {A} \left(\beta^ {\star}\right) \right\| _ {2} = 1 - \sqrt {\eta \cdot \lambda_ {\min } ^ {(\Lambda)}} \leq 1 - \sqrt {\frac {\left(1 - c _ {I}\right) \eta \rho}{K ^ {2}}}. \tag {57}
$$
$$
\begin{array}{l} \left\| \boldsymbol {W} _ {\ell} ^ {(t, m + 1)} - \boldsymbol {W} _ {\ell} ^ {\star} \right\| _ {2} \leq \left(1 - \sqrt {\frac {(1 - c _ {I}) \eta \rho}{K ^ {2}}}\right) \cdot \left\| \boldsymbol {W} _ {\ell} ^ {(t, m)} - \boldsymbol {W} _ {\ell} ^ {\star} \right\| _ {2} \\ + \eta \cdot \| \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) - g _ {t} ^ {(m)} (\boldsymbol {W} ^ {(t, m)}) \| _ {2} \tag {58} \\ \lesssim \left(1 - \left(1 - \frac {c _ {I}}{2}\right) \sqrt {\frac {\eta \rho}{K ^ {2}}}\right) \cdot \| \boldsymbol {W} _ {\ell} ^ {(t, m)} - \boldsymbol {W} _ {\ell} ^ {\star} \| _ {2} \\ + \eta \cdot \| \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) - g _ {t} ^ {(m)} (\boldsymbol {W} ^ {(t, m)}) \| _ {2}. \\ \end{array}
$$
Take the sum of (58) from  $\ell = 1$  to  $\ell = L$ , we have
$$
\begin{array}{l} \left\| \boldsymbol {W} ^ {(t, m + 1)} - \boldsymbol {W} ^ {\star} \right\| _ {2} \leq \left(1 - \left(1 - \frac {c _ {I}}{2}\right) \sqrt {\frac {\eta \rho}{K ^ {2}}}\right) \cdot \left\| \boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star} \right\| _ {2} (59) \\ + \eta \cdot \sum_ {\ell} ^ {L} \| \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) - g _ {t} ^ {(m)} (\boldsymbol {W} ^ {(t, m)}) \| _ {2}. (59) \\ \end{array}
$$
$$
\begin{array}{l} \left\| \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) - g _ {t} ^ {(m)} (\boldsymbol {W} _ {\ell}; \boldsymbol {W} ^ {(t, m)}) \right\| _ {2} \lesssim \frac {2 - \varepsilon_ {t}}{K} \sqrt {\frac {K _ {\ell} \log q}{N _ {t}}} \cdot \| \boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star} \| _ {2} \\ + \frac {(1 - \varepsilon_ {t} / 2) \gamma}{K} \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \tag {60} \\ + C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon\right) \cdot \frac {R _ {\max}}{1 - \gamma}. \\ \end{array}
$$
For some small constant  $c_{N}\geq 0$  , let
$$
\eta \cdot \frac {1}{K} \sqrt {\frac {K _ {\ell} \log q}{N _ {t}}} \leq \frac {c _ {N}}{L} \sqrt {\frac {\eta \rho}{K ^ {2}}}, \tag {61}
$$
which requires
$$
\begin{array}{l} N _ {t} \gtrsim c _ {N} ^ {- 2} \cdot \rho^ {- 1} \cdot \eta^ {- 1} \cdot L ^ {2} \cdot \max  _ {\ell} K _ {\ell} \cdot \log q \tag {62} \\ = c _ {N} ^ {- 2} \cdot \rho^ {- 1} \cdot L \cdot d \cdot \log q. \\ \end{array}
$$
Then, the sample complexity
$$
N = \sum_ {t = 1} ^ {T} N _ {t} \gtrsim c _ {N} ^ {- 2} \cdot \rho^ {- 1} \cdot L \cdot d \cdot \log q \cdot T. \tag {63}
$$
$$
\begin{array}{l} \left\| \boldsymbol {W} ^ {(t, m + 1)} - \boldsymbol {W} ^ {\star} \right\| _ {2} \leq \left(1 - \left(1 - (2 - \varepsilon_ {t}) c _ {N} - \frac {c _ {I}}{2}\right) \sqrt {\frac {\rho}{T K ^ {2}}}\right) \cdot \left\| \boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {\star} \right\| _ {2} \\ + \sqrt {\eta} \cdot \frac {(1 - \varepsilon_ {t} / 2) \gamma}{K} \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \tag {64} \\ + \eta \cdot C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon\right) \cdot \frac {R _ {\max}}{1 - \gamma}. \\ \end{array}
$$
By mathematical induction, when  $M = \log \gamma^{-1}$  and  $\eta = 1 / T = 1 / \Theta(N)$ , we have
$$
\begin{array}{l} \left\| \boldsymbol {W} ^ {(t, M)} - \boldsymbol {W} ^ {\star} \right\| _ {2} \\ \lesssim \sqrt {\frac {K ^ {2}}{N}} \cdot C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon_ {t}\right) \cdot \frac {R _ {\max}}{1 - \gamma} + (1 - \varepsilon_ {t} / 2) \gamma \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \\ \leq \frac {c _ {N} \cdot C _ {d} \cdot \left(C _ {t} + \left(1 - C _ {t}\right) \cdot \varepsilon_ {t}\right)}{K} \cdot \frac {R _ {\max }}{1 - \gamma} + (1 - \varepsilon_ {t} / 2) \gamma \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \tag {65} \\ \leq \frac {c _ {N} \cdot C _ {d} \cdot \left(C _ {\max } + \left(1 - C _ {\max }\right) \cdot \varepsilon_ {t}\right)}{K} \cdot \frac {R _ {\max }}{1 - \gamma} + (1 - \varepsilon_ {t} / 2) \gamma \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2}. \\ \end{array}
$$
From Algorithm 1, we know that  $\pmb{W}^{(t + 1,0)} = \pmb{W}^{(t,M)}$ . To guarantee that iteration converges to the ground truth  $\pmb{W}^{\star}$ , namely,  $\| \pmb{W}^{(t + 1,0)} - \pmb{W}^{\star}\|_{2} < \| \pmb{W}^{(t,0)} - \pmb{W}^{\star}\|$ , we need
$$
\varepsilon_ {t} \leq \frac {(1 - \gamma) ^ {2} \cdot K \cdot \| \boldsymbol {W} ^ {(t , 0)} - \boldsymbol {W} ^ {\star} \| _ {2}}{\left(1 - C _ {t}\right) \cdot c _ {N} \cdot C _ {d} \cdot R _ {\max }} - \frac {C _ {t}}{1 - C _ {t}}. \tag {66}
$$
To guarantee that  $\varepsilon_T\geq 0$  , then we have
$$
\left\| \boldsymbol {W} ^ {(T, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F} \gtrsim \frac {C _ {T} \cdot c _ {N} \cdot C _ {d} \cdot R _ {\max }}{(1 - \gamma) ^ {2} \cdot K}. \tag {67}
$$
Specifically, let
$$
\varepsilon_ {t} = \frac {c _ {\varepsilon} \cdot K \cdot \| \boldsymbol {W} ^ {(t , 0)} - \boldsymbol {W} ^ {\star} \| _ {2}}{\left(1 - C _ {t}\right) \cdot c _ {N} \cdot C _ {d} \cdot R _ {\max }} - \frac {C _ {t}}{1 - C _ {t}}, \tag {68}
$$
$$
\left\| \boldsymbol {W} ^ {(t + 1, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {2} \lesssim \gamma + c _ {\varepsilon} (1 - \gamma) \cdot \left\| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {2}, \tag {69}
$$
$$
a n d \quad \| \boldsymbol {W} ^ {(T, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \lesssim \left[ \gamma + c _ {\varepsilon} (1 - \gamma) \right] ^ {T} \cdot \| \boldsymbol {W} ^ {(0, 0)} - \boldsymbol {W} ^ {\star} \| _ {2},
$$
C Proof of Lemma 2
Lemma 2 provides the lower and upper bounds for the eigenvalues of the Hessian matrix of population risk function in (31). According to Weyl's inequality in Lemma 5, the eigenvalues of  $\nabla_{\ell}^{2}f(\cdot)$  at any fixed point  $\pmb{W}$  can be bounded in the form of (75). Therefore, we first provide the lower and upper bounds for  $\nabla_{\ell}^{2}f$  at the desired ground truth  $W^{\star}$ . Then, the bounds for  $\nabla_{\ell}^{2}f$  at any other point  $\pmb{W}$  is bounded through (31) by utilizing the conclusion in Lemma 6. Lemma 6 illustrates the distance between the Hessian matrix of  $f$  at  $\pmb{W}$  and  $W^{*}$ . Lemma 7 provides the lower bound of  $\mathbb{E}_{\pmb{x}}\big(\sum_{j = 1}^{K}\alpha_{j}^{\top}\frac{\partial Q}{\partial w_{\ell,k}} (\pmb{W}^{\star})\big)^{2}$  when  $\pmb{x}$  belongs to sub-Gaussian distribution, which is used in proving the lower bound of the Hessian matrix in (76).
Lemma 5 (Weyl's inequality, [5]). Let  $\pmb{B} = \pmb{A} + \pmb{E}$  be a matrix with dimension  $m \times m$ . Let  $\lambda_i(\pmb{B})$  and  $\lambda_i(\pmb{A})$  be the  $i$ -th largest eigenvalues of  $\pmb{B}$  and  $\pmb{A}$ , respectively. Then, we have
$$
\left| \lambda_ {i} (\boldsymbol {B}) - \lambda_ {i} (\boldsymbol {A}) \right| \leq \| \boldsymbol {E} \| _ {2}, \quad \forall \quad i \in [ m ]. \tag {70}
$$
Lemma 6. Let  $f(W)$  be the population risk function defined in (31). If  $W$  is close to  $W^{\star}$  such that
$$
\left\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \right\| _ {2} \lesssim \frac {\rho}{K} \tag {71}
$$
$$
\left\| \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) - \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {\star}\right) \right\| _ {2} \lesssim \frac {1}{K} \cdot \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2}. \tag {72}
$$
Lemma 7. Suppose the following assumptions hold:
1.  $\{\pmb{w}_j\}_{j=1}^K \in \mathbb{R}^{K_\ell}$  are linear independent,
2.  $p_H(\pmb {h}): \mathbb{R}^{K_\ell}\longrightarrow [01]$  be the probability density for  $\pmb{h}$  such that  $\mathbb{E}_h\| h\| _2^2\leq +\infty$
Let  $\alpha \in \mathbb{R}^{K_1K_2}$  be the unit vector defined in (34), we have
$$
\rho := \min  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \int_ {\mathcal {R}} \left(\sum_ {j = 1} ^ {K} \boldsymbol {\alpha} ^ {\top} \boldsymbol {h} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j} ^ {\top} \boldsymbol {h}\right)\right) ^ {2} p _ {H} (\boldsymbol {h}) \cdot d \boldsymbol {h} > 0, \tag {73}
$$
where  $\mathcal{R} \subset \mathbb{R}^{K_{\ell}}$  with  $\int_{\mathcal{R}} f_H(h) > 0$ . Moreover, if further assuming  $\mathcal{P}$  is Gaussian distribution and  $\mathcal{R} = \mathbb{R}^{K_{\ell}}$ , we have  $\rho > 0.091$ .
Lemma 8. Let  $h^{(\ell)}(W)$  be the function defined in (39). When  $W$  is sufficiently close to  $W^{\star}$ , i.e.,  $\| W - W^{\star}\|_{2}$  is smaller than some positive constant  $c < 1$ , we have
$$
\left\| \boldsymbol {h} ^ {(\ell)} (\boldsymbol {W}) \right\| _ {2} \lesssim \| \boldsymbol {x} \| _ {2},
$$
$$
\left\| \boldsymbol {h} ^ {(\ell)} (\boldsymbol {W}) - \boldsymbol {h} ^ {(\ell)} \left(\boldsymbol {W} ^ {\star}\right) \right\| _ {2} \lesssim \left\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \right\| _ {2} \cdot \left\| \boldsymbol {x} \right\| _ {2}. \tag {74}
$$
Proof of Lemma 2. Let  $\lambda_{\max}(\pmb{W})$  and  $\lambda_{\min}(\pmb{W})$  denote the largest and smallest eigenvalues of  $\nabla_{\ell}^{2}f(\pmb{W})$  at a point  $\pmb{W}$ , respectively. Then, from Lemma 5, we have
$$
\lambda_ {\max } (\boldsymbol {W}) \leq \lambda_ {\max } (\boldsymbol {W} ^ {\star}) + \| \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) - \nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) \| _ {2},
$$
$$
\lambda_ {\min } (\boldsymbol {W}) \geq \lambda_ {\min } \left(\boldsymbol {W} ^ {\star}\right) - \left\| \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) - \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {\star}\right) \right\| _ {2}. \tag {75}
$$
Then, we provide the lower bound of the Hessian matrix of the population function at  $\pmb{W}^{\star}$ . Let  $\mathcal{P}$  be the distribution for  $\pmb{h}^{(\ell)}(\pmb{W})$  when  $\pmb{x} \sim \mu_t$  with probability density function denoted as  $p_H$ . For any  $\alpha \in \mathbb{R}^{K_{\ell}K}$  defined in (34) with  $\| \alpha \|_2 = 1$ , we have
$$
\begin{array}{l} \min  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \boldsymbol {\alpha} ^ {\top} \nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) \boldsymbol {\alpha} \\ = \frac {1}{K ^ {2}} \min  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \mathbb {E} _ {\boldsymbol {h} \sim \mathcal {P}} \left(\sum_ {j = 1} ^ {K} \boldsymbol {\alpha} _ {j} ^ {\top} \boldsymbol {h} ^ {(\ell)} \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j} ^ {\star \top} \boldsymbol {h} ^ {(\ell)})\right) ^ {2} \\ = \frac {1}{K ^ {2}} \min  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \int_ {\mathbb {R} ^ {K _ {\ell} - 1}} \left(\sum_ {j = 1} ^ {K} \boldsymbol {\alpha} _ {j} ^ {\top} \boldsymbol {h} ^ {(\ell)} \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j} ^ {\star \top} \boldsymbol {h} ^ {(\ell)}\right)\right) ^ {2} p _ {H} \left(\boldsymbol {h} ^ {(\ell)}\right) \cdot d \mathbf {h} ^ {(\ell)} \tag {76} \\ = \frac {1}{K ^ {2}} \min  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \int_ {\left\{\boldsymbol {h} ^ {(\ell)} \mid \mathcal {I} _ {\ell , k} \neq 0 \right\}} \left(\sum_ {j = 1} ^ {K} \boldsymbol {\alpha} _ {j} ^ {\top} \boldsymbol {h} ^ {(\ell)} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j} ^ {\star \top} \boldsymbol {h} ^ {(\ell)})\right) ^ {2} p _ {H} (\boldsymbol {h} ^ {(\ell)}) \cdot d \mathbf {h} ^ {(\ell)} \\ \gtrsim \frac {\rho}{K ^ {2}}, \\ \end{array}
$$
where the last inequality comes from Lemma 7, and Lemma 7 holds since  $\pmb{h}^{(\ell)}$  belongs to sub-Gaussian distribution and  $\pmb{W}_{\ell}$  is full rank.
Next, the upper bound of  $\nabla_{\ell}^{2}f$  can be bounded as
$$
\begin{array}{l} \max _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \boldsymbol {\alpha} ^ {\top} \nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) \boldsymbol {\alpha} \\ = \frac {1}{K ^ {2}} \max  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \mathbb {E} _ {\boldsymbol {x}} \left(\sum_ {j = 1} ^ {K} \boldsymbol {\alpha} _ {j} ^ {\top} \boldsymbol {h} ^ {(\ell)} \cdot \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j} ^ {\star \top} \boldsymbol {h} ^ {(\ell)})\right) ^ {2} \\ = \frac {1}{K ^ {2}} \max  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \mathbb {E} _ {\boldsymbol {x}} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \boldsymbol {\alpha} _ {j _ {1}} ^ {\top} \boldsymbol {h} ^ {(\ell)} \cdot \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star \top} \boldsymbol {h} ^ {(\ell)}) \cdot \boldsymbol {\alpha} _ {j _ {2}} ^ {\top} \boldsymbol {h} ^ {(\ell)} \cdot \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {h} ^ {(\ell)}) \\ = \frac {1}{K ^ {2}} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \mathbb {E} _ {\boldsymbol {x}} \boldsymbol {\alpha} _ {j _ {1}} ^ {\top} \boldsymbol {h} ^ {(\ell)} \cdot \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star T} \boldsymbol {h} ^ {(\ell)}) \cdot \boldsymbol {\alpha} _ {j _ {2}} ^ {\top} \boldsymbol {h} ^ {(\ell)} \cdot \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \boldsymbol {h} ^ {(\ell)}) \\ \leq \frac {1}{K ^ {2}} \max  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \left[ \mathbb {E} _ {\boldsymbol {x}} (\boldsymbol {\alpha} _ {j _ {1}} ^ {\top} \boldsymbol {h} ^ {(\ell)}) ^ {4} \cdot \mathbb {E} (\phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star \top} \boldsymbol {h} ^ {(\ell)})) ^ {4} \cdot \mathbb {E} _ {\boldsymbol {x}} (\boldsymbol {\alpha} _ {j _ {2}} ^ {\top} \boldsymbol {h} ^ {(\ell)}) ^ {4} \cdot \mathbb {E} _ {\boldsymbol {x}} (\phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {h} ^ {(\ell)}) ^ {4} \right] ^ {1 / 4} \\ \leq \frac {1}{K ^ {2}} \max  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \left[ \mathbb {E} _ {\boldsymbol {x}} \left(\boldsymbol {\alpha} _ {j _ {1}} ^ {\top} \boldsymbol {x}\right) ^ {4} \cdot \mathbb {E} _ {\boldsymbol {x}} \left(\boldsymbol {\alpha} _ {j _ {2}} ^ {\top} \boldsymbol {x}\right) ^ {4} \right] ^ {1 / 4} \\ \leq \frac {3}{K ^ {2}} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \| \boldsymbol {\alpha} _ {j _ {1}} \| _ {2} \cdot \| \boldsymbol {\alpha} _ {j _ {2}} \| _ {2} \leq \frac {6}{K ^ {2}} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \frac {1}{2} \left(\| \boldsymbol {\alpha} _ {j _ {1}} \| _ {2} ^ {2} + \| \boldsymbol {\alpha} _ {j _ {2}} \| _ {2} ^ {2}\right) \\ = \frac {6}{K}. \tag {77} \\ \end{array}
$$
$$
\lambda_ {\max } \left(\boldsymbol {W} ^ {\star}\right) = \max  _ {\| \boldsymbol {\alpha} \| _ {2} = 1} \boldsymbol {\alpha} ^ {\top} \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {\star}; p\right) \boldsymbol {\alpha} \leq \frac {6}{K}. \tag {78}
$$
Then, given (71), we have
$$
\left\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \right\| _ {2} \lesssim \frac {2 \rho}{K}. \tag {79}
$$
Combining (79) and Lemma 6, we have
$$
\left\| \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) - \nabla_ {\ell} ^ {2} f \left(\boldsymbol {W} ^ {\star}\right) \right\| _ {2} \lesssim \frac {\rho}{K ^ {2}}. \tag {80}
$$
Therefore, from (80) and (75), we have
$$
\lambda_ {\max } (\boldsymbol {W}) \leq \lambda_ {\max } (\boldsymbol {W} ^ {\star}) + \| \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) - \nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) \| _ {2} \leq \frac {6}{K} + \frac {\rho}{2 K ^ {2}} \leq \frac {7}{K}, \tag {81}
$$
$$
\lambda_ {\min } (\boldsymbol {W}) \geq \lambda_ {\min } (\boldsymbol {W} ^ {\star}) - \| \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) - \nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) \| _ {2} \geq \frac {\rho}{K ^ {2}} - \frac {\rho}{2 K ^ {2}} = \frac {\rho}{2 K ^ {2}},
$$
D Proof of Lemma 3
Before illustrating the whole proof, we first introduce some preliminary lemmas and definitions. Lemma 9 is the concentration theorem for independent random matrices. The definitions of the sub-Gaussian and sub-exponential variables are summarized in Definitions 3 and 4, and it is easy to verify that any bounded variables belong to sub-Gaussian distribution. Lemmas 10 and 11 serve as the technical tools in bounding matrix norms under the framework of the confidence interval.
The error bound between  $\| \nabla_{\ell}f - g_t\| _2$  is divided into bounding  $I_{1},I_{2}$ , and  $I_{3}$  as shown in (91).  $I_{1}$  in (92) represent the deviation of the mean of several random variables to their expectation, which can be bounded through concentration inequality, i.e, Chernoff bound.  $I_{2}$  in (93) come from the inconsistency of "noisy" label in (8) and the "ground truth" label in the population risk function (31).  $I_{3}$  in (94) come from the data distribution shift defined in Definition 1.
Lemma 9 ([72], Theorem 1.6). Consider a finite sequence  $\{\mathbf{Z}_k\}$  of independent, random matrices with dimensions  $d_1 \times d_2$ . Assume that such a random matrix satisfies
$$
\mathbb {E} \left(\boldsymbol {Z} _ {k}\right) = 0 \quad a n d \quad \| \boldsymbol {Z} _ {k} \| \leq R \quad a l m o s t s u r e l y.
$$
$$
\delta^ {2} := \max  \left\{\left\| \sum_ {k} \mathbb {E} \left(\boldsymbol {Z} _ {k} \boldsymbol {Z} _ {k} ^ {\top}\right) \right\|, \left\| \sum_ {k} \mathbb {E} \left(\boldsymbol {Z} _ {k} ^ {\top} \boldsymbol {Z} _ {k}\right) \right\| \right\}.
$$
Then for all  $t \geq 0$ , we have
$$
\operatorname {P r o b} \left\{\left\| \sum_ {k} \boldsymbol {Z} _ {k} \right\| \geq t \right\} \leq \left(d _ {1} + d _ {2}\right) \exp \left(\frac {- t ^ {2} / 2}{\delta^ {2} + R t / 3}\right).
$$
Definition 3 (Definition 5.7, [74]). A random variable  $X$  is called a sub-Gaussian random variable if it satisfies
$$
\left(\mathbb {E} | X | ^ {p}\right) ^ {1 / p} \leq c _ {1} \sqrt {p} \tag {82}
$$
for all  $p \geq 1$  and some constant  $c_1 > 0$ . In addition, we have
$$
\mathbb {E} e ^ {s (X - \mathbb {E} X)} \leq e ^ {c _ {2} \| X \| _ {\psi_ {2}} ^ {2} s ^ {2}} \tag {83}
$$
for all  $s \in \mathbb{R}$  and some constant  $c_2 > 0$ , where  $\| X \|_{\phi_2}$  is the sub-Gaussian norm of  $X$  defined as  $\| X \|_{\psi_2} = \sup_{p \geq 1} p^{-1/2} (\mathbb{E}|X|^p)^{1/p}$ .
Moreover, a random vector  $\mathbf{X} \in \mathbb{R}^d$  belongs to the sub-Gaussian distribution if one-dimensional marginal  $\alpha^\top \mathbf{X}$  is sub-Gaussian for any  $\alpha \in \mathbb{R}^d$ , and the sub-Gaussian norm of  $\mathbf{X}$  is defined as  $\| \mathbf{X}\|_{\psi_2} = \sup_{\| \boldsymbol {\alpha}\| _2 = 1}\| \boldsymbol{\alpha}^\top \mathbf{X}\|_{\psi_2}$ .
Definition 4 (Definition 5.13, [74]). A random variable  $X$  is called a sub-exponential random variable if it satisfies
$$
\left(\mathbb {E} | X | ^ {p}\right) ^ {1 / p} \leq c _ {3} p \tag {84}
$$
for all  $p \geq 1$  and some constant  $c_3 > 0$ . In addition, we have
$$
\mathbb {E} e ^ {s (X - \mathbb {E} X)} \leq e ^ {c _ {4} \| X \| _ {\psi_ {1}} ^ {2} s ^ {2}} \tag {85}
$$
for  $s \leq 1 / \| X\|_{\psi_1}$  and some constant  $c_{4} > 0$ , where  $\| X\|_{\psi_1}$  is the sub-exponential norm of  $X$  defined as  $\| X\|_{\psi_1} = \sup_{p\geq 1}p^{-1}(\mathbb{E}|X|^p)^{1 / p}$ .