SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \boldsymbol {w} _ {O} ^ {t + 1} - \boldsymbol {w} _ {O} ^ {0} \right\| _ {2} \leq \sum_ {s = 0} ^ {t} \left\| \boldsymbol {w} _ {O} ^ {s + 1} - \boldsymbol {w} _ {O} ^ {s} \right\| _ {2} = \gamma \sum_ {s = 0} ^ {t} \| \nabla_ {\boldsymbol {w} _ {O}} \Phi (\boldsymbol {W} ^ {s}) \| _ {2} \leq \gamma \rho \bar {\lambda} _ {V} \sum_ {s = 0} ^ {t} \sqrt {2 \ell (\boldsymbol {\theta} ^ {s})} \\ \leq \gamma \rho \bar {\lambda} _ {V} \sum_ {s = 0} ^ {t} \left(1 - \gamma \frac {\alpha^ {2}}{2}\right) ^ {s / 2} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \leq \rho \bar {\lambda} _ {V} \frac {2}{\alpha^ {2}} (1 - u ^ {2}) \frac {1 - u ^ {t + 1}}{1 - u} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \leq \rho \bar {\lambda} _ {V} \frac {4}{\alpha^ {2}} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \leq C _ {O}. \tag {31} \\ \end{array}
$$
By Weyl's inequality, this implies
$$
\left\| \boldsymbol {w} _ {O} ^ {t + 1} \right\| _ {2} \leq \left\| \boldsymbol {w} _ {O} ^ {0} \right\| _ {2} + C _ {O} = \bar {\lambda} _ {O}. \tag {32}
$$
Similarly, we can obtain that
$$
\left\| \boldsymbol {W} _ {K} ^ {t + 1} - \boldsymbol {W} _ {K} ^ {0} \right\| _ {\mathrm {F}} \leq 2 \rho \tau_ {0} \bar {\lambda} _ {Q} \bar {\lambda} _ {V} \bar {\lambda} _ {O} d _ {s} C _ {x} ^ {2} \frac {4}{\alpha^ {2}} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})}, \quad \left\| \boldsymbol {W} _ {K} ^ {t + 1} \right\| _ {2} \leq \bar {\lambda} _ {K}, \tag {33}
$$
$$
\left\| \boldsymbol {W} _ {V} ^ {t + 1} - \boldsymbol {W} _ {V} ^ {0} \right\| _ {\mathrm {F}} \leq \rho \bar {\lambda} _ {O} \frac {4}{\alpha^ {2}} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})}, \quad \left\| \boldsymbol {W} _ {V} ^ {t + 1} \right\| _ {2} \leq \bar {\lambda} _ {V}. \tag {34}
$$
Next, we will prove the fifth inequality in Eq. (25):
$$
\begin{array}{l} \left\| \boldsymbol {F} _ {\text {p r e}} ^ {t + 1} - \boldsymbol {F} _ {\text {p r e}} ^ {0} \right\| _ {\mathrm {F}} \\ \leq \rho \left\{\left\| \boldsymbol {W} _ {V} ^ {t + 1} - \boldsymbol {W} _ {V} ^ {0} \right\| _ {2} + \left\| \boldsymbol {W} _ {V} ^ {0} \right\| _ {2} 2 \tau_ {0} C _ {x} ^ {2} \left(\left\| \boldsymbol {W} _ {Q} ^ {t + 1} \right\| _ {2} \left\| \boldsymbol {W} _ {K} ^ {t + 1} - \boldsymbol {W} _ {K} ^ {0} \right\| _ {2} + \left\| \boldsymbol {W} _ {K} ^ {0} \right\| _ {2} \left\| \boldsymbol {W} _ {Q} ^ {t + 1} - \boldsymbol {W} _ {Q} ^ {0} \right\| _ {2}\right) \right\} \\ \leq \sum_ {s = 0} ^ {t} \gamma \rho \left\{\rho \bar {\lambda} _ {O} \sqrt {2 \ell (\boldsymbol {\theta} ^ {s})} + \bar {\lambda} _ {V} 2 \tau_ {0} C _ {x} ^ {2} d _ {s} \left[ 2 \rho \tau_ {0} \left(\bar {\lambda} _ {K} ^ {2} + \bar {\lambda} _ {Q} ^ {2}\right) \bar {\lambda} _ {V} \bar {\lambda} _ {O} C _ {x} ^ {2} d _ {s} \sqrt {2 \ell (\boldsymbol {\theta} ^ {s})} \right] \right\} \quad (b y \\ \leq \rho^ {2} \bar {\lambda} _ {O} \left[ 1 + 4 \tau_ {0} ^ {2} C _ {x} ^ {4} d _ {s} ^ {2} \bar {\lambda} _ {V} ^ {2} \left(\bar {\lambda} _ {Q} ^ {2} + \bar {\lambda} _ {K} ^ {2}\right) \right] \frac {1 6}{\alpha^ {2}} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \\ = \rho^ {2} \frac {z}{\bar {\lambda} _ {O}} \frac {1 6}{\alpha^ {2}} \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \\ \leq \frac {1}{2} \alpha , \quad (\text {b y E q .} (3)) \\ \end{array}
$$
where the first inequality holds by Lemma 9. This result further implies that  $\sigma_{\mathrm{min}}\left(\boldsymbol{F}_{\mathrm{pre}}^{t + 1}\right) \geq \frac{1}{2}\alpha$  by Weyl's inequality. It remains to prove the last inequality in Eq. (25) holds for step  $t + 1$ .
We start by proving the Lipschitz constant for the gradient of the loss restricted to the interval  $[\pmb{\theta}^{\top},\pmb{\theta}^{t + 1}]$ . We define  $\pmb{\theta}^{t + \phi}\coloneqq \pmb{\theta}^{\top} + \phi (\pmb{\theta}^{t + 1} - \pmb{\theta}^{\top})$ , for  $\phi \in [0,1]$ . Then, by triangle inequality and Cauchy-Schwarz inequality, we have:
$$
\begin{array}{l} \left\| \nabla_ {\boldsymbol {\theta}} \ell (\boldsymbol {\theta} ^ {t + \phi}) - \nabla_ {\boldsymbol {\theta}} \ell (\boldsymbol {\theta} ^ {\top}) \right\| _ {2} \\ = \left\| \nabla_ {\boldsymbol {\theta}} \boldsymbol {f} ^ {t + \phi} \cdot (\boldsymbol {f} ^ {t + \phi} - \boldsymbol {y}) - \nabla_ {\boldsymbol {\theta}} \boldsymbol {f} ^ {t} \cdot (\boldsymbol {f} ^ {t} - \boldsymbol {y}) \right\| _ {2} \tag {35} \\ \leq \left\| \boldsymbol {f} ^ {t + \phi} - \boldsymbol {f} ^ {t} \right\| _ {2} \left\| \nabla_ {\theta} \boldsymbol {f} ^ {t + \phi} \right\| _ {2} + \left\| \nabla_ {\theta} \boldsymbol {f} ^ {t + \phi} - \nabla_ {\theta} \boldsymbol {f} ^ {t} \right\| _ {2} \left\| \boldsymbol {f} ^ {t} - \boldsymbol {y} \right\| _ {2} \\ \leq \left\| \boldsymbol {f} ^ {t + \phi} - \boldsymbol {f} ^ {t} \right\| _ {2} \left\| \nabla_ {\boldsymbol {\theta}} \boldsymbol {f} ^ {t + \phi} \right\| _ {2} + 2 \left\| \nabla_ {\boldsymbol {\theta}} \boldsymbol {f} ^ {t + \phi} - \nabla_ {\boldsymbol {\theta}} \boldsymbol {f} ^ {t} \right\| _ {2} \ell (\boldsymbol {\theta} ^ {0}), \\ \end{array}
$$
where the last inequality is by induction rule. Then, we bound each term separately. Note that:
$$
\begin{array}{l} \left\| \boldsymbol {W} _ {Q} ^ {t + \phi} - \boldsymbol {W} _ {Q} ^ {0} \right\| _ {\mathrm {F}} \leq \left\| \boldsymbol {W} _ {Q} ^ {t + \phi} - \boldsymbol {W} _ {Q} ^ {t} \right\| _ {\mathrm {F}} + \sum_ {s = 0} ^ {t - 1} \left\| \boldsymbol {W} _ {Q} ^ {s + 1} - \boldsymbol {W} _ {Q} ^ {s} \right\| _ {\mathrm {F}} \\ = \phi \gamma \left\| \nabla_ {\boldsymbol {W} _ {Q}} \ell (\boldsymbol {\theta} ^ {\top}) \right\| _ {\mathrm {F}} + \gamma \sum_ {s = 0} ^ {t - 1} \left\| \nabla_ {\boldsymbol {W} _ {Q}} \ell (\boldsymbol {\theta} ^ {s}) \right\| _ {\mathrm {F}} \leq \gamma \sum_ {s = 0} ^ {t} \left\| \nabla_ {\boldsymbol {W} _ {Q}} \ell (\boldsymbol {\theta} ^ {s}) \right\| _ {\mathrm {F}}. \\ \end{array}
$$
Then following exact the same step as in Eqs. (26), (27) and (30), we have:  $\left\| \boldsymbol{W}_Q^{t + \phi}\right\|_2 \leq \bar{\lambda}_Q$ . By the same method, we have:  $\left\| \boldsymbol{W}_K^{t + \phi}\right\|_2 \leq \bar{\lambda}_K$ ,  $\left\| \boldsymbol{W}_V^{t + \phi}\right\|_2 \leq \bar{\lambda}_V$ ,  $\left\| \boldsymbol{w}_O^{t + \phi}\right\|_2 \leq \bar{\lambda}_O$ . Now we proceed to bound the first term in Eq. (35).
$$
\begin{array}{l} \left\| \boldsymbol {f} ^ {t + \phi} - \boldsymbol {f} ^ {t} \right\| _ {2} \\ \leq \rho \left\| \boldsymbol {W} _ {V} ^ {t + \phi} \right\| _ {2} \left\| \boldsymbol {w} _ {O} ^ {t + \phi} - \boldsymbol {w} _ {O} ^ {t} \right\| _ {2} + \left\| \boldsymbol {w} _ {O} ^ {t} \right\| _ {2} \left\| \boldsymbol {F} _ {\mathrm {p r e}} ^ {t + \phi} - \boldsymbol {F} _ {\mathrm {p r e}} ^ {t} \right\| _ {2} \\ \leq \rho \left\| \boldsymbol {W} _ {V} ^ {t + \phi} \right\| _ {2} \left\| \boldsymbol {w} _ {O} ^ {t + \phi} - \boldsymbol {w} _ {O} ^ {t} \right\| _ {2} \\ \left. \right. + \rho \| \boldsymbol {w} _ {O} ^ {t} \| _ {2} \left(\left\| \boldsymbol {W} _ {V} ^ {t + \phi} - \boldsymbol {W} _ {V} ^ {t} \right\| _ {2} + \left\| \boldsymbol {W} _ {V} ^ {t} \right\| _ {2} 2 \tau_ {0} C _ {x} ^ {2} d _ {s} \left(\left\| \boldsymbol {W} _ {Q} ^ {t + \phi} \right\| _ {2} \left\| \boldsymbol {W} _ {K} ^ {t + \phi} - \boldsymbol {W} _ {K} ^ {t} \right\| _ {2} + \left\| \boldsymbol {W} _ {K} ^ {t} \right\| _ {2} \left\| \boldsymbol {W} _ {Q} ^ {t + \phi} - \boldsymbol {W} _ {Q} ^ {t} \right\| _ {2}\right)\right) \\ \leq \rho \bar {\lambda} _ {V} \left\| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \right\| _ {2} + \rho \bar {\lambda} _ {O} \left(1 + \bar {\lambda} _ {V} 2 \tau_ {0} C _ {x} ^ {2} d _ {s} (\bar {\lambda} _ {Q} + \bar {\lambda} _ {K})\right) \left\| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \right\| _ {2} \\ = \rho (\bar {\lambda} _ {V} + \bar {\lambda} _ {O} + \bar {\lambda} _ {V} 2 \tau_ {0} C _ {x} ^ {2} d _ {s} (\bar {\lambda} _ {Q} + \bar {\lambda} _ {K})) \| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \| _ {2} \\ \triangleq c _ {1} \left\| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \right\| _ {2}, \tag {36} \\ \end{array}
$$
where the first inequality is by Lemma 10, the second inequality is by Lemma 9. Next, the second term in Eq. (35) can be bounded by Lemma 13. The third term in Eq. (35) can be bounded by Lemma 14. As a result, Eq. (35) has the following upper bound:
$$
\left\| \nabla_ {\boldsymbol {\theta}} \ell \left(\boldsymbol {\theta} ^ {t + \phi}\right) - \nabla_ {\boldsymbol {\theta}} \ell \left(\boldsymbol {\theta} ^ {\top}\right) \right\| _ {2} \leq c _ {1} c _ {2} \left\| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \right\| _ {2} + 2 c _ {3} \ell \left(\boldsymbol {\theta} ^ {0}\right) \left\| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \right\| _ {2} \triangleq C \left\| \boldsymbol {\theta} ^ {t + \phi} - \boldsymbol {\theta} ^ {\top} \right\| _ {2}, \tag {37}
$$
where  $c_{1}, c_{2}, c_{3}$  are defined at Eqs. (11), (14) and (36), and we further define  $C \triangleq c_{1}c_{2} + 2c_{3}\ell(\theta^{0})$ . Lastly, by applying Lemma 4.3 in Nguyen and Mondelli [2020] and Eq. (37), we have:
$$
\begin{array}{l} \ell (\boldsymbol {\theta} ^ {t + 1}) \leq \ell (\boldsymbol {\theta} ^ {\top}) + \langle \nabla_ {\boldsymbol {\theta}} \ell (\boldsymbol {\theta} ^ {\top}), \boldsymbol {\theta} ^ {t + 1} - \boldsymbol {\theta} ^ {\top} \rangle + \frac {C}{2} \left\| \boldsymbol {\theta} ^ {t + 1} - \boldsymbol {\theta} ^ {\top} \right\| _ {\mathrm {F}} ^ {2} \\ = \ell (\pmb {\theta} ^ {\top}) - \gamma \left\| \nabla_ {\pmb {\theta}} \ell (\pmb {\theta} ^ {\top}) \right\| _ {\mathrm {F}} ^ {2} + \frac {C}{2} \gamma^ {2} \left\| \nabla_ {\pmb {\theta}} \ell (\pmb {\theta} ^ {\top}) \right\| _ {\mathrm {F}} ^ {2} \\ \leq \ell (\boldsymbol {\theta} ^ {\top}) - \frac {1}{2} \gamma \| \nabla_ {\boldsymbol {\theta}} \ell (\boldsymbol {\theta} ^ {\top}) \| _ {\mathrm {F}} ^ {2} \quad (\text {B y} \\ \leq \ell (\boldsymbol {\theta} ^ {\top}) - \frac {1}{2} \gamma \left\| \nabla_ {\boldsymbol {w} _ {o}} \ell (\boldsymbol {\theta} ^ {\top}) \right\| _ {2} ^ {2} \\ = \ell (\pmb {\theta} ^ {\top}) - \frac {1}{2} \gamma \left\| (\pmb {F} _ {\mathrm {p r e}} ^ {t}) ^ {\top} (\pmb {f} ^ {t} - \pmb {y}) \right\| _ {2} ^ {2} \\ \leq \ell (\boldsymbol {\theta} ^ {\top}) - \frac {1}{2} \gamma \left(\sigma_ {\min } \left(\boldsymbol {F} _ {\text {p r e}} ^ {t}\right)\right) ^ {2} \left\| \boldsymbol {f} ^ {t} - \boldsymbol {y} \right\| _ {2} ^ {2} \\ \leq (1 - \gamma \frac {\alpha}{2}) \ell (\boldsymbol {\theta} ^ {\top}) \quad (\text {B y i n d u c t i o n a s s u m p t i o n}), \\ \end{array}
$$
Lemma 15. Let  $\Phi = [X_1^\top \beta_{1,1},\dots,X_N^\top \beta_{1,N}]^\top \in \mathbb{R}^{N\times d}$ , where  $\beta_{1,n} = \sigma_s\left(\tau_0\pmb{X}_n^{(1,:)}\pmb{W}_Q^\top \pmb{W}_K\pmb{X}_n^\top\right)^\top$ ,  $n \in [N]$ , then under Assumptions 2 and 3, with probability at
least  $1 - \exp \left(-\Omega ((N - 1)^{-\hat{c}}d_s^{-1})\right)$ , one has:  $\lambda_0\coloneqq \lambda_{\mathrm{min}}\left(\mathbb{E}_{\boldsymbol {w}\sim \mathcal{N}(0,\eta_V\mathbb{I}_d)}[\sigma_r(\boldsymbol {\Phi}\boldsymbol {w})\sigma_r(\boldsymbol {\Phi}\boldsymbol {w})^T ]\right)\geq$ $\Theta (\eta_V / d_s)$
Proof. Due to Assumption 2, for any data  $\mathbf{X}$ , the matrix  $\mathbf{X}\mathbf{X}^{\top}$  is positive definite and thus has positive minimum eigenvalue. We denote it as  $\lambda_{\min}(\mathbf{X}\mathbf{X}^{\top}) \geq C_{\lambda}$ .
According to [Nguyen et al., 2021, Lemma 5.3], using the Hermite expansion of  $\sigma_r$ , one has:
$$
\lambda_ {0} \geq \eta_ {V} \mu \left(\sigma_ {r}\right) ^ {2} \lambda_ {\min } \left(\Phi \Phi^ {\top}\right), \tag {38}
$$
where  $\mu (\sigma_r)$  is the 1-st Hermite coefficient of ReLU satisfying  $\mu (\sigma_r) > 0$
Now we proceed to provide a lower bound for  $\lambda_{\mathrm{min}}(\Phi \Phi^{\top})$ . For notational simplicity, define:
$$
\boldsymbol {B} _ {i j} = \boldsymbol {\beta} _ {1, i} \boldsymbol {\beta} _ {1, j} ^ {\top} \in \mathbb {R} ^ {d _ {s} \times d _ {s}}, \boldsymbol {C} _ {i j} = \boldsymbol {X} _ {i} \boldsymbol {X} _ {j} ^ {\top} \in \mathbb {R} ^ {d _ {s} \times d _ {s}}.
$$
Then we can rewrite  $\Phi \Phi^{\top}$  as follows:
$$
\boldsymbol {\Phi} \boldsymbol {\Phi} ^ {\top} = \left[ \begin{array}{c c c c} \mathrm {T r a c e} (\boldsymbol {B} _ {1 1} ^ {\top} \boldsymbol {C} _ {1 1}) & \mathrm {T r a c e} (\boldsymbol {B} _ {1 2} ^ {\top} \boldsymbol {C} _ {1 2}) & \dots & \mathrm {T r a c e} (\boldsymbol {B} _ {1 N} ^ {\top} \boldsymbol {C} _ {1 N}) \\ \mathrm {T r a c e} (\boldsymbol {B} _ {2 1} ^ {\top} \boldsymbol {C} _ {2 1}) & \mathrm {T r a c e} (\boldsymbol {B} _ {2 2} ^ {\top} \boldsymbol {C} _ {2 2}) & \dots & \mathrm {T r a c e} (\boldsymbol {B} _ {2 N} ^ {\top} \boldsymbol {C} _ {2 N}) \\ \vdots & \vdots & \vdots & \vdots \\ \mathrm {T r a c e} (\boldsymbol {B} _ {N 1} ^ {\top} \boldsymbol {C} _ {N 1}) & \mathrm {T r a c e} (\boldsymbol {B} _ {N 2} ^ {\top} \boldsymbol {C} _ {N 2}) & \dots & \mathrm {T r a c e} (\boldsymbol {B} _ {N N} ^ {\top} \boldsymbol {C} _ {N N}) \end{array} \right].
$$
By Gershgorin circle theorem [Gershgorin, 1931], there exists  $k \in [N]$  such that:
$$
\lambda_ {\min } \left(\Phi \Phi^ {\top}\right) \geq \operatorname {T r a c e} \left(\boldsymbol {B} _ {k k} ^ {\top} \boldsymbol {C} _ {k k}\right) - \sum_ {j \neq k} \operatorname {T r a c e} \left(\boldsymbol {B} _ {k j} ^ {\top} \boldsymbol {C} _ {k j}\right). \tag {39}
$$
Using Von Neumann's trace inequality [Mirsky, 1975] and noting that  $B_{kj}$  is a rank one matrix, one has:
$$
\begin{array}{l} \operatorname {T r a c e} \left(\boldsymbol {B} _ {k j} ^ {\top} \boldsymbol {C} _ {k j}\right) \leq \sigma_ {\max } \left(\boldsymbol {B} _ {k j}\right) \sigma_ {\max } \left(\boldsymbol {C} _ {k j}\right) = \underbrace {\| \beta_ {1 , k} \| _ {2} \| \beta_ {1 , j} \| _ {2}} \sqrt {\lambda_ {\max } \left(\boldsymbol {C} _ {k j} \boldsymbol {C} _ {k j} ^ {\top}\right)} \tag {40} \\ \leq \| \boldsymbol {\beta} _ {1, k} \| _ {2} \sqrt {\mathrm {T r a c e} (\boldsymbol {C} _ {k j} \boldsymbol {C} _ {k j} ^ {\top})} = \| \boldsymbol {\beta} _ {1, k} \| _ {2} \sqrt {\langle \boldsymbol {X} _ {k} ^ {\top} \boldsymbol {X} _ {k} , \boldsymbol {X} _ {j} ^ {\top} \boldsymbol {X} _ {j} \rangle}, \\ \end{array}
$$
where we use the definition of the inner product between two matrices and  $\| \beta_{1,j}\| _2\leq 1$ . By Assumption 2, we have  $\lambda_{\mathrm{min}}(\pmb {X}_k\pmb {X}_k^\top)\geq C_\lambda$ , where  $C_\lambda$  is some positive constant. By setting  $t\coloneqq \| \beta_{1,k}\|_2^2 C_\lambda^2 /(N - 1)^2$  in Assumption 3, with probability at least  $1 - \exp (-\Omega ((N - 1)^{-\hat{c}}))$ , one has
$$
\sqrt {\langle \boldsymbol {X} _ {k} ^ {\top} \boldsymbol {X} _ {k} , \boldsymbol {X} _ {j} ^ {\top} \boldsymbol {X} _ {j} \rangle} \leq \| \beta_ {1, k} \| _ {2} C _ {\lambda} (N - 1) ^ {- 1}, \quad \forall j \neq k.
$$
Plugging back Eq. (39) and Eq. (40), we obtain:
$$
\begin{array}{l} \lambda_ {\min } \left(\boldsymbol {\Phi} \boldsymbol {\Phi} ^ {\top}\right) \geq \operatorname {T r a c e} \left(\boldsymbol {B} _ {k k} ^ {\top} \boldsymbol {C} _ {k k}\right) - C _ {\lambda} \| \boldsymbol {\beta} _ {1, k} \| _ {2} ^ {2} \geq \lambda_ {\min } \left(\boldsymbol {C} _ {k k}\right) \operatorname {T r a c e} \left(\boldsymbol {B} _ {k k}\right) - C _ {\lambda} \| \boldsymbol {\beta} _ {1, k} \| _ {2} ^ {2} \tag {41} \\ = \lambda_ {\mathrm {m i n}} (\pmb {X} _ {k} \pmb {X} _ {k} ^ {\top}) \| \pmb {\beta} _ {1, k} \| _ {2} ^ {2} - C _ {\lambda} \| \pmb {\beta} _ {1, k} \| _ {2} ^ {2} \geq \Theta (\| \pmb {\beta} _ {1, k} \| _ {2} ^ {2}) \geq \Theta (1 / d _ {s}), \\ \end{array}
$$
where the last inequality is by the lower bound of  $\beta_{1,k}$  in Lemma 8. Lastly, plugging the lower bound of  $\lambda_{\min}(\Phi \Phi^{\top})$  back Eq. (38) finishes the proof.
Remark: The estimation of  $\lambda_0$  is actually tight because its upper bound is also in a constant order. To be specific, denote  $\pmb{G} \coloneqq \sigma_r(\pmb{\Phi}\pmb{w})\sigma_r(\pmb{\Phi}\pmb{w})^\top$ , we have
$$
\lambda_ {0} := \lambda_ {\min } \left(\mathbb {E} _ {\boldsymbol {w}} \boldsymbol {G}\right) \leq \frac {\operatorname {t r} \left(\mathbb {E} _ {\boldsymbol {w}} \boldsymbol {G}\right)}{N} = \frac {\sum_ {n = 1} ^ {N} \mathbb {E} _ {\boldsymbol {w}} \left[ \sigma_ {r} \left(\boldsymbol {\Phi} ^ {(n , :)} \boldsymbol {w}\right) \right] ^ {2}}{N}, \tag {42}
$$
where  $\Phi^{(n;;)} = \beta_{1,n}^{\top}\mathbf{X}_n$ . Next, by Liao and Couillet [2018] (Sec.A in Supplementary Material):
$$
\mathbb {E} _ {\boldsymbol {w}} \left[ \sigma_ {r} \left(\boldsymbol {\Phi} ^ {(n,:)} \boldsymbol {w}\right) \right] ^ {2} = \frac {\eta_ {V}}{2 \pi} \left\| \boldsymbol {\Phi} ^ {(n,:)} \right\| ^ {2} \operatorname {a r c c o s} (- 1) = \frac {\left\| \boldsymbol {\Phi} ^ {(n , :)} \right\| ^ {2}}{2}. \tag {43}
$$
Combine Eq. (42) and Eq. (43), we have
$$
\lambda_ {0} \leq \frac {\sum_ {n = 1} ^ {N} \eta_ {V} \| \boldsymbol {\Phi} ^ {(n , :)} \| _ {2} ^ {2}}{2 N} \leq \eta_ {V} d _ {s} C _ {x} ^ {2} \leq \mathcal {O} (1).
$$
That means, our estimation on  $\lambda_0$  is tight as its upper and lower bounds match with each other.
Now we are ready to present the proof for LeCun initialization under  $\tau_0 = d_m^{-1/2}$  scaling.
Proof. We select  $C_Q = C_K = 1 = C_V = C_O = 1$ , then by Lemma 5, with probability at least  $1 - 8e^{-d_m / 2}$ , we have:
$$
\bar {\lambda} _ {V} = \mathcal {O} (\sqrt {d _ {m} / d}), \quad \bar {\lambda} _ {O} = \mathcal {O} (1), \tag {44}
$$
$$
\bar {\lambda} _ {Q} = \mathcal {O} (\sqrt {d _ {m} / d}), \quad \bar {\lambda} _ {K} = \mathcal {O} (\sqrt {d _ {m} / d}).
$$
Plugging Eq. (44) into Eqs. (2) and (3), it suffices to prove the following equations.
$$
\alpha^ {2} \geq \mathcal {O} \left(\sqrt {N} d _ {s} ^ {3 / 2} C _ {x} \sqrt {d _ {m} / d}\right) \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \tag {45}
$$
$$
\alpha^ {3} \geq \mathcal {O} \left(N d _ {s} ^ {3} C _ {x} ^ {2} \left(1 + 4 C _ {x} ^ {4} d _ {s} ^ {2} d _ {m} d ^ {- 2}\right)\right) \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})} \tag {46}
$$
Next, we will provide the lower bound for  $\alpha^2 = \lambda_{\mathrm{min}}((F_{\mathrm{pre}}^0)(F_{\mathrm{pre}}^0)^\top)$ . In the following context, we hide the index 0 for simplification. One can note that  $F_{\mathrm{pre}}F_{\mathrm{pre}}^\top$  is the summation of PSD matrices, thus, it suffices to lower bound:  $\lambda_{\mathrm{min}}(\hat{F}_{\mathrm{pre}}\hat{F}_{\mathrm{pre}}^\top)$ , where we introduce the following notations:
$$
\hat {\boldsymbol {F}} _ {\mathrm {p r e}} \hat {\boldsymbol {F}} _ {\mathrm {p r e}} ^ {\top} = \tau_ {1} ^ {2} \sigma_ {r} \left(\boldsymbol {\Phi} \boldsymbol {W} _ {v} ^ {\top}\right) \sigma_ {r} \left(\boldsymbol {\Phi} \boldsymbol {W} _ {v} ^ {\top}\right) ^ {\top}
$$
$$
\boldsymbol {\Phi} = \left[ \boldsymbol {X} _ {1} ^ {\top} \beta_ {1, 1}, \dots , \boldsymbol {X} _ {N} ^ {\top} \beta_ {1, N} \right] ^ {\top} \tag {47}
$$
$$
\boldsymbol {\beta} _ {1, n} = \sigma_ {s} \left(\tau_ {0} \boldsymbol {X} _ {n} ^ {(1,:)} \boldsymbol {W} _ {Q} ^ {\top} \boldsymbol {W} _ {K} \boldsymbol {X} _ {n} ^ {\top}\right) ^ {\top} n \in [ N ].
$$
By Matrix-Chernoff inequality, we can obtain that (e.g. Lemma 5.2 of Nguyen et al. [2021]) w.p at least  $1 - \delta_{1}$
$$
\lambda_ {\min } \left(\hat {\boldsymbol {F}} _ {\text {p r e}} \hat {\boldsymbol {F}} _ {\text {p r e}} ^ {\top}\right) \geq d _ {m} \lambda_ {0} / 4, \tag {48}
$$
as long as it holds  $d_m \geq \tilde{\Omega}(N / \lambda_0)$ , where  $\lambda_0 = \lambda_{\min}\left(\mathbb{E}_{\boldsymbol{w} \sim \mathcal{N}(0, \eta_V \mathbb{I}_d)}[\sigma_r(\boldsymbol{\Phi}\boldsymbol{w})\sigma_r(\boldsymbol{\Phi}\boldsymbol{w})^\top]\right)$ , and  $\tilde{\Omega}$  hides logarithmic factors depending on  $\delta_1$ . Lastly, w.p. at least  $1 - \delta_2$ , one has  $\sqrt{2\ell(\boldsymbol{\theta}^0)} \leq \tilde{\mathcal{O}}(\sqrt{N})$ . Plugging back Eqs. (2) and (3), it suffices to prove the following inequality.
$$
d _ {m} \lambda_ {0} / 4 \geq \tilde {\mathcal {O}} \left(N d _ {s} ^ {3 / 2} C _ {x} \sqrt {d _ {m} / d}\right), \tag {49}
$$
$$
\left(d _ {m} \lambda_ {0} / 4\right) ^ {3 / 2} \geq \tilde {\mathcal {O}} \left(N ^ {3 / 2} d _ {s} ^ {3} C _ {x} ^ {2} \left(1 + 4 C _ {x} ^ {4} d _ {s} ^ {2} d _ {m} d ^ {- 2}\right)\right), \tag {50}
$$
By Lemma 15, with probability at least  $1 - \exp \left(-\Omega \left(\left(N - 1\right)^{-\hat{c}} d_{s}^{-1}\right)\right)$ , one has  $\lambda_0 \geq \Theta (\eta_V / d_s) = \Theta (d^{-1}d_s^{-1})$ . Thus, when  $d_m \geq \tilde{\Omega}(N^3)$ , all of the above conditions hold. As a result, the conditions in Eqs. (45) and (46) are satisfied and the convergence of training Transformer is guaranteed as in Eq. (4). Note that one can achieve the same width requirement and probability for He initialization, and the proof bears resemblance to the LeCun initialization.
For the proof under LeCun initialization and  $\tau_0 = d_m^{-1}$  scaling, we follow the same strategy. Specifically: As the same in the proof with  $\tau_0 = d_m^{-1/2}$  scaling, we select  $C_Q = C_K = C_V = C_O = 1$ , then plugging Eq. (44) into Eqs. (2) and (3), it suffices to prove the following equations.
$$
\alpha^ {2} \geq \mathcal {O} \left(\sqrt {N} d _ {s} ^ {3 / 2} C _ {x} \sqrt {d _ {m} / d}\right) \sqrt {2 \ell (\boldsymbol {\theta} ^ {0})}, \tag {51}
$$
$$
\alpha^ {3} \geq \mathcal {O} \left(N d _ {s} ^ {3} C _ {x} ^ {2} \left(1 + 4 C _ {x} ^ {4} d _ {s} ^ {2} d ^ {- 2}\right)\right) \sqrt {2 \ell \left(\boldsymbol {\theta} ^ {0}\right)}. \tag {52}
$$
Next, we will provide the lower bound for  $\alpha^2 = \lambda_{\mathrm{min}}((F_{\mathrm{pre}}^0)(F_{\mathrm{pre}}^0)^\top)$ . In the following context, we hide the index 0 for simplification. One can note that  $F_{\mathrm{pre}}F_{\mathrm{pre}}^\top$  is the summation of PSD matrices,
thus, it suffices to lower bound:  $\lambda_{\mathrm{min}}(\hat{\pmb{F}}_{\mathrm{pre}}\hat{\pmb{F}}_{\mathrm{pre}}^{\top})$ , where we introduce the following notations:
$$
\hat {\boldsymbol {F}} _ {\mathrm {p r e}} \hat {\boldsymbol {F}} _ {\mathrm {p r e}} ^ {\top} = \tau_ {1} ^ {2} \sigma_ {r} \left(\boldsymbol {\Phi} \boldsymbol {W} _ {v} ^ {\top}\right) \sigma_ {r} \left(\boldsymbol {\Phi} \boldsymbol {W} _ {v} ^ {\top}\right) ^ {\top},
$$
$$
\boldsymbol {\Phi} = \left[ \boldsymbol {X} _ {1} ^ {\top} \beta_ {1, 1}, \dots , \boldsymbol {X} _ {N} ^ {\top} \beta_ {1, N} \right] ^ {\top}, \tag {53}
$$
By Eq. (48) w.p at least  $1 - \delta_{1}$ ,  $\lambda_{\mathrm{min}}(\hat{\pmb{F}}_{\mathrm{pre}}\hat{\pmb{F}}_{\mathrm{pre}}^{\top})\geq d_{m}\lambda_{0} / 4$  as long as it holds  $d_m\geq \tilde{\Omega} (N / \lambda_0)$  where  $\lambda_0 = \lambda_{\mathrm{min}}$  ( $\mathbb{E}_{\boldsymbol{w}\sim \mathcal{N}(0,\eta_V\mathbb{I}_d)}[\sigma_r(\Phi \boldsymbol {w})\sigma_r(\Phi \boldsymbol {w})^\top ]$ ), and  $\tilde{\Omega}$  hides logarithmic factors depending on  $\delta_{1}$ . Lastly, note that the activation function in the output layer  $\sigma_r$  is 1-Lipschitz and is applied to  $\sigma_r(W_VX^\top \beta_i)$ , where  $X^{\top}\beta_{i}$  is bounded due to the softmax's property in Lemma 8, then by Lemma C.1 of Nguyen and Mondelli [2020], w.p. at least  $1 - \delta_{3}$ , one has  $\sqrt{2\ell(\pmb{\theta}^{0})}\leq \tilde{\mathcal{O}} (\sqrt{N})$ . Plugging back Eqs. (2) and (3), it suffices to prove the following inequality.
$$
d _ {m} \lambda_ {0} / 4 \geq \tilde {\mathcal {O}} \left(N d _ {s} ^ {3 / 2} C _ {x} \sqrt {d _ {m} / d}\right), \tag {54}
$$
$$
\left(d _ {m} \lambda_ {0} / 4\right) ^ {3 / 2} \geq \tilde {\mathcal {O}} \left(N ^ {3 / 2} d _ {s} ^ {3} C _ {x} ^ {2} \left(1 + 4 C _ {x} ^ {4} d _ {s} ^ {2} d ^ {- 2}\right)\right). \tag {55}
$$