SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \boldsymbol {K} _ {t} - \boldsymbol {K} _ {0} \right\| = \left\| \mathcal {J} (\boldsymbol {W} _ {t}) \mathcal {J} (\boldsymbol {W} _ {t}) ^ {\top} - \mathcal {J} (\boldsymbol {W} _ {0}) \mathcal {J} (\boldsymbol {W} _ {0}) ^ {\top} \right\| \\ \leq 2 \left\| \mathcal {J} \left(\boldsymbol {W} _ {0}\right) \right\| \left\| \mathcal {J} \left(\boldsymbol {W} _ {t}\right) - \mathcal {J} \left(\boldsymbol {W} _ {0}\right) \right\| + \left\| \mathcal {J} \left(\boldsymbol {W} _ {t}\right) - \mathcal {J} \left(\boldsymbol {W} _ {0}\right) \right\| ^ {2}. \tag {26} \\ \end{array}
$$
By [71, Lemma 6.6], we know  $\| \mathcal{J}(\boldsymbol{W}_0) \|^2 = \left\| \boldsymbol{K}_0^{\mathrm{NTK}} \right\|$  is upper bounded by some constant  $C > 0$  with high probability. Then, we apply the inequalities from Lemma 6.5 of [71] to obtain
$$
\begin{array}{l} \| \mathcal {J} (\boldsymbol {W} _ {t}) - \mathcal {J} (\boldsymbol {W} _ {0}) \| ^ {2} \\ = \left\| \left(\left(\sigma^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma^ {\prime} \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right)\right) ^ {\top} \frac {\operatorname {d i a g} (\boldsymbol {v}) ^ {2}}{h} \left(\sigma^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma^ {\prime} \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right)\right)\right) \odot \left(\frac {\boldsymbol {X} ^ {\top} \boldsymbol {X}}{d}\right) \right\| \\ \leq \left\| \left(\sigma^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma^ {\prime} \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right)\right) ^ {\top} \frac {\operatorname {d i a g} (\boldsymbol {v})}{\sqrt {h}} \right\| ^ {2} \left(\max  _ {i \in [ n ]} \left\| \boldsymbol {x} _ {i} / \sqrt {d} \right\| ^ {2}\right) \\ \leq \frac {1}{h} \| \boldsymbol {v} \| _ {\infty} ^ {2} \left\| \sigma^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma^ {\prime} \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| ^ {2} \left(\max  _ {i \in [ n ]} \left\| \boldsymbol {x} _ {i} / \sqrt {d} \right\| ^ {2}\right) \\ \leq \frac {\lambda_ {\sigma} ^ {2}}{h} \frac {\| \boldsymbol {X} \| ^ {2}}{d} \| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \| _ {F} ^ {2} \left(\max  _ {i \in [ n ]} \left\| \boldsymbol {x} _ {i} / \sqrt {d} \right\| ^ {2}\right), \tag {27} \\ \end{array}
$$
where the last inequality is due to the mean value theorem, the uniform bound on  $\sigma''$ , and the assumption on the second layer  $\pmb{v}$ . Notice that Gaussian random vectors satisfy
$$
\mathbb {P} \left(\max  _ {i \in [ n ]} \frac {1}{d} \| \boldsymbol {x} _ {i} \| ^ {2} \geq 2\right) \leq 2 n e ^ {- c n}, \tag {28}
$$
as  $n / d\to \gamma_1$  and  $h / d\rightarrow \gamma_{2}$  . Thus, with (22) and Lemma C.2, we obtain
$$
\mathbb {P} \left(\| \mathcal {J} (\boldsymbol {W} _ {t}) - \mathcal {J} (\boldsymbol {W} _ {0}) \| \geq \frac {C \lambda_ {\sigma} (1 + \gamma_ {1})}{n}\right) \leq 4 n e ^ {- c n},
$$
where constant  $C$  relies on the number of steps  $t$ . Hence, by (26), we can finally bound in norm the difference between the initial and the trained NTK matrices at the early phase ( $t$  is finite).
Corollary C.3. For any fixed  $t \in \mathbb{N}$ ,  $i \in [d]$  and  $k \in [n]$ , denote  $\lambda_i^t$ ,  $\nu_k^t$ , and  $\mu_k^t$  the  $i$ -th, and  $k$ -th eigenvalues of  $\frac{1}{h} \pmb{W}_t^\top \pmb{W}_t$ ,  $\pmb{K}_t^{CK}$  and  $\pmb{K}_t^{NTK}$ , respectively. Then, under the assumptions of Lemma 4.1, we have
$$
\left| \lambda_ {i} ^ {t} - \lambda_ {i} ^ {0} \right|, \left| \nu_ {k} ^ {t} - \nu_ {k} ^ {0} \right|, \left| \mu_ {k} ^ {t} - \mu_ {k} ^ {0} \right|\rightarrow 0,
$$
almost surely in LWR. Consequently, the eigenvalues of  $\frac{1}{h}\pmb{W}_t^\top \pmb{W}_t$ ,  $\pmb{K}_t^{CK}$  and  $\pmb{K}_t^{NTK}$  are the same as corresponding the eigenvalues of initial  $\frac{1}{h}\pmb{W}_0^\top \pmb{W}_0$ ,  $\pmb{K}_0^{CK}$  and  $\pmb{K}_0^{NTK}$ , respectively.
This corollary is a direct outcome of Weyl's inequality from Theorem A.46 in [8]. Consequently, this corollary concludes that for any fixed  $t \geq 0$ , almost surely, the limiting spectra of  $\frac{1}{h} \mathbf{W}_t^\top \mathbf{W}_t$ ,  $\mathbf{K}_t^{\mathrm{CK}}$  and  $\mathbf{K}_t^{\mathrm{NTK}}$  are the same as those of  $\frac{1}{h} \mathbf{W}_0^\top \mathbf{W}_0$ ,  $\mathbf{K}_0^{\mathrm{CK}}$  and  $\mathbf{K}_0^{\mathrm{NTK}}$  in LWR. This corollary claims that not only does the bulk of distributions stay identical to the initialization, but also that any eigenvalues stay the same as at the initialization. This shows that the smallest eigenvalue of  $\mathbf{K}_t^{\mathrm{NTK}}$  has the same lower bound as  $\mathbf{K}_0^{\mathrm{NTK}}$  in the early phase of training.
C.3 Global Convergence for GD Under LWR
In this section, we study the final stage of (10) as training loss is approaching zero and prove Theorem 4.2. Figure 29 shows that the spectra are unchanged globally, even after training in this case. In Corollary 4.3, we confirm this observation for the weight, CK, and NTK matrices via Frobenius norm control. In the simulation, the second layer is initialized as  $\pmb{v} \sim \mathcal{N}(0, \mathbf{I})$ , which is more general than our assumption on  $\pmb{v}$  in Theorem 4.2.
Figure 29: The initial and trained spectra (until training loss is less than  $10^{-5}$ ) when using GD only for the first layer ( $h = 3000$ ,  $n = 2000$ ,  $d = 1000$ ): (a) Weight spectra. (b)  $\pmb{K}^{\mathrm{CK}}$  spectra. (c)  $\pmb{K}^{\mathrm{NTK}}$  spectra. The final  $R^2$  score is 0.55964 and the test loss is 0.44724. The activation is a normalized ReLU, and the target is Sigmoid.
Proof of Theorem 4.2. Recall the Jacobian matrix  $\mathcal{J}(\boldsymbol{W})$  defined in the proof of Lemma 4.1, and the definition of  $\alpha$  based on (11) in Section 3. Denote the event
$$
\mathcal {A} := \left\{\| \boldsymbol {X} \| \leq 2 (1 + \sqrt {\gamma_ {1}}) \sqrt {d}, \max  _ {i \in [ n ]} \| \boldsymbol {x} _ {i} \| ^ {2} \leq 2 d, \sigma_ {\min } (\mathcal {J} (\boldsymbol {W} _ {0})) \geq 2 \alpha \right\}.
$$
By (22), (28) and Theorem 2.9 of [81], we have  $\mathbb{P}(\mathcal{A})\geq 1 - 2e^{-cn} - 2ne^{-cn} - n^{-7 / 3}$  for some constant  $c > 0$  and all large  $n$  in LWR. In the following, conditionally on event  $\mathcal{A}$ , we will apply Theorem 6.10 of [71] to obtain the global convergence. Conditionally on  $\mathcal{A}$ , Lemma 6.6 of [71] implies
$$
\left\| \mathcal {J} (\boldsymbol {W}) \right\| \leq \lambda_ {\sigma} \| \boldsymbol {v} \| _ {\infty} \left\| \boldsymbol {X} / \sqrt {d} \right\| \leq 2 \lambda_ {\sigma} (1 + \sqrt {\gamma_ {1}}), \tag {29}
$$
for any  $\mathbf{W}$ . Define  $\beta = 2\lambda_{\sigma}(1 + \sqrt{\gamma_1})$ . Moreover, in terms of (27), we can verify the Lipschitz property for the Jacobian matrix as follows: conditionally on  $\mathcal{A}$
$$
\left\| \mathcal {J} (\tilde {\boldsymbol {W}}) - \mathcal {J} (\boldsymbol {W}) \right\| \leq \frac {2 \beta}{\sqrt {h}} \left\| \tilde {\boldsymbol {W}} - \boldsymbol {W} \right\| _ {F}, \tag {30}
$$
for any  $\tilde{W}, W \in \mathbb{R}^{h \times d}$ . Therefore, conditionally on  $\mathcal{A}$ ,  $\mathcal{J}(W)$  is a  $L$ -Lipschitz function with respect to  $\pmb{W}$  where  $L := \frac{2\beta}{\sqrt{h}}$ . To complete the proof, it suffices to investigate the smallest singular value of  $\mathcal{J}(W)$  when  $\pmb{W}$  is in the vicinity of  $\pmb{W}_0$ . Recall  $\ell(\pmb{W}) = \| \pmb{y} - f_{\pmb{W}}(\pmb{X})\|$ . Notice that for any unit vector  $\pmb{u} \in \mathbb{R}^n$ , we have  $\pmb{u}^\top f_{\pmb{W}_0}(\pmb{X}) = \frac{1}{\sqrt{h}} \sum_{i=1}^{h} v_i \sigma(\pmb{w}_i^\top \pmb{X} / \sqrt{d}) \pmb{u}$ , where  $\pmb{w}_i^\top$  is the  $i$ -th row of  $\pmb{W}_0$  for  $i \in [N]$ . Consider event  $\mathcal{B} := \left\{\left\| \sigma(\pmb{W}_0 \pmb{X} / \sqrt{d}) \right\| \leq C \sqrt{n}\right\}$  for some universal constant  $C > 0$ . Lemma C.1 proves  $\mathbb{P}(\mathcal{B}) \geq 1 - 2e^{-cn}$ . By the assumption of  $\pmb{v}$ , we know each entry  $v_i$  is a sub-Gaussian random variable with a sub-Gaussian norm at most 1. Then, according to Hoeffding's inequality, conditionally on the event  $\mathcal{B}$ , we have
$$
\mathbb {P} \left(\left| \frac {1}{\sqrt {h}} \sum_ {i = 1} ^ {h} v _ {i} \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {X} / \sqrt {d}\right) \boldsymbol {u} \right| \geq t\right) \leq 2 \exp \left(- c t ^ {2}\right),
$$
for every  $t \geq 0$  and some constant  $c > 0$ . Let  $t = 2\sqrt{n}$ . Considering an  $\frac{1}{4}$ -net  $\mathcal{N}$  of the unit sphere  $\mathbb{S}^{n-1}$ , we can get
$$
\mathbb {P} \left(\| f _ {\boldsymbol {W} _ {0}} (\boldsymbol {X}) \| \geq \sqrt {n}\right) \leq \mathbb {P} \left(2 \max  _ {\boldsymbol {u} \in \mathcal {N}} \left| \boldsymbol {u} ^ {\top} f _ {\boldsymbol {W} _ {0}} (\boldsymbol {X}) \right| \geq \sqrt {n}\right) \leq 9 ^ {n} 2 \exp (- c n) \leq 2 e ^ {- c ^ {\prime} n}, \tag {31}
$$
for some constant  $c' > 0$ . Hence, based on Lemma C.1 and (31), we can obtain  $\ell(\mathbf{W}_0) \leq C_0\sqrt{n}$  with high probability for some universal constant  $C_0 > 0$ . Let us denote this event as  $\mathcal{C} : \{\ell(\mathbf{W}_0) \leq C_0\sqrt{n}\}$ . Define  $R := 4\ell(\mathbf{W}_0)/\alpha$ . For any  $\mathbf{W}$  in a ball of radius  $R$  centered at  $\mathbf{W}_0$ , we have  $\| \mathbf{W}_0 - \mathbf{W} \|_F \leq R$  and  $\| \mathcal{J}(\mathbf{W}) - \mathcal{J}(\mathbf{W}_0) \| \leq LR$ , conditionally on event  $\mathcal{A}$ . Thus, by (30), on event  $\mathcal{A} \cap \mathcal{C}$ , the smallest singular value  $\sigma_{\min}(\mathcal{J}(\mathbf{W}))$  of the Jacobian matrix  $\mathcal{J}(\mathbf{W})$  can be bounded by
$$
\begin{array}{l} \sigma_ {\min } (\mathcal {J} (\boldsymbol {W})) \geq \sigma_ {\min } (\mathcal {J} (\boldsymbol {W} _ {0})) - \| \mathcal {J} (\boldsymbol {W}) - \mathcal {J} (\boldsymbol {W} _ {0}) \| \\ \geq 2 \alpha - L R \geq 2 \alpha - \frac {8 \beta}{\alpha} \frac {\ell (\boldsymbol {W} _ {0})}{\sqrt {h}} \geq 2 \alpha - \frac {8 C \beta}{\alpha} \sqrt {\frac {\gamma_ {1}}{\gamma_ {2}}}, \\ \end{array}
$$
for some universal constant  $C > 0$  and sufficiently large  $n, d, h$ . Notice that here constants  $C, \beta$ , and  $\alpha$  do not rely on  $\gamma_{2}$ . Therefore, there exists a sufficiently large  $\gamma^{*} > 0$  such that for all  $\gamma_{2} \geq \gamma^{*}$ , we have  $2\alpha - \frac{8C\beta}{\alpha}\sqrt{\frac{\gamma_{1}}{\gamma_{2}}} \geq \alpha$ . In other words, when  $h$  is sufficiently large but still in the same order as  $n$  and  $d$ , for all  $\|W - W_{0}\|_{F} \leq R$ , we have  $\sigma_{\min}(\mathcal{J}(W)) \geq \alpha$  conditionally on  $\mathcal{C} \cap \mathcal{A}$ . Combining with (29) and (30), conditionally on  $\mathcal{C} \cap \mathcal{A}$ , all the assumptions of Theorem 6.10 by [71] are satisfied when  $\|W - W_{0}\|_{F} \leq R$ . Therefore, when the learning rate  $\frac{\eta}{n} \leq \frac{1}{\beta^{2}}\min\left\{1, \frac{4\alpha}{LR}\right\}$ , we can get (12)-(14) for all  $t \in \mathbb{N}$ , conditionally on  $\mathcal{C} \cap \mathcal{A}$ . Both events  $\mathcal{A}$  and  $\mathcal{C}$  occur with high probability and only depend on initialization  $W_{0}, X$  and  $y$ . Hence we complete the proof of this theorem. Notice that since  $\gamma_{2} \geq \gamma^{*}$  is sufficiently large,  $\frac{4\alpha}{LR} \geq \frac{\alpha^{2}}{2C\beta}\sqrt{\frac{\gamma_{2}}{\gamma_{1}}} > 1$ . Therefore, it suffices to require  $\eta \leq n / \beta^{2}$  to conclude that (12), (13) and (14) hold with high probability. This completes the proof. Moreover, (14) further shows that for all  $t \in \mathbb{N}$ ,
$$
\left\| \boldsymbol {W} _ {0} - \boldsymbol {W} _ {t} \right\| _ {F} \leq R \leq C \sqrt {n} + o _ {d, \mathbb {P}} (1), \tag {32}
$$
where we again apply Lemma C.1 in the following way:
$$
\ell \left(\boldsymbol {W} _ {0}\right) \leq C \sqrt {n} + o _ {d, \mathbb {P}} (1),
$$
for some constant  $C > 0$  only depending on  $\gamma_1, \gamma_2, \sigma_\varepsilon, \sigma$  and  $\sigma^*$ .
(a) Changes for  $\frac{1}{\sqrt{d}} W$
(b) Changes for CK.
(c) Changes for NTK.
Figure 30: Measuring the change for the weight, CK, and NTK matrices when training NN with (10). We fix  $d / n = 1.2$  and  $h / n = 0.6$  when  $n$  is increasing. Here,  $\sigma$  is normalized ReLU and the target is normalized tanh. The largest  $n = 6400$  and the learning rate  $\eta = 5.0$  for all training processes. We train each neural network until the training losses approach zero. Each experiment repeats 4 times. In (a), we consider the changes  $\frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\| ,\frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\| _F,$  and  $\frac{1}{\sqrt{d}}\| \pmb {W}_t - \pmb {W}_0\|_{2,\infty}$
As a corollary, (14) controls the deviation of the final step weight from the initial weight. This is empirically shown in Figure 30(a), which shows that  $\frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\|, \frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\|_F$ , and  $\frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\|_{2,\infty}$  are  $\Theta(1)$  when trainable parameters are convergent. This implies that the final  $\pmb{W}_t$  is still close to the initial weight  $\pmb{W}_0$ , even after training. Consequently, with this observation, we can prove Corollary 4.3 in the following.
Proof of Corollary 4.3. Based on (32), we know  $\frac{1}{\sqrt{d}}\| W_o - W_t\|_F \leq C_0$  holds with high probability for some universal constant  $C_0 > 0$ . Conditionally on this event, we can then estimate changes in CK and NTK after training. The method is analogous to Lemma 4.1. For CK, we employ Lemma C.1 and (25) to get
$$
\begin{array}{l} \left\| \boldsymbol {K} _ {t} ^ {\mathrm {C K}} - \boldsymbol {K} _ {0} ^ {C K} \right\| _ {F} \\ \leq \frac {2}{h} \left(\left\| \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| + \left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d} \right\| \right)\right) \cdot \left\| \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| _ {F} \\ \lesssim \frac {2 \lambda_ {\sigma} ^ {2} (1 + \sqrt {\gamma_ {1}}) ^ {2}}{h} \| \boldsymbol {W} _ {0} - \boldsymbol {W} _ {t} \| _ {F} ^ {2} + \frac {2 C \sqrt {n} \lambda_ {\sigma} (1 + \sqrt {\gamma_ {1}})}{h} \| \boldsymbol {W} _ {0} - \boldsymbol {W} _ {t} \| _ {F} \\ \lesssim \frac {2 \lambda_ {\sigma} (1 + \sqrt {\gamma_ {1}}) C _ {0}}{\gamma_ {2} ^ {2}} (\lambda_ {\sigma} (1 + \sqrt {\gamma_ {1}}) C _ {0} + C \sqrt {\gamma_ {1}}) = O _ {d, \mathbb {P}} (1). \\ \end{array}
$$
Hence, this shows control of the change for the CK matrix after training, compared with the initial CK.
Let us denote  $\boldsymbol{w}_i^t \in \mathbb{R}^{1 \times d}$  as the  $i$ -th row of  $\boldsymbol{W}_t$ , and  $\boldsymbol{x}_j$  as the  $j$ -th column of  $\boldsymbol{X}$ . Additionally, by Assumption 3.2, we know that
$$
\left| \sigma^ {\prime} (x) - \sigma^ {\prime} (y) \right| \leq \lambda_ {\sigma} | x - y |, \tag {33}
$$
for any  $x, y \in \mathbb{R}$ . For NTK, by modifying (27), one can deduce that
$$
\begin{array}{l} \left\| \mathcal {J} (\boldsymbol {W} _ {t}) - \mathcal {J} (\boldsymbol {W} _ {0}) \right\| _ {F} ^ {2} = \sum_ {i = 1} ^ {h} \left\| \mathcal {J} (\boldsymbol {w} _ {i} ^ {t}) - \mathcal {J} (\boldsymbol {w} _ {i} ^ {0}) \right\| _ {F} ^ {2} \\ \stackrel {(i)} {\leq} \frac {1}{h} \sum_ {i = 1} ^ {h} \left\| \operatorname {d i a g} \left(\sigma^ {\prime} (\boldsymbol {w} _ {i} ^ {t} \boldsymbol {X} / \sqrt {d} - \sigma^ {\prime} (\boldsymbol {w} _ {i} ^ {0} \boldsymbol {X} / \sqrt {d})\right) \right\| _ {F} ^ {2} \left\| \frac {\boldsymbol {X}}{\sqrt {d}} \right\| ^ {2} \\ \stackrel {(i i)} {\leq} \frac {(1 + \sqrt {\gamma_ {1}}) ^ {2}}{h} \sum_ {i = 1} ^ {h} \left\| \operatorname {d i a g} \left(\sigma^ {\prime} \left(\boldsymbol {w} _ {i} ^ {t} \boldsymbol {X} / \sqrt {d} - \sigma^ {\prime} \left(\boldsymbol {w} _ {i} ^ {0} \boldsymbol {X} / \sqrt {d}\right)\right) \right\| _ {F} ^ {2} + o _ {d, \mathbb {P}} (1) \right. \\ \stackrel {(i i i)} {\leq} \frac {\lambda_ {\sigma} ^ {2} (1 + \sqrt {\gamma_ {1}}) ^ {2}}{h} \sum_ {i = 1} ^ {h} \sum_ {j = 1} ^ {n} \left(\frac {1}{\sqrt {d}} \left(\boldsymbol {w} _ {i} ^ {t} - \boldsymbol {w} _ {i} ^ {0}\right) \boldsymbol {x} _ {j}\right) + o _ {d, \mathbb {P}} (1) \\ \stackrel {(i v)} {\leq} \frac {\lambda_ {\sigma} ^ {2} (1 + \sqrt {\gamma_ {1}}) ^ {4}}{h} \| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \| _ {F} ^ {2} + o _ {d, \mathbb {P}} (1) \leq \frac {\lambda_ {\sigma} ^ {2} (1 + \sqrt {\gamma_ {1}}) ^ {4} C _ {0} ^ {2}}{\gamma_ {2}} + o _ {d, \mathbb {P}} (1), \\ \end{array}
$$
where  $(i)$  is because of [80, Exercise 6.3.3] and the assumption on  $\pmb{v}$ ,  $(ii)$  is due to (22),  $(iii)$  is due to the definition of Frobenius norm and (33), and  $(iv)$  is due to [80, Exercise 6.3.3] and (22). As a result, from (26), we can finally conclude that  $\left\| \pmb{K}_t^{\mathsf{CK}} - \pmb{K}_0^{CK}\right\|_F = O_{d,\mathbb{P}}(1)$  as  $n / d\to \gamma_1$  and  $h / d\rightarrow \gamma_2$ .
As for the limiting spectra of weight and kernel matrices, since we know that
$$
\frac {1}{\sqrt {d}} \left\| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \right\| _ {F}, \left\| \boldsymbol {K} _ {t} ^ {\mathrm {C K}} - \boldsymbol {K} _ {0} ^ {\mathrm {C K}} \right\| _ {F}, \left\| \boldsymbol {K} _ {t} ^ {\mathrm {N T K}} - \boldsymbol {K} _ {0} ^ {\mathrm {N T K}} \right\| _ {F} = O _ {d, \mathbb {P}} (1),
$$
we can automatically apply Corollary A.41 of [8]. This directly implies that the limiting empirical spectra of  $\frac{1}{h} W_t^\top W_t$ ,  $K_t^{\mathrm{CK}}$  and  $K_t^{\mathrm{NTK}}$  are the same as the limiting spectra of  $\frac{1}{h} W_0^\top W_0$ ,  $K_0^{\mathrm{CK}}$  and  $K_0^{\mathrm{NTK}}$ , respectively, as  $n / d \to \gamma_1$  and  $h / d \to \gamma_2$  (see Figure 29).
Further Simulations for Changes in Norms. The simulation of Figure 30 empirically coincides with the norm bounds in Theorem 4.2 for different norms. Because of (17), it suffices to only consider the Frobenius norm of the change for each matrix. As a remark, Theorem 4.2 requires  $\gamma_{2}$  to be larger than some threshold  $\gamma^{*}$  to ensure norms of the change throughout training. However, Figure 30 indicates Theorem 4.2 still holds even when  $\gamma^{*}$  is small i.e. the width  $h$  is not very large. Here in Figure 30,  $\gamma_{2} = \frac{1}{2}\gamma_{1} = 0.6$ . Figure 31 suggests that similar results to Theorem 4.2 and Corollary 4.3 hold for SGD when training the first layer. This is also akin to Figure 21. Moreover, we further conjecture that similar results to Theorem 4.2 and Corollary 4.3 will hold even when training both layers (3). Denote the second layer at step  $t$  by  $\pmb{v}_{t}$ . Then, indicated by Figure 32,  $\frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\|, \frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\|_F, \frac{1}{\sqrt{d}}\| \pmb{W}_t - \pmb{W}_0\|_{2,\infty}, \| \pmb{K}_t^{\mathrm{CK}} - \pmb{K}_0^{\mathrm{CK}}\|, \| \pmb{K}_t^{\mathrm{CK}} - \pmb{K}_0^{\mathrm{CK}}\|_F, \| \pmb{K}_t^{\mathrm{NTK}} - \pmb{K}_0^{\mathrm{NTK}}\|$ , and  $\| \pmb{v}_t - \pmb{v}_0\| / \| \pmb{v}_0\|$  are all  $\Theta(1)$  in LWR. Since  $\| \pmb{v}_0\| = O(\sqrt{h})$ , we observe that  $\| \pmb{v}_t - \pmb{v}_0\| = \Theta(\sqrt{h})$  in this case. Meanwhile, unlike Corollary 4.3, Figure 32(c) cannot verify that  $\| \pmb{K}_t^{\mathrm{NTK}} - \pmb{K}_0^{\mathrm{NTK}}\|_F$  is still upper bounded by a constant. One possible explanation is that in this case the change in the second layer  $\pmb{v}_t$  is much more significant than in the first-layer weight  $\pmb{W}_t$ , hence the NTK matrix may change a lot in this general case.
Similarly, Figure 33 shows norms of the change when training the NN with SGD for both layers. Here we use the same batch size 128 and learning rate  $\eta = 1.0$  for all experiments when varying the dimension  $n$  but fixing the aspect ratios  $\gamma_{1}$  and  $\gamma_{2}$ . All the observations are similar to the results of Corollary 4.3 except the Frobenius norm of the change for NTK. Figure 33(c) indicates that  $\left\| K_t^{\mathrm{NTK}} - K_0^{\mathrm{NTK}}\right\| _F$  will not be  $\Theta (1)$  anymore, which fluctuates more in this case.
(b) Changes for CK.
Figure 31: Measuring the change for weight  $\mathbf{W}_t$ ,  $\mathbf{K}^{\mathrm{CK}}$ , and  $\mathbf{K}^{\mathrm{NTK}}$  matrices when training NN with (10) for the first layer  $\mathbf{W}_t$  using SGD with batch size 200. We fix  $d / n = 0.6$  and  $h / n = 1.2$  when  $n$  is increasing. Here  $\sigma$  is normalized softplus and the target is normalized tanh. The largest  $n$  is 11000,  $\sigma_{\varepsilon} = 0.3$  and the learning rate is  $\eta = 3.6$  for all training processes. We train each neural network until the training losses approach zero. Each experiment repeats 15 times.
(c) Changes for NTK.
(c) Changes for NTK.
Figure 32: Measuring the change for  $\pmb{W}_{t}$ ,  $\pmb{K}^{\mathrm{CK}}$ , the second layer  $\pmb{v}_{t}$  and  $\pmb{K}^{\mathrm{NTK}}$  when training NN with (3) for both layer  $\pmb{W}_{t}$  and  $\pmb{v}_{t}$  using GD. We fix  $d / n = 0.5$  and  $h / n = 0.8$  when  $n$  is increasing. Here,  $\sigma$  is normalized softplus and the target is normalized tanh. The largest  $n$  is 11000,  $\sigma_{\varepsilon} = 0.3$  and the learning rate is  $\eta = 3.6$  for all training processes. We train each neural network until the training losses approach zero. Each experiment repeats 15 times.
(a) Changes for  $\frac{1}{\sqrt{d}} W$
Figure 33: Measuring the change for  $\pmb{W}_{t}$ ,  $\pmb{K}^{\mathrm{CK}}$ , the second layer  $\pmb{v}_{t}$  and  $\pmb{K}^{\mathrm{NTK}}$  when training NN with SGD for both layers  $\pmb{W}_{t}$  and  $\pmb{v}_{t}$ . We fix  $d / n = 0.6$  and  $h / n = 1.2$  when  $n$  is increasing. Here,  $\sigma$  is normalized ReLU and the target is normalized tanh.  $\sigma_{\varepsilon} = 0.1$  and the learning rate is  $\eta = 1.0$  for all training processes. We train each neural network until the training losses approach zero. Each experiment repeats 12 times.