SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 16: Additional performance for Case 4 in Table 1: (a) The initial and trained weight spectra. Notice that there are several outliers after training, while the bulk has a heavier tail. (b) The spectra of the NTK (5) at initialization and after training. (c) The test/training losses and  $R^2$  score (green line) at each epoch  $t$  throughout training process. (d) Alignment between the leading PC of the weight matrix and the signal  $\beta$  in the teacher model before (red) and after (blue) training. (e) Alignment between the leading PC of the CK matrix and the training labels  $y$  before/after training. (f) Alignment between the leading PC of the NTK matrix and  $y$  before/after training.
(a) Evolution of metrics.
Figure 17: (a) The evolution of power  $\alpha$ , weighted Alpha and Log  $\alpha$ -norm (several metrics of power law tails; see [60]) during the training process in Case 4 of Table 1. (b) Evolutions of the first PC angle  $\theta_{1}$  between feature subspace  $U = \mathrm{span}\{\beta_{i}\}_{i=1}^{k}$  of the multiple-index model (8) and the eigenspace spanned by top 100 of eigenvectors of  $\boldsymbol{W}_t^\top \boldsymbol{W}_t$  during training with Adam (blue solid line), SGD (red dashed line) and GD (green dash-dot). The final test error is 0.33865 and the  $R^2$  score is -0.71065 for GD. The test error is 0.10814 and the  $R^2$  score is 0.45373 for SGD, where one spike emerges in the weight spectrum after training. The test error is 0.08672 and the  $R^2$  score is 0.56195 for Adam. (c) Initial and trained spectra for weight matrices when training with Adam (blue solid line in (b)). Heavy tail emerges in this case.
(b) Evolution of PC angles.
(c) Weight spectra.
can get heavy-tailed distributions after training, and the phenomena are essentially the same as Case 4 of Table 1. There is no strong alignment for the first leading eigenvector, while useful features may be learned by a few top eigenvectors in the heavy-tailed spectra after training. In the second experiment, we directly apply Adam with a small initial learning rate for training. In contrast, the results of this case, presented in Figures 26 and 27, are similar to Case 3 in Table 1. The test loss and  $R^2$  score are close to previous examples, though slightly worse. As explained in the previous section, because there is only one spike appearing outside the bulk after training in the second case, the leading PC is highly aligned with the training dataset structure after training and the feature learning mainly stems from the outliers in this situation. This interprets the strongly anisotropic structures in trained spectra of NNs in the second case [67, 68, 78]. These two different spectral properties reveal significant differences between the global minima of these two training processes and different evolutions of the
(a) Losses.
(b)  $\lambda_{\mathrm{max}}$  (CK).
(c)  $\lambda_{\mathrm{max}}(\mathrm{NTK})$
(a) Losses vs learning rates.
Figure 18: The training dynamic when training neural networks with SGD and learning rate 24 in the example of Figure 2(b-c). The learning rate we chose here is above the threshold we showed in Figure 2(b-c). We use the same architecture, dataset, and teacher model as in Section 3 of our paper. The batch size is 32. (a) The evolution of the training and test errors during training. (b) The evolution of the largest eigenvalue of the CK matrix. (c) The evolution of the largest eigenvalue of the NTK matrix. This regime corresponds to the catapult phenomenon [49].
(d)  $R^2$  vs learning rates.
(b) Weight spectra with a small learning rate.
(e) Weight spectra with a large learning rate.
(c) Weight alignments with a small learning rate.
(f) Weight alignments with a large learning rate.
Figure 19: Grid search for different learning rates when training NNs with full-batch GD in the same setting as Case 1 in Table 1. (a) Final test losses when varying learning rates. (d)  $R^2$  scores when varying learning rates. For all these learning rates, we did not observe heavy-tailed spectra. (b-c) present the spectral behaviors for the smallest learning rate we used in (a)&(d). (e-f) present the spectral behaviors for the largest learning rate we can use which still ensures the convergence of GD. In this case, analogously to the SGD case in Section 4.2, we observe an outlier in the trained weight matrix and strong alignment with the spike.
spectra in NNs. Remarkably, based on the different spectral behaviors in trained weight and kernel matrices, these two experiments exhibit disparate features learned by distinct training procedures. Hence, analyzing the spectral properties in trained kernel matrices is beneficial for clarifying what features our NNs have learned during the training processes.
C Proofs of Results in Section 4.1
C.1 GD Analysis at Early Phase
From (10), the GD process with learning rate  $\eta > 0$  can be written by
$$
\boldsymbol {W} _ {t + 1} = \boldsymbol {W} _ {t} + \eta \cdot \boldsymbol {G} _ {t}, \text {w h e r e} \tag {18}
$$
$$
\boldsymbol {G} _ {t} = \frac {1}{n \sqrt {d h}} \left[ \left(\boldsymbol {v} \left(\boldsymbol {y} - \frac {1}{\sqrt {h}} \boldsymbol {v} ^ {\top} \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right)\right)\right) \odot \sigma^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) \right] \boldsymbol {X} ^ {\top}, \tag {19}
$$
for  $t \in \mathbb{N}$ , where  $\pmb{y} \in \mathbb{R}^{1 \times n}$ . Following [6, Appendix B], in this section we prove the control for gradient step  $\pmb{G}_t$ . For simplicity, denote  $f_t(\pmb{X}) := f_{\pmb{\theta}_t}(\pmb{X}) = \frac{1}{\sqrt{h}} \pmb{v}^\top \sigma(\pmb{W}_t \pmb{X} / \sqrt{d})$  for  $t \in \mathbb{N}$ .
Figure 20: (Left) Initial and trained spectra for weight matrices when training with Adam. Five leading spikes emerge in this case. (Right) Evolutions of the angles between the first four PCs of  $\mathbf{W}_t^\top \mathbf{W}_t$  and feature subspace  $U = \mathrm{span}\{\beta_i\}_{i=1}^k$  of the multiple-index model (8) during training with Adam. Here  $k = 5$ . The final test error is 0.33865 and  $R^2$  score is -0.71065 for GD. The test error is 0.01681 and  $R^2$  score is 0.9154 for Adam.
Figure 21: Additional spectral performance when training the NN by Adam with small learning rate  $\eta = 0.001$ , where  $n = 2000, h = 3000, d = 1000, \sigma_{\varepsilon} = 0.3$  and batch size is 100. In this simulation, we only train the first hidden layer  $\mathbf{W}_t$ . The activation function  $\sigma$  is a normalized softmax and the target function is a normalized tanh. The final test loss is 0.36219 and  $R^2$  score is around  $63.70\%$ .
Lemma C.1. Under the same assumptions as in Lemma 4.1, we have
$$
\mathbb {P} \left(\left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| \geq C \sqrt {n}\right) \leq 2 e ^ {- c n},
$$
$$
\mathbb {P} \left(\| \boldsymbol {y} \| \geq C \sqrt {n}\right) \leq 2 e ^ {- c n},
$$
for some constants  $C, c > 0$  only depending on  $\sigma_{\varepsilon}, \lambda_{\sigma}, \gamma_{1}$ , and  $\gamma_{2}$ .
Proof. Due to [30, Lemma D.4.], we can directly obtain that
$$
\mathbb {P} \left(\left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| \geq C ^ {\prime} (\sqrt {n} + \sqrt {h}) \sqrt {\frac {h}{d}}\right) \leq 2 e ^ {- c n}.
$$
Here we use the fact that both  $W_0$  and  $X$  are i.i.d. Gaussian random matrices. Then by Assumption 3.1, we conclude that we control  $\sigma(W_0X/\sqrt{d})$ . Recall that Assumption 3.3 implies that  $y = f^*(X) + \varepsilon$ . Hence, by Lipschitz Gaussian concentration inequality [80, Theorem 5.2.2], each entry of  $f^*(X)$  has independent sub-Gaussian coordinates, whence we can get  $\|f^*(X)\| \leq C\sqrt{n}$  with probability at least  $1 - 2ne^{-cn}$  for some constants  $c, C > 0$ . On the other hand,  $[\varepsilon]_i = \varepsilon_i$  are i.i.d. centered sub-Gaussian noises with variance  $\sigma_{\varepsilon}^2$ . By [80, Theorem 3.1.1], we have
$$
\mathbb {P} \left(\| \boldsymbol {\varepsilon} \| \leq 2 \sigma_ {\varepsilon} \sqrt {n}\right) \geq 1 - 2 \exp \left(- \frac {c n}{K ^ {4}}\right),
$$
where the constant  $K$  is the sub-Gaussian norm defined by  $K = \max_{i}\| \varepsilon_{i}\|_{\psi_{2}}$ . Hence, combining all things together, we obtain the second inequality of this lemma.
Figure 22: Additional performance for AdaGrad with learning rate  $\eta = 0.5$ , where  $n = 2000, h = 1500, d = 1000$ ,  $\sigma_{\varepsilon} = 0.3$  and small batch size is 8. Activation  $\sigma$  is normalized softmax and target is normalized tanh. The final test loss is 0.23555 and  $R^2$  score is around 0.76249. The black lines in the Q-Q subplots are the line of  $y = x$ .
(a) First PC in trained & initial weights.
(b) First PC in trained & initial CKs.
(C) First PC in trained & initial NTKs.
Figure 23: Alignment between the leading PC of the weight/kernel matrices and the signal  $\beta$  or the training labels  $y$  before/after training for experiment in Figure 22. Analogously to Case 4 in Appendix B.2, there is no strong alignment in the leading component of weight/kernel matrices.
Figure 24: Additional performance for Adam with learning rate  $\eta = 0.09$  and 4 epochs, then SGD with learning rate  $\eta = 5 \times 10^{-4}$  and 100 epochs, where  $n = 2000$ ,  $h = 1500$ ,  $d = 1000$ ,  $\sigma_{\varepsilon} = 0.3$  and the batch size is 32. We train the NN until the training loss is less than  $10^{-10}$ . The activation  $\sigma$  is normalized softplus and the target is normalized tanh. The final test loss is 0.22511 and  $R^2$  score is around 0.77462.
Lemma C.2. Under the assumptions of Lemma 4.1, given any fixed  $t \in \mathbb{N}$  and learning rate  $\eta = \Theta(1)$ , the weight matrix after  $t$  gradient steps  $\mathbf{W}_t$  defined in (18) satisfies
$$
\mathbb {P} \left(\left\| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \right\| _ {F} \geq \frac {C}{\sqrt {n}}\right) \leq \exp (- c n), \tag {20}
$$
for some positive constants  $c, C > 0$  only depending on  $t, \eta, \sigma_{\varepsilon}, \lambda_{\sigma}, \gamma_{1}$  and  $\gamma_{2}$ .
Proof. Denote  $\sigma_{\perp}(x) = \sigma(x) - \mu_1 x$  which is the nonlinear part of  $\sigma$  and  $\mu_1 = \mathbb{E}[z \sigma(z)]$ . Thus,  $\mathbb{E}[\sigma_{\perp}(z) z] = 0$  for  $z \sim \mathcal{N}(0, 1)$ . Based on this, we can further decompose the gradient  $\mathbf{G}_t$  into
$$
\boldsymbol {G} _ {t} = \underbrace {\frac {\mu_ {1}}{n \sqrt {d h}} \boldsymbol {v} \left(\boldsymbol {y} - f _ {t} (\boldsymbol {X})\right) \boldsymbol {X} ^ {\top}} _ {\boldsymbol {A} ^ {t}} + \underbrace {\frac {1}{n \sqrt {d h}} \left(\boldsymbol {v} \left(\boldsymbol {y} - f _ {t} (\boldsymbol {X})\right) \odot \sigma_ {\perp} ^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right)\right) \boldsymbol {X} ^ {\top}} _ {\boldsymbol {B} ^ {t}}. \tag {21}
$$
(a) First PC in trained weight.
(b) First PC in trained CK.
(c) First PC in trained NTK.
Figure 26: Additional performance for Adam with learning rate  $\eta = 0.002$  and 700 epochs, where  $n = 2000$ ,  $h = 1500$ ,  $d = 1000$ ,  $\sigma_{\varepsilon} = 0.3$  and batch size is 64. The activation  $\sigma$  is normalized softplus and the target is normalized tanh. The final test loss is 0.23954 and  $R^2$  score is around 0.76027. The orange arrows show the positions of the outliers. Spectra behaviors in this case differ from Figure 24.
Figure 25: Alignment between the leading PC of the weight/kernel matrices and the signal  $\beta$  or the training labels  $y$  before/after training for the experiment in Figure 24. This is analogous to Case 4 in Appendix B.2.
At first, consider  $t = 0$  and bound the spectral norm of  $W_{1}$ . By assumption, we know  $\| \pmb{v} \| \leq \sqrt{h}$ . Due to Corollary 7.3.3 in [80], we have
$$
\mathbb {P} \left(\frac {1}{\sqrt {d}} \| \boldsymbol {X} \| \geq 2 \left(1 + \sqrt {\frac {n}{d}}\right)\right) \leq 2 \exp (- c n). \tag {22}
$$
Therefore, by (21), we can control  $A^0$  and  $B^0$  separately. Notice that, as a rank-one matrix,
$$
\begin{array}{l} \left\| \boldsymbol {A} ^ {0} \right\| = \left\| \boldsymbol {A} ^ {0} \right\| _ {F} \leq \frac {\mu_ {1}}{\sqrt {n}} \frac {\| \boldsymbol {X} \|}{\sqrt {d}} \frac {1}{\sqrt {n}} (\| \boldsymbol {y} \| + \| f _ {0} (\boldsymbol {X}) \|) \frac {\| \boldsymbol {v} \|}{\sqrt {h}} \\ \leq \frac {\mu_ {1}}{\sqrt {n}} \frac {\| \boldsymbol {X} \|}{\sqrt {d}} \frac {\| \boldsymbol {v} \|}{\sqrt {h}} \frac {1}{\sqrt {n}} \left(\| \boldsymbol {y} \| + \frac {\| \boldsymbol {v} \|}{\sqrt {h}} \left\| \sigma (\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}) \right\|\right). \\ \end{array}
$$
Hence, by Lemma C.1 and (22), one can easily claim that  $\| A^0\| \leq C / \sqrt{n}$  with probability at least  $1 - e^{-cn}$  for some constants  $c,C > 0$  . On the other hand, since  $\pmb {v}\left(\pmb {y} - f_t(\pmb {X})\right)$  is rank-one and  $\sigma_{\perp}^{\prime} = \sigma^{\prime} - \mu_{1}$  with  $|\sigma '(x)|\leq \lambda_{\sigma}$  , we can similarly obtain
$$
\begin{array}{l} \left\| \boldsymbol {B} ^ {0} \right\| _ {F} \leq \frac {1}{n \sqrt {d h}} \left\| \boldsymbol {v} \left(\boldsymbol {y} - f _ {t} (\boldsymbol {X})\right) \odot \sigma_ {\perp} ^ {\prime} \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) \right\| _ {F} \| \boldsymbol {X} \| \\ \leq \frac {1}{n \sqrt {h d}} \| \boldsymbol {X} \| \left(\| \boldsymbol {y} \| + \| f _ {0} (\boldsymbol {X}) \|\right) \| \boldsymbol {v} \| \max  _ {i, j} \left| \sigma_ {\perp} ^ {\prime} \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right| _ {i, j} \\ \leq \frac {\mu_ {1} + \lambda_ {\sigma}}{\sqrt {n}} \frac {\| \boldsymbol {X} \|}{\sqrt {d}} \frac {\| \boldsymbol {v} \|}{\sqrt {h}} \frac {1}{\sqrt {n}} \left(\| \boldsymbol {y} \| + \frac {\| \boldsymbol {v} \|}{\sqrt {h}} \left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\|\right). \\ \end{array}
$$
As  $A^0$ , we can apply Lemma C.1 and (22) again to conclude (20) for  $t = 1$ .
For general  $t$ , we apply induction. We assume that after the  $t$ -th gradient step with  $\eta = \Theta(1)$ , Eq. (20) holds for some constants  $C, c > 0$ . Following [6, Lemma 16], we now show that the similar high-probability statement also holds for  $\mathbf{W}_{t+1}$  (for some different constants  $c', C'$ ). Firstly,
(a) First PC in trained & initial weights.
(c) First PC in trained & initial NTKs.
Figure 27: Alignment between the leading PC of the weight/kernel matrices and the signal  $\beta$  or the training labels  $y$  before/after training for the experiment in Figure 26. We can observe strong alignments, in this case, comparing with Figure 25 because of outliers in above Figure 26. These kernel alignments induce anisotropic structures in the kernel matrices during training [78].
following the same argument as [71, Setion 6.6.1], we know that
$$
\begin{array}{l} \left\| f _ {t} (\boldsymbol {X}) \right\| \leq \left\| f _ {0} (\boldsymbol {X}) \right\| + \left\| f _ {t} (\boldsymbol {X}) - f _ {0} (\boldsymbol {X}) \right\| \\ \leq \left\| f _ {0} (\boldsymbol {X}) \right\| + \frac {\lambda_ {\sigma}}{\sqrt {h}} \left\| \boldsymbol {v} \right\| \frac {\left\| \boldsymbol {X} \right\|}{\sqrt {d}} \left\| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \right\| _ {F}. \tag {23} \\ \end{array}
$$
Note that  $\| W_t - W_0\|_F = O(1/\sqrt{n})$  with high probability by the induction hypothesis. Hence, by Lemma C.1 and (22), we have  $\| f_t(\mathbf{X})\| \leq C\sqrt{n}$  with high probability. Indeed, the difference between  $f_t(\mathbf{X})$  and  $f_0(\mathbf{X})$  is significantly negligible comparing with the initial value  $f_0(\mathbf{X})$ . Similarly with  $\mathbf{A}_0$ ,  $\mathbf{A}^t$  satisfies
$$
\left\| \boldsymbol {A} ^ {t} \right\| = \left\| \boldsymbol {A} ^ {t} \right\| _ {F} \leq \frac {\mu_ {1}}{\sqrt {n}} \frac {\| \boldsymbol {X} \|}{\sqrt {d}} \frac {1}{\sqrt {n}} (\| \boldsymbol {y} \| + \| f _ {t} (\boldsymbol {X}) \|) \frac {\| \boldsymbol {v} \|}{\sqrt {h}}.
$$
Analogously for  $B^t$ , we have
$$
\left\| \boldsymbol {B} ^ {t} \right\| _ {F} \leq \frac {\mu_ {1} + \lambda_ {\sigma}}{\sqrt {n}} \frac {\| \boldsymbol {X} \|}{\sqrt {d}} \frac {\| \boldsymbol {v} \|}{\sqrt {h}} \frac {1}{\sqrt {n}} (\| \boldsymbol {y} \| + \| f _ {t} (\boldsymbol {X}) \)).
$$
Thus, Lemma C.1, (22), and (23) ensure that
$$
\mathbb {P} \left(\left\| \boldsymbol {A} ^ {t} \right\| _ {F} \geq \frac {C ^ {\prime}}{\sqrt {n}}\right) \leq \exp (- c ^ {\prime} n), \mathbb {P} \left(\left\| \boldsymbol {B} ^ {t} \right\| _ {F} \geq \frac {C ^ {\prime}}{\sqrt {n}}\right) \leq \exp (- c ^ {\prime} n),
$$
for constants  $c', C' > 0$ . Since  $\| \pmb{W}_{t+1} - \pmb{W}_0 \|_F \leq \| \pmb{W}_t - \pmb{W}_0 \|_F + \eta \| \pmb{A}^t \|_F + \eta \| \pmb{B}^t \|_F$ , by induction hypothesis, we can conclude that (20) holds for the  $(t+1)$ -th step with some constants  $C, c > 0$ , which are different from the constants at the  $t$ -th step.
As a corollary, by (17), we can also deduce the following norm bounds:
$$
\mathbb {P} \left(\| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \| \geq \frac {C}{\sqrt {n}}\right) \leq \exp (- c n), \quad \mathbb {P} \left(\| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \| _ {2, \infty} \geq \frac {C}{\sqrt {n}}\right) \leq \exp (- c n).
$$
Lemma C.2 and the above bounds are empirically verified by Figure 28(a) for  $t = 3$ . Not only upper bounds, this simulation also shows that at early phase  $\| W_{t} - W_{0}\|$ ,  $\| W_{t} - W_{0}\|_{F}$ , and  $\| W_{t} - W_{0}\|_{2,\infty}$  are all of the same  $\Theta (1 / \sqrt{n})$  order.
As a remark, from the bound of the second term of (23), we can deduce that the change of the output of the NN satisfies
$$
\left| f _ {t} (\boldsymbol {x}) - f _ {0} (\boldsymbol {x}) \right| \leq \frac {C}{\sqrt {n}},
$$
for some  $t$ -dependent constant  $C > 0$ , any  $\pmb{x} \sim \mathcal{N}(0, \mathbf{I})$  and any finite time  $t$ . In other words, when  $\eta = \Theta(1)$ , the change of the output of the NN at the early phase (i.e.  $t = \Theta(1)$ ) is negligible and its order is  $O\left(\frac{1}{\sqrt{n}}\right)$ .
C.2 Proof of Lemma 4.1
In this section, we complete the proof of Lemma 4.1. We first mention the empirical validation of Lemma 4.1 in Figure 28. Notice that the changes in Frobenius norm for  $\mathbf{W}$  and  $K^{\mathrm{CK}}$  are exactly  $\Theta(1/\sqrt{n})$  and  $\Theta(1/n)$ , respectively. The operator norm of  $K^{\mathrm{NTK}}$  matches with Lemma 4.1, while the Frobenius norm of the change decays slower than the rate  $\Theta(1/n)$ . Additionally, in the simulation, we use  $v \sim \mathcal{N}(0, \mathbf{I})$ , which indicates that our assumption for  $v$  in Lemma 4.1 can be weakened.
(a) Early phase for  $\mathbf{W}_t$ .
(b) Early phase for  $K^{\mathrm{CK}}$
Figure 28: Empirical validations for Lemma 4.1 and Lemma C.2 at  $t = 3$ . Here  $\sigma_{\varepsilon} = 0.2$ , activation  $\sigma$  is a normalized ReLU and the target function  $\sigma^{*}$  is normalized tanh. Fix  $d / n = 0.6$  and  $N / n = 1.2$  as  $n$  is increasing. At each dimension, we take 25 trials to average. (a) Norms of the changes for  $\pmb{W}_{3} - \pmb{W}_{0}$ . (b) Norms of the changes for  $K_3^{\mathrm{CK}} - K_0^{\mathrm{CK}}$ . (c) Norms of the changes for  $K_3^{\mathrm{NTK}} - K_0^{\mathrm{NTK}}$ .
(c) Early phase for  $K^{\mathrm{NTK}}$
Proof of Lemma 4.1. Lemma C.2 directly validates the control of  $\frac{1}{\sqrt{d}}\| W_t - W_0\|_F$ . By virtue of this result, we now present estimates for CK and NTK. Based on [71, Section 6.6.1], we have
$$
\left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) \right\| \leq \frac {\lambda_ {\sigma}}{\sqrt {d}} \| \boldsymbol {X} \| \| \boldsymbol {W} _ {0} - \boldsymbol {W} _ {t} \| _ {F}. \tag {24}
$$
We apply the mean value theorem to obtain this inequality. Recall the operator norm bound for Gaussian random matrix  $\mathbf{X}$  in (22). We know  $\| \mathbf{X} / \sqrt{d}\| \lesssim 1 + \sqrt{\gamma_1}$  with high probability as  $n / d\to \gamma_{1}$ . Hence, with the help of Lemma C.2, we can claim
$$
\left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) \right\| \leq C \lambda_ {\sigma} (1 + \sqrt {\gamma_ {1}}) / \sqrt {n},
$$
with probability at least  $1 - \exp (-cn)$ , for any fixed finite  $t\in [n]$ . Similarly, we can control the change in the Frobenius norm as follows:
$$
\left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) \right\| _ {F} ^ {2} \leq \frac {\lambda_ {\sigma} ^ {2}}{d} \| \boldsymbol {X} \| ^ {2} \| \boldsymbol {W} _ {0} - \boldsymbol {W} _ {t} \| _ {F} ^ {2} \leq C \lambda_ {\sigma} ^ {2} (1 + \sqrt {\gamma_ {1}}) ^ {2} / n, \tag {25}
$$
with probability at least  $1 - \exp (-cn)$ . Therefore, we can control the change in the CK matrix in the Frobenius norm by the following inequalities:
$$
\begin{array}{l} \left\| \boldsymbol {K} _ {t} ^ {\mathrm {C K}} - \boldsymbol {K} _ {0} ^ {C K} \right\| _ {F} \\ \leq \frac {1}{h} \left\| \sigma (\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}) ^ {\top} \left(\sigma (\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}) - \sigma (\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d})\right) \right\| _ {F} + \frac {1}{h} \left\| \sigma (\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}) ^ {\top} \left(\sigma (\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}) - \sigma (\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d})\right) \right\| _ {F} \\ \leq \frac {1}{h} \left(\left\| \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| + \left\| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d} \right\| \right)\right) \cdot \left\| \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \right\| _ {F} \\ + \frac {1}{h} \| \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \| \cdot \| \sigma \left(\boldsymbol {W} _ {t} \boldsymbol {X} / \sqrt {d}\right) - \sigma \left(\boldsymbol {W} _ {0} \boldsymbol {X} / \sqrt {d}\right) \| _ {F}. \\ \end{array}
$$
Therefore, by (24), (25) and Lemma C.1, we can claim that there exist constants  $c, C > 0$  such that with probability at least  $1 - \exp(-cn)$ ,  $\left\| \boldsymbol{K}_t^{\mathrm{CK}} - \boldsymbol{K}_0^{CK} \right\|_F$  is upper bounded by  $C/n$  in the LWR.
Now we consider the change in the NTK matrix during training. Since the empirical NTK can be decomposed into two parts, one of which is exactly the CK, it suffices to consider the change of the first part of the empirical NTK. Recall that
$$
\boldsymbol {K} _ {t} := \frac {1}{d} \boldsymbol {X} ^ {\top} \boldsymbol {X} \odot \frac {1}{h} \sigma^ {\prime} \left(\frac {1}{\sqrt {d}} \boldsymbol {W} _ {t} \boldsymbol {X}\right) ^ {\top} \operatorname {d i a g} (\boldsymbol {v} _ {t}) ^ {2} \sigma^ {\prime} \left(\frac {1}{\sqrt {d}} \boldsymbol {W} _ {t} \boldsymbol {X}\right).
$$
Following the notation in [71], we denote  $\mathcal{J}(\pmb{W}_t) \coloneqq [\mathcal{J}(\pmb{w}_1^t), \dots, \mathcal{J}(\pmb{w}_N^t)] \in \mathbb{R}^{n \times hd}$  with  $\mathcal{J}(\pmb{w}_i) \coloneqq \frac{v_i}{\sqrt{h}} \mathrm{diag}(\sigma'(\pmb{X}^\top \pmb{w}_i / \sqrt{d})) \pmb{X}^\top / \sqrt{d} \in \mathbb{R}^{n \times d}$ . Hence,  $\pmb{K}_t = \mathcal{J}(\pmb{W}_t) \mathcal{J}(\pmb{W}_t)^\top$  and