SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhichao Wang  
University of California San Diego  
zhw036@ucsd.edu
Andrew Engel  
Pacific Northwest National Laboratory  
andrew.engel@pnnl.gov
Anand Sarwate  
Rutgers, The State University of New Jersey  
ads221@soe.rutgers.edu
Ioana Dumitriu  
University of California San Diego  
idumitriu@ucsd.edu
Tony Chiang  
Pacific Northwest National Laboratory  
University of Washington  
University of Texas at El Paso  
tony.chiang@pnnl.gov
We investigate the spectral properties of linear-width feed-forward neural networks, where the sample size is asymptotically proportional to network width. Empirically, we show that the spectra of weight in this high dimensional regime are invariant when trained by gradient descent for small constant learning rates; we provide a theoretical justification for this observation and prove the invariance of the bulk spectra for both conjugate and neural tangent kernels. We demonstrate similar characteristics when training with stochastic gradient descent with small learning rates. When the learning rate is large, we exhibit the emergence of an outlier whose corresponding eigenvector is aligned with the training data structure. We also show that after adaptive gradient training, where a lower test error and feature learning emerge, both weight and kernel matrices exhibit heavy tail behavior. Simple examples are provided to explain when heavy tails can have better generalizations. We exhibit different spectral properties such as invariant bulk, spike, and heavy-tailed distribution from a two-layer neural network using different training strategies, and then correlate them to the feature learning. Analogous phenomena also appear when we train conventional neural networks with real-world data. We conclude that monitoring the evolution of the spectra during training is an essential step toward understanding the training dynamics and feature learning.
Deep learning theory has made insightful connections between the behavior of neural networks (NNs) and kernel machines through asymptotic analyses of the so-called kernel regime [65, 83, 48, 40, 12, 4, 84]. When the neural network (NN) is infinitely wide, the behavior of NN coincides with a kernel machine, and the training process, as well as the generalization performance of this ultra-wide NN, can be fully described. The performance of finite-width NNs, however, does not correspond to this theory, as NNs optimized with gradient-based methods perform better than infinitely wide networks in many circumstances [47, 38, 34, 51, 25, 32, 76, 42]. This gap heavily relies on the task complexity, data distribution, architecture of the NN and the training strategy [31]. We consider a more realistic
setting, a linear-width regime (LWR), when the sample size  $n$ , the input feature dimension  $d$ , and the width  $h$  of the hidden layer approach infinity at comparable rates. Under the LWR, we aim to empirically study this theoretical gap in generalization and spectral properties by training various NNs with different optimization tools.
The ultra-wide NN ( $h \gg n$ , fixed  $d$ ) stays close to the kernel machine induced by initial NN, throughout the gradient-based training processes [88, 28, 27, 11]. There are two kernels commonly studied in theory: the conjugate kernel (CK) and the neural tangent kernel (NTK). CK (or the equivalent Gaussian process kernel) is the Gram matrix of the last hidden layer, which represents training only the last layer of the network [48, 57, 61]; by contrast, the NTK is the Gram matrix of the Jacobian of the NN for all trainable parameters, which governs the gradient flow of NN [40, 28, 4]. In most theoretical results, these kernels remain fixed throughout training, which leads to a kernel gradient descent with the initial kernel [40, 16], whereas in practice the spectra of the weight matrix, CK, and NTK of the NN change while learning the features from the training data [58, 59, 30, 19, 68]. In this paper, under the LWR, we experimentally and theoretically explore the following question:
How do the spectra of weight and kernel matrices of the NN evolve during the training process?
This question is crucial to extend our understanding beyond the kernel regime and will help us analyze the generalization of the NN in instances when it performs better than the kernel machine. For this case, the spectral properties of the trained NN could be entirely different from the initial kernel [55, 10, 78]. Also, various spectral properties of weight and kernel matrices can reveal different features learned by different training procedures [82]. Understanding the dynamics of the spectral properties may aid in finding better approaches to training and tuning hyper-parameters for NNs. From a theoretical perspective, random matrix theory (RMT) can be further exploited to study and elucidate the NN training under the proportional limit in high dimensions [45, 72, 57, 74, 37, 61].
Our main findings/contributions are as follows.
- We find a simple scenario that exhibits different spectral properties for both weight and kernel matrices through different training procedures. With the kernel regime as a benchmark, we compare how NN generalizes with different spectral evolutions of weight and kernel matrices in NN.
- The spectra of NNs trained with full batch gradient descent (GD) are globally invariant, indicating that the NN is still close to a kernel machine; we prove the global convergence of GD and the invariance of the limiting spectra for both weight and kernel matrices in this scenario.
- We observe a phase transition of the alignment and the emergence of a spike outside the bulk of the spectrum when the learning rate exceeds some threshold. The strong alignment of the spike with the teacher model when step sizes are large confirms that the NN is indeed learning germane features from data. This observation justifies the theoretical result of [6] in an ideal two-stage training process.
- The evolution towards heavy-tailed spectra is also discovered by using adaptive methods. Our experiments rule out a causal relationship between the occurrence of a heavy-tailed spectrum for the weight matrices and a good generalization. This complements the work of [60, 63, 86] where the authors had observed a strong correlation between the two; while at the same time, we provide simple examples of when heavy-tailed spectra exhibit feature learning and better generalizations.
For more details on how our results fit into existing literature, please see Appendix A.
Throughout this paper,  $\| \cdot \|$  denotes the  $\ell_2$  norm for vectors,  $\ell_2 \to \ell_2$  is the operator norm for matrices, while  $\| \cdot \|_F$  is the Frobenius norm, and  $\odot$  represents the Hadamard product between matrices.  $o_{d,\mathbb{P}}(\cdot)$  represents little-o in probability as  $d \to \infty$ .
Neural Tangent Kernel Parameterization. Consider a  $L$ -layer fully connected feedforward NN at initialization without bias term: for  $1 \leq \ell \leq L - 1$
$$
\boldsymbol {h} _ {0} = \frac {\boldsymbol {x}}{\sqrt {d}}, \quad \boldsymbol {h} ^ {(\ell)} = \frac {1}{\sqrt {n _ {\ell}}} \sigma \left(\boldsymbol {W} _ {\ell} \boldsymbol {h} ^ {(\ell - 1)}\right), \tag {1}
$$
and  $f_{\theta}(\pmb{x}) = \pmb{v}^{\top}\pmb{h}^{(L - 1)}$ , where the input vector is  $\pmb{x} \in \mathbb{R}^d$ ,  $\pmb{W}_{\ell} \in \mathbb{R}^{n_{\ell} \times n_{\ell - 1}}$  is the weight matrix for the  $\ell$ -th layer, and  $\pmb{v} \coloneqq [v_1, \dots, v_h]^{\top} \in \mathbb{R}^{n_L - 1}$  is the last-layer weight. Let  $n_0 = d$ . Denote all trainable parameters by  $\pmb{\theta} \coloneqq [\mathrm{vec}(W_1), \dots, \mathrm{vec}(W_{L - 1}), \pmb{v}]^{\top} \in \mathbb{R}^p$  where each parameter's initial value is independently sampled from some distribution and  $p$  is the total number of parameters. Let the training dataset be  $(\pmb{X}, \pmb{y}) \coloneqq ([\pmb{x}_1, \dots, \pmb{x}_n], \pmb{y}) \in \mathbb{R}^{d \times n} \times R^{1 \times n}$ ; the output of this NN with respect to this dataset is  $f_{\theta}(\pmb{X}) = [f_{\theta}(\pmb{x}_1), \dots, f_{\theta}(\pmb{x}_n)]$ . We call the above parameterization the NTK parameterization. The loss function for training is a mean squared error (MSE)
$$
\mathcal {L} (\boldsymbol {\theta}) := \frac {1}{2 n} \| \boldsymbol {y} - f _ {\boldsymbol {\theta}} (\boldsymbol {X}) \| ^ {2}. \tag {2}
$$
We focus on the NTK parameterization and consider the kernel machine (6) induced by the initial NTK of the NN. We aim to seek the cases when the NN outperforms this kernel during the training process. For this purpose, we adopt different optimizers of training this NN to obtain different testing performances and spectral properties of trained weights and empirical kernels.
Training Processes of NNs. NNs are usually trained by gradient-based methods such as full-batch gradient descent (GD), mini-batch stochastic gradient descent (SGD), Adaptive Gradients (AdaGrad), and Adam [43]. We can represent GD by
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} - \eta \nabla_ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta} _ {t}), \tag {3}
$$
where  $\eta$  is the learning rate and  $\nabla_{\theta}\mathcal{L}(\theta_t)$  is the gradient of the training loss w.r.t. trainable parameters  $\theta$  at step  $t\geq 0$ . We will prove the global convergence of GD in some special (overparameterized) cases ensuring the convergence to a NN that interpolates the data. We will also show the hyperparameters (e.g. learning rate  $\eta$ ) affect the spectral properties of NNs during training.
Conjugate Kernel and Neural Tangent Kernel. When  $L = 2$ , let  $n_1 = h$  and  $n_0 = d$  be the widths of the output and input layer. The CK is defined as
$$
\boldsymbol {K} ^ {\mathrm {C K}} := \boldsymbol {X} _ {1} ^ {T} \boldsymbol {X} _ {1} \in \mathbb {R} ^ {n \times n}, \tag {4}
$$
where  $X_{1} \coloneqq \frac{1}{\sqrt{h}} \sigma \left( W X / \sqrt{d} \right)$ . We can view the NN as a function of all training parameters  $\theta$  and input data  $X$ . The neural tangent kernel (NTK) is related to the gradient of this neural network function with respect to  $\theta$ , which is the Gram matrix of the Jacobian of the neural network function with respect to  $\theta$ .  $K^{\mathrm{NTK}} \coloneqq (\nabla_{\theta} f_{\theta}(X))^{\top} (\nabla_{\theta} f_{\theta}(X))$ . Specifically, the empirical NTK of two-layer NN can be explicitly written<sup>2</sup> as
$$
\boldsymbol {K} ^ {\mathrm {N T K}} = \frac {1}{d} \boldsymbol {X} ^ {\top} \boldsymbol {X} \odot \frac {1}{h} \sigma^ {\prime} \left(\frac {1}{\sqrt {d}} \boldsymbol {W} \boldsymbol {X}\right) ^ {\top} \operatorname {d i a g} (\boldsymbol {v}) ^ {2} \sigma^ {\prime} \left(\frac {1}{\sqrt {d}} \boldsymbol {W} \boldsymbol {X}\right) + \boldsymbol {K} ^ {\mathrm {C K}}. \tag {5}
$$
In this paper, we are interested in comparing the spectral distributions for these three matrices (weight, CK, and NTK) at initialization and the end of training.
Lazy Training. Lazy training [21] can be viewed as a linear approximation of the NN, i.e.  $f_{\theta}(\pmb {x})\approx f_{\theta_0}(\pmb {x}) + (\pmb {\theta} - \pmb {\theta}_0)^\top \nabla_\pmb {\theta}f_{\theta_0}(\pmb {x})$ , defined by minimum-norm interpolation  $\hat{\pmb{\theta}} := \arg \min \left\{\| \pmb {\theta} - \pmb {\theta}_0\| :(\pmb {\theta} - \pmb {\theta}_0)^\top \nabla_\pmb {\theta}f_{\pmb {\theta}_0}(\pmb {X}) = \pmb {y} - f_{\pmb {\theta}_0}(\pmb {X})\right\}$ . Then, lazy training also represents a kernel machine
$$
\hat {f} (\boldsymbol {x}) = f _ {\theta_ {0}} (\boldsymbol {x}) + \left(\boldsymbol {y} - f _ {\theta_ {0}} (\boldsymbol {X})\right) \boldsymbol {K} (\boldsymbol {X}, \boldsymbol {X}) ^ {- 1} \boldsymbol {K} (\boldsymbol {X}, \boldsymbol {x}) \tag {6}
$$
In this work, we only consider empirical conjugate and neural tangent kernels of finite-width NNs.
Here we train both layers, so we have two parts in the NTK expression. If we only train the first-hidden layer, we can simply remove the second CK part. In the following, we further introduce more empirical results for practical NNs in Section 5 and two-layer NNs with Gaussian dataset in Section 3. For general formula of the empirical NTK, see [39, 30].
where  $\hat{f}(\pmb{x})$  is the unregularized regression prediction on test data  $\pmb{x} \in \mathbb{R}^d$ , the kernel  $\pmb{K}(\pmb{X}, \pmb{X})$  is the initial  $\pmb{K}^{\mathrm{NTK}}$  on training data, and  $\pmb{K}(\pmb{X}, \pmb{x}) = (\nabla_{\theta} f_{\theta_0}(\pmb{X}))^\top (\nabla_{\theta} f_{\theta_0}(\pmb{x}))$ . The asymptotic performance of  $\hat{f}(\pmb{x})$  has been analyzed by [1] under the LWR. We view this regime as a benchmark: [21, 11] prove that NN through gradient flow is close to lazy training if  $h \gg n$ ; [38] shows NN can go beyond lazy training under a non-proportional regime.
3 Case Study for Linear-width NNs
In this section, we investigate a two-layer NN with synthetic data. This setting is promising for future theoretical studies by virtue of RMT. We will showcase the evolution of its spectral properties over training. A two-layer NN in (1) is defined by
$$
f _ {\boldsymbol {\theta}} (\boldsymbol {x}) := \frac {1}{\sqrt {h}} \sum_ {i = 1} ^ {h} v _ {i} \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} / \sqrt {d}\right). \tag {7}
$$
At initialization, we assume that the first hidden-layer  $\pmb{W} = [\pmb{w}_1,\dots ,\pmb{w}_h]^\top \in \mathbb{R}^{h\times d}$  is composed of independent standard normal random vectors.
Assumption 3.1 (Linear-width regime (LWR)). Assume that  $\frac{n}{d} \to \gamma_1$  and  $\frac{h}{d} \to \gamma_2$  as  $n \to \infty$  where the aspect ratios  $\gamma_1, \gamma_2 \in (0, \infty)$  are two fixed constants.
LWR stands as a pivotal setting grounded in high-dimensional statistics [1, 61]. It offers valuable insights especially when addressing real-world datasets. This is in contrast to the infinite-width regime, in which we are already in the asymptotic limit for width at first. Hence, LWR is a better approximation of real-world datasets and practical neural networks compared with the infinite-width regime.
Assumption 3.2 (Activation function). Suppose that the activation function  $\sigma(x)$  is nonlinear and  $\lambda_{\sigma}$ -Lipschitz with  $|\sigma'(x)|, |\sigma''(x)| \leq \lambda_{\sigma}$  for all  $x \in \mathbb{R}$ . Moreover,  $\mathbb{E}[\sigma(z)] = 0$  for  $z \sim \mathcal{N}(0,1)$ .
Though the LWR is somewhat impractical, it is still more aligned with deployed models than the infinite-width regime ( $h \gg n$ , fixed  $d$ ). As a kernel machine, the infinite-width NN has been studied extensively [40, 28, 27, 84]. This infinite-width limit is special, however, as NNs may generally evolve beyond the kernel regime and achieve superior performance [31, 55, 10, 78].
Optimization	Learning rate η	R2score	Test error	Spectra
Case 1	GD	5.0	0.63582	0.36381	Invariant Bulk
Case 2	SGD	0.1	0.60605	0.36879	Invariant Bulk
Case 3	SGD	22.0	0.76081	0.23791	Bulk+spike
Case 4	Adam	0.092	0.78829	0.21071	Heavy tail
Lazy regime	0.68092	0.3185
Table 1: Four models with the same architecture ( $n = 2000$ ,  $h = 1500$ ,  $d = 1000$ , and  $\sigma$  is normalized  $\tanh$ ), but different choices of initial learning rates and optimizers listed in Table 1. The training label noise  $\sigma_{\varepsilon} = 0.3$  and the teacher model is defined by (9) with  $\sigma^{*}$  a normalized softplus and  $\tau = 0.2$ . We observe that simply choosing an optimizer and learning rate can affect the shapes of the final spectra and the performance of the NN, as measured by  $R^2$  scores and test errors.
Assumption 3.3 (Synthetic dataset and teacher model). Training data is  $\mathbf{X} \coloneqq [\pmb{x}_1, \dots, \pmb{x}_n] \in \mathbb{R}^{d \times n}$ , where  $\pmb{x}_i \stackrel{i.i.d.}{\sim} \mathcal{N}(\mathbf{0}, \mathbf{I}_d)$ . The training labels  $\pmb{y} = [y_1, \dots, y_n]$  are defined by  $y_i = f^*(\pmb{x}_i) + \varepsilon_i$ , for  $i \in [n]$ , where  $f^* : \mathbb{R}^d \to \mathbb{R}$  is the teacher model, and  $\varepsilon_i$  is centered sub-Gaussian noise with variance  $\sigma_{\varepsilon}^2$ .
One of the simplest nonlinear teacher models we can generate is the single-index model, namely  $f^{*}(\pmb{x}) = \sigma^{*}(\pmb{x}^{\top}\beta)$  for a fixed vector  $\beta$  with  $\| \beta \| = 1$  and nonlinear function  $\sigma^{*}$ ; the hidden feature is simply  $\beta \in \mathbb{R}^{d}$ . In general, we can consider a multiple-index model
$$
f ^ {*} (\boldsymbol {x}) = \frac {1}{k} \sum_ {i = 1} ^ {k} \sigma^ {*} \left(\boldsymbol {x} ^ {\top} \boldsymbol {\beta} _ {i}\right) \tag {8}
$$
Figure 1: Different spectral behaviors in Table 1: (a) The initial and trained spectra of  $W$  in Case 1. The spectrum is invariant based on the Q-Q subplot. (b) The initial and trained spectra of  $K^{\mathrm{CK}}$  in Case 3. There is an outlier (orange arrow) in the spectrum after training. (c) The initial and trained spectra of  $K^{\mathrm{CK}}$  in Case 4. We refer to Appendix B.2 for other spectra of weight, CK, and NTK matrices in Case 1-4, where analogous phenomena hold for other matrices.
where  $\beta_{i}$  are some orthogonal unit vectors. We will specifically consider a mixture of single-index and quadratic models as our teacher model in this section:
$$
f ^ {*} (\boldsymbol {x}) = \sigma^ {*} \left(\boldsymbol {x} ^ {\top} \boldsymbol {\beta}\right) + \frac {\tau}{d} \| \boldsymbol {x} \| ^ {2}, \tag {9}
$$
for some nonlinear target  $\sigma^{*}$ , signal  $\beta$  and constant  $\tau^3$ . Following the above assumptions and constructions, we show different spectral properties (Figure 1) for this two-layer NN using different training procedures (Table 1). Figure 1 exhibits three types of spectra after training: unchanged bulk distribution, bulk with one spike, and heavy tail in spectra. Putting things together, we can see close relationships between the spectra and the generalization of the NN. These different spectral properties actually reveal disparate features learned via different training strategies.
The advantage of this toy model is that we can easily extract the spectral behaviors over training and then compare them with the kernel machine. We use lazy training defined in (6) as our benchmark to assist us in determining whether a NN outperforms the associated kernel machine. Table 1 compares the test errors and  $R^2$  scores for different optimization cases and the lazy training. By tuning the hyper-parameters, we can find specific situations where NN outperforms the lazy training (see also Figure 10(c) in Appendix B.2).
From Figures 1(a), 12 and 13 in Appendix B.2, one can observe the spectral distributions of the weight, CK and NTK matrices remain invariant and static during training in Cases 1&2, which indicates both cases still belong to the lazy regime. This spectral invariance impedes further feature learning during the training process. The emergence of the outlier in Figures 1(b) and 14 of Appendix B.2, however, shows the improvement over lazy training and potential feature learning via the training process, where the spectra possibly inherit the structures in teacher models (see Section 4.2). Comparing with Case 2, Case 3 of Table 1 suggests the importance of the large learning rate regime for training NNs [52, 64, 55, 15, 3]. As a remark, our spectral results of Case 3 are consistent with the observations in [79] through RMT hypothesis testing, where the majority of trained weight matrices remain random, and the learned feature may be contained in the largest singular value (outlier) and associated vector only. From Figures 1(c) and 16 in Appendix B.2, Case 4 further exhibits more spikes and heavy tails in the trained spectra, which thoroughly goes beyond the realm of initial kernel machine. Notably, this phenomenon is not unique to Adam since heavy tails also occur with AdaGrad in Figure 22 in Appendix B.6. Although all of these cases have the same identical initialization, different methods of optimization eventually lead to various training trajectories and evolutions of the spectra of the weight and kernel matrices. To acquire feature learning, Cases 3&4 cause weights to deviate far from initialization. In the following Section 4, we prove the invariance of the bulk distributions and provide more refined analyses of spikes and heavy tails in terms of feature learning.
Here, the norm term of  $\pmb{x}$  in (9) is designed to make the teacher model more complicated to be learned. All our empirical results still hold when  $\tau = 0$ .
4 Different Spectral Behaviors in NNs
We now further explore the spectral behaviors in different cases of Table 1 by clarifying how the spectra evolve through different training processes and how this evolution may affect the NN. Following Figure 1, we study the training processes case-by-case: invariant bulk, spikes outside the bulk, and heavy-tailed distribution. Additional experiments are exhibited in Appendix B.
4.1 Invariant Bulk Distributions
In Figure 1(a) (also Figures 12 and 13 in Appendix B.2), we observe the bulk distributions of weight and kernel matrices in Cases 1&2 remain globally unchanged (invariant) over the training process. under the LWR, this is also empirically verified by Figures 10(a)&(b) in Appendix B.2. In this section, by investigating the global convergence of GD, we prove this invariant-bulk phenomenon under certain assumptions.
For simplicity, we focus on analyzing the training process of the first-hidden layer with the second layer  $\pmb{v}$  fixed. Denote  $f_{\theta}(\pmb{X})$  by  $f_{\pmb{W}}(\pmb{X})$  in this case. At any time  $t \in \mathbb{N}$ , consider the gradient steps:
$$
\boldsymbol {W} _ {t + 1} = \boldsymbol {W} _ {t} - \eta \nabla_ {\boldsymbol {W}} \mathcal {L} (\boldsymbol {W} _ {t}). \tag {10}
$$
Denote the CK and NTK at gradient step  $t \in \mathbb{N}$  by  $K_{t}^{\mathrm{CK}} := \frac{1}{h}\sigma(\boldsymbol{W}_{t}\boldsymbol{X})^{\top}\sigma(\boldsymbol{W}_{t}\boldsymbol{X})$ , and  $K_{t}^{\mathrm{NTK}} := \frac{1}{d}\boldsymbol{X}^{\top}\boldsymbol{X} \odot \frac{1}{h}\sigma'\left(\frac{1}{\sqrt{d}}\boldsymbol{W}_{t}\boldsymbol{X}\right)^{\top}\operatorname{diag}(\boldsymbol{v}_{t})^{2}\sigma'\left(\frac{1}{\sqrt{d}}\boldsymbol{W}_{t}\boldsymbol{X}\right)$  respectively. First, we present an elaborate description of the changes in the weight, CK, and NTK at the early phase of the training (after any finite  $t$  steps) as follows.
Lemma 4.1 (Early phase). Under Assumptions 3.1, 3.2 and 3.3, we further assume that  $\| \pmb{v}\|_{\infty} \leq 1$  and  $f^{*}$  is a  $\lambda_{\sigma}$ -Lipschitz function. Given any fixed  $t \in \mathbb{N}$  and learning rate  $\eta = \Theta(1)$ , after  $t$  gradient steps, the changes  $\frac{1}{\sqrt{d}}\|\pmb{W}_t - \pmb{W}_0\|_F$ ,  $\|\pmb{K}_t^{CK} - \pmb{K}_0^{CK}\|_F$ , and  $\|\pmb{K}_t^{NTK} - \pmb{K}_0^{NTK}\|$  are all less than  $\frac{C}{n}$  with probability at least  $1 - 4n\exp(-cn)$ , for some positive constants  $c, C > 0$  which only depend on step  $t$  and parameters  $\eta, \gamma_1, \gamma_2, \lambda_\sigma, \sigma_\varepsilon$ .
Lemma 4.1 shows  $\frac{1}{\sqrt{d}}\| W_t - W_0\|$ ,  $\| K_t^{\mathrm{CK}} - K_0^{\mathrm{CK}}\|$ , and  $\| K_t^{\mathrm{NTK}} - K_0^{\mathrm{NTK}}\|$  are asymptotically vanishing for any fixed time  $t$ . Therefore, all the eigenvalues/eigenvectors are asymptotically unchanged at the early phase of the training (see Corollary C.3 in Appendix C). Now we aim to analyze the spectra at the end of the training process (10). In this case, although we are unable to show the invariance for each eigenvalue, we can verify the invariance of the limiting bulk distributions for  $K_t^{\mathrm{CK}}$  and  $K_t^{\mathrm{CK}}$  for all  $t$ .
By [81, Theorem 2.9], the smallest eigenvalue of  $K_0^{\mathrm{NTK}}$  has an asymptotic lower bound:
$$
\lambda_ {\min } \left(\boldsymbol {K} _ {0} ^ {\mathrm {N T K}}\right) \geq \left(a _ {\sigma} - \sum_ {k = 0} ^ {2} \eta_ {k} ^ {2}\right) (1 - o _ {d, \mathbb {P}} (1)), \tag {11}
$$