Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Authors: Xiangchao Li
Xiao Han
Qing Yang
Venue: ICML 2025 poster (2025)
Keywords: Deep learning theory
high-dimensional statistics
neural tangent kernel
random matrix theory
Primary area: theory->deep_learning

Abstract: In this paper, we investigate deep feedforward neural networks with random weights. The input data matrix $\boldsymbol{X}$ is drawn from a Gaussian mixture model. We demonstrate that certain eigenvalues of the conjugate kernel and neural tangent kernel may lie outside the support of their limiting spectral measures in the high-dimensional regime. The existence and asymptotic positions of such isolated eigenvalues are rigorously analyzed. Furthermore, we provide a precise characterization of the entrywise limit of the projection matrix onto the eigenspace associated with these isolated eigenvalues. Our findings reveal that the eigenspace captures inherent group features present in $\boldsymbol{X}$. This study offers a quantitative analysis of how group features from the input data evolve through hidden layers in randomly weighted neural networks.

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Xiangchao Li
Xiao Han 2 Qing Yang² 2
Abstract
Radhakrishnan et al., 2022; Shi al., 2022) have demon- strated the advantages of feature learning in neural networks. However, understanding the mechanism of DNNs remains challenging and the current theoretical framework is incom- plete, largely due to the heavy over- weerparaaeeteizztton and high non-convexity of modern neural networks (Allen-Zhh et al., 2019).
In this paper; we investigate deep feedforward neural networks with random weights. The input data matrix X is drawn from a Gaussian mix- ture model. We demonstrate that certain eigen- values of the conjugate kernel and neural tangent kernel may lie outside the support of their lim- iting spectral measures in the high-dimensional regime. The existence and asymptotic positions of such isolated eigenvalues are rigorously ana- lyzed. Furthermore, we provide precise char- acterization of the entrywise limit of the projec- tion matrix onto the eigenspace associated with these isolated eigenvalues. Our findings reveal that the eigenspace captures inherent group fea- tures present in X. This study offers a quantita- tive analysis of how group features from the input data evolve through hidden layers in randomly weighted neural networks.
Among the theoretical analyses, the Neural Tangent Kernel (NTK, Jacot et al., 2018) has become powerful tool for understanding neural networks. It is well established that NTK, which is associated with the nonlinear feature map (Bietti & Mairal, 2019), is able to describe the dynamics of heavily peerpaaraetterized neural networks under specific initialization (Jacot et al., 2018; Du et al., 2019). It also provides insights into the convergence and generalization properties of very wide DNNs (Xie et al., 2017). The NTK theory has also been extended to other network architectures, such as deep residual networks (Belfer et al., 2024) and deep attention networks (Hron et al., 2020).
The eigenvalues and eigenvectors of both the NTK and its closely related counterpart, the Conjugate Kernel (CK), are crucial in understanding the training behavior and general- ization performance of the underlying DNN (Fan & Wang, 2020; Yang & Salman, 2019; Yang, 2020; Engel et al., 2023). A recent line of work has analyzed the spectral properties of such kernel matrices. For instance, one may refer to Pen- nington & Worah (2017); Fan & Wang (2020); Wang et al. (2023); Wang & Zhu (2024); Belfer al. (2024); Wang et al. (2024); Murray et al. (2022). The spectral properties of these kernel matrices, particularly spiked or extreme eigenvalues, are often closely associated with certain properties of neural networks, including their memorization capacity (Nguyen et al., 2021), training and generalization performance (Liao et al., 2020), and their ability to capture the low-dimensional signal structure inherent in the learning problem (Benigni & Péché, 2022; Wang et al., 2024; Ba al., 2023). We are interested in investigating the limiting properties of the isolated eigenvalues and their corresponding eigenvectors of kereel matrices in the high -igh--imensional regime. Through this research, we aim to gain deeper insights into how fea- tures of the input data propagate through DNNs.
1. Introduction
In the last decade, deep neural networks (DNNs) have demonstrated dominating performance in various machine- learning tasks such as computer vision (Krizhevsky et al., 2012; He et al., 2016), natural language processing (Amodei et al., 2016; Brown, 2020: Graves et al., 2013; Torfi al., 2020) and game playing (Silver et al., 2016). One of the empirical findings is their ability adapt to the features present in the training data, which is considered fundamen- tal reason for their superior performance (see, for instance, (Bengio et al., 2013; Donahue et al., 2016; Moniri et al., 2023)). Nowadays, many thcoretical results have been estab- lished to understand the training and generalization of neural networks. In particular, significant research efforts (Abbe et al., 2022; Ba al., 2022; Bai & Lee, 2019; Damian et al., 2022; Dandi et al., 2024; Mousavi-Hossein et al., 2022;
Equal contribution 'School of Management, University of Sei- ence and Technology of China 'International Institute of Finance, School of Management, University of Science and Technology of China. Correspondence to: Qing Yang <yangq@ustc.edde edd.....ddcn......
Proceedings of the 4 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we study the CK and NTK matrices of deep fully-connected neural networks. Recent studies have

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
demonstrated that, as the network width approaches infin- ity; the empirical CK and NTK converge their expected values (Arora et al., 2019; Jacot et al., 2018). Therefore, we focus exclusively on the expected forms of these kernel matrices throughout this paper. We assume that the input data X E IRpx are independently drawn from a K K-class Gaussian mixture model (GMM) and are independent of the random weights. The analysis of class-structured data is common in the literature. For example, Papyan et al. (2020) demonstrated that when the training error reaches zero, the last-layer classifiers collapse to the class-means; Liao & Couillet (2018) conducted a spectral analysis of the Gram matrix associated with random feature mappings, while Couillet et al. (2018) analyzed the asymptotic per- formance of several classical classification methods under GMM assumptions; Ali al. (2022) investigated the spec- tral behavior of the CK in digh-dimensional regime; and Gu et al. (2024) derived the deterministic equivalents for both the CK and NTK. Based on the results established by Gu et al. (2024), we further show that a finite number of cigenvalues of these kernel matrices lic outside the support of the limiting measure. Our analysis reveals possible phase transition (Baik al., 2005) that depends on the choice of activation functions and the differences in covari- ance among different classes. We also precisely determine the positions of these isolated eigenvalues and eigenvector alignments.
random variable (or random vector) following distribution
v. The indicator function is represented by 5. For function f, we denote its i-th derivative by f(i Specifically, for = 1,2, we also use f' f() and f" = f Addition- ally, I1 = [a() denotes the supremum norm of f- The notation 0 may indicate a zero value, a zero vector or zero matrix in this paper, changing from line to line. We use and C' to denote positive constants, whose values may change from one line to the next.
2. Preliminaries
Let T1,..., In E R be n random vectors indepen- dently drawn from one of the K-class Gaussian mixtures CI,..., CK, that is
22 E C. = ~,000 p C.)
for some non-negative definite matrix Ca. For each a E {I,..., K}. class C. has cardinality na, satisfying M1 + ...nn
n. Write the input features in a matrix X [z] [=1.-.,, F. RRx, We define the fully-connected DNN with L hidden layers by
X 1 (W E Rdxxn for t= L..,,,,, Vi
with weight matrices We E Rdexde- (with convention do = p. Xo = X) and nonlinear activation functions 01 o1,-., oz applied entrywise. The CK of the ε-th layer is given by the Gram matrix
In this article, by demonstrating that the eigenvectors associ- ated with the isolated eigenvalues may contain information relevant to unsupervised classification (clustering), we es- tablish connection between the features of the input data and neural networks. The techniques employed in this paper are grounded in random matrix theory, and from a theoreti- cal perspective, our analysis falls within the framework of finite-rank deformation models in this field.
KCK_ KcK, = E[ x E RR
where the expectation is taken with respect to the random weights W1, W..... L. Following Bietti & Mairal (2019) and Jacot et al. (2018), the CK satisfies that
Notations: Throughout the paper. we use || to denote the Euclidean norm for vectors and the spectral norm for ma- trices. The spectrum of a matrix A is denoted by Spec(A). We denote an all-ones vector of dimension p by 1 and the identity matrix of size X p by pp For two sequences of random matrices A = {Anin21 and B = [B,1n21, we denote ~ if
K KCK,lly B[ae(u)ox((o]
(I)
with
[u, w/J ~ N (0. [Kc,e-ilas
[K (ic,e-
K KcK,e-lle
[K (Kck,-ilis)
while the NTK denoted by K KNTKE takes the form
1 LrD AA, BB)) + and u, (A,, - B.)uz + a.s. n
KNTK, = KOK.t + KNTK.6-1 o K'KK..
(2)
KNTK,0 = KCKO=KK = x
for all deterministic sequences [Dn]n21 and all determin- istic vectors u, "R",, i 1,2 2 with bounded norms. The symbols O(-) and o() stand for the standard big-O and little- notations. Moreover, if || A B| - O almost surely, we write A = B- + oa.a. (1). The Hadamard product between two matrices of the same size is denoted by o. The distance between two sets A, B C is denoted by dAst(A, B). Let &9 4 denote the law of ax + b, where x v is a
where
[KCK, ((d(())
We consider the figh-dimensiooal regime where n and are comparable and assume the following conditions on the input data, weights and activation functions.
Assumption 2.1. As n + xo, the following conditions hold:
2

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
The ratios & - co (0, 00), and as (0, 1) for each {1,... K).
+ co
E
We further assume Eloe(Te-1E)] O. Then as n + 00, we
a
have
Denoting C = -u=1 Re Ca. Ca = C. C and
| K Kok, (x)Il + 0 a.s.,
TO = =VVuC" /trCC//, we have -em TO E (0, 00).
(5)
K KOK, = -0,, x VAV + +0,0o
For E E (1....,), it holds that ||Call = (1) and
where ago0 = T8 -/ 2 0.
IrC,CC, = Op))
J/VP. E (nx((+)),
This assumption ensures the classification task is feasible and non-trivial (see Gu et al. (2024) for example).
A = aa + a4.3
2222
E <R(*+))((+1), (6)
Assumption 2.2. The random weights W1 E Rdxx,.... Wr. E Rd xd are independent. Moreover, the entries of We are i.d. and satisfy
ar,2
06,
for class label vectors J 1i.....,kkl E Baxk with j. (ō,, F The non-negative scalars ag,1, 02 06 sat- isfy the following recursions:
E W lij =0 0,
E] 1,
B[ < co.
011 Bl0(((=-1E))(on
This assumption on the random weights is relatively mild and holds for many common weights such as i.i.d. standard Gaussian weights.
0E22 = B[o 0z + Bio (T-- (T)=
4
Assumption 2.3. Let ~V 1). o1,,., ØL satisfy that
The activation functions
013 Blo 2 Eia (Te-
max kE(0.123344
[ Biog (5) <C, IsesL
(3)
with
0ee4 11 (tw. (0)
for some universal constant C.
The boundness of E|o,") (8)
(4) in (3) is needed to derive the deterministic equivalents for the CK and NTK. Moreover, as noted by Gu al. (2024), for non-differentiable functions, Assumption 2.3 holds if [ is bounded above by some polynomial function through the application of Gaussian integration by parts: (g) = (gG(E). Consequently, Assumption 2.3 accommodates commonly used activation functions such as ReLU, Sigmoid, and Tanh, provided that these activation functions are centered and normalized to ensure compliance with (3).
- 1, 00,2 - 000 0.
Lemma 2.5. (Asymptotic spectral equivalent for the NTK (Gu et al., 2024)). Suppose Assumptions 2.1-2.3 hold under the same notation and settings as in Lemma 2.4. Let to = O, $1, fr. 2 be a sequence non-negative numbers such that
:: = (loo(Te-1E))3 E (1....,T.) T.}
and let
x22 = 7? + +2 with No = To- Then as n -> 0o, we
have
Before presenting the preliminary theoretical results, is necessary to introduce several key expressions:
K KNTK,e(X) K KNIK, el (X)I + a.s.,
(8)
KNTKE - x VB. + Be
+ ( == {100 2|
E
1 JEz
E
R",
where $8,0 = k2 T8 2 and
(
t=
{
tr
E E
Vp
a=1
(4)
B = Br,2l + Br.3T
3r.2
E (
T +-ffuc.c.)
Be,
BI,2
E RRK*
a,b=1
The non-negative scalars Be,1, Br,2, Br,3 satisfy
The following two lemmas, proved by Gu et al. (2024), provide asymptotic equivalents for the CK and NTK.
3r,1 az,1 E(o 8p.
Lemma 2.4. (Asymptotic spectral equivalent for the CK (Gu et al.. 2024)). Suppose Assumptions 2.1-2.3 hold. Let To, Ti..., TL be sequence of non-negative numbers satisfy- ing the recursion:
31. a²2 + E(o 3
(9)
3e.3 = 01.3 + E (0'(((1- 38-1.3
+ E. (T(((T=18)) 0# 14.1,1181-117
Tt (Boo(te-11E) E (1,..,..)
with Bo, = 1, 3o, fo.3 = 0.
3

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
For an pxp p matrix M with real eigenvalues X 2 its empirical spectral distribution is defined as
and
K
G ~ G(=)~Q2(2) = - I + Σ
cama
(=)
F M (x) = p
#(i=1, P. A S x)
(11)
where (ma(=)] E CK a unique vector such that
and the probability measure induced by FM is denoted by PM- We define the Stieltjes transform of a probability measure 1/ as
3zSma 2 20, 323(2ma 2 0, col colma(2)) (32)
and
(z) / - =EC = ZZ: 3z> 0}.
coma(2)
1
ma(z)²
It admits a natural extension to the lower-half of the complex space by the fact that
11 ma(z) -- tr C. 2P
(12)
b=1
(=(=) = (z) ZEC = (s: S2 < 0).
trC (2()).
Besides, cym (2). cgmp((2)) are Stieltjes transforms
Prior presenting our results, we need the following lemma, which guarantees the existence of the limiting spectral dis- tribution of X X This lemma is established by Benaych- Georges & Couillet (2016).
of some Rt+ RR-compuctly supported probubility measures
Vi...., V.. The probabiliiy measure p defined by the Stieljtes transform
K
m2(2) = Co
Σ (a
(13)
a=1
is deterministic probability measure with compact support S - supp(va) such that
80
70
M - in distribution, dist(Spoc( X x (0)) +
(14)
60
(15)
50
almost surely. For x E such that dist S (0)) c for some positive constant c, (12) also holds.
40
According to Lemma 2.4 and Lemma 2.5, the (-th layer of kernel matrices can be approximated by a linear com- bination of xTx, X, I, and a low rank matrix. Based on above results and Lemma A.4, it follows immediately that | - cxc,
20
10
cx. || + 0 (|) [[ - F vTK.<) + 0). Therefore, the LSD of KcK,e(K (NTK,t) can be written as μCK, = 0g,1 @ f ① 0reo0 (UNTK,F = Be, @ u @ Br,o).
2
Figure 1. Spectrum of K KCK, under the parameter settings: n
To visualize the ESD of the CK, we present the empirical spectral distribution of K CK,3 in Figure 1, obtained using the same polynomial activation function across layers. As shown, this ESD does not exhibit any isolated eigenvalues. In the following section, will investigate scenarios in which eigenvalues detach from the bulk.
1200, 600, = Cg = 0.3, 02 = 0.4 and C. (1 + 2(a
1)/ I, for 2, 3. The activation functions for the three layers are 01 d2 = The weights W, E R x W₂ E Rdaxdd and W, E Rdaxda
Poly, where Poly(t) = 0.2t + t.
consist of i.d. standard normal entries, where dj = d2 2000, d3 1000.
Lemma 2.6. Suppose Assumption 2.1 holds. For 2 E C. we define the resolvents
3. Main results: eigen analysis of the CK and NTK
G(2) = x x - =Im)-1 G(x) = (Xx - -1
In this section, we follow the notation established in Lem- mas 2.4-2.5, and without loss of generality, assume that
Then as 0G, we have
x E C. for 1 1+
Σ n j=1
SiS nj.
G(2) ~ (z) = co diag (2)1 j
(10)
j=l

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
constant C, then K CK. t has Ki eigenvalues
ACK
jthkp-1
outside SCK, succ that
max 14 - (aE,1P + a20 - 0.8.
(16)
OSiSk
Moreover, if there exists a pt ESa satisfying H (p+)
0 and pt + p H4e where H H1(z) is defined in (28),
then there are k eigenvalues of K KCK,6
(a) Eigenvalues histogram of (b) Eigenvalues histogram of KCK,3
2 2.....2 X CCK
KCK,3
J+b
outside Sck,t such that
Figure 2. Spectra of Kek, KK,, and K KeK,3 (with the expectation es- timated from 500 realizations of weights).
max PACK (00,1 oc.oo) +0 a.s., (17) 0gig -1
where kj is the multiplicity of zero as an eigenvalue of (i+)
This assumption is also adopted in the simulations. We denote the supports of the limiting spectral measures of K KCK,E and K NTK. by Sck, and SNTK,,, respectively. According to Lemma 2.4, Lemma 2.5 and Lemma 2.6, we have
Lemma 2.5 shows that the asymptotic properties of the eigenvalues of the CK matrices also hold for the NTK, up to change of the associated coefficients at; to Se, for = 1, 3.4 Therefore, we can immediately obtain the following theorem, which describes the behaviors of the isolated eigenvalues of the NTK.
Sck,t 0 + ag,u SNTKE Br,18 + Be,v-
3.1. Isolated eigenvalues
Theorem 3.2. (Isolated eigenvalues of the NTK). De- fine h2(22) = 1 1+ Be.2 1 Exs ca and the set Be. p
This subsection provides precise characterization of the isolated eigenvalues of the kernel matrices. Based on the findings from the previous section, it suffices to focus on the isolated eigenvalues of K KcK, and K KTTK,E To illustrate this, we visualize the spectra of K K and K CK. in Figure 2, for the case where C. = (1 + 8(a 1)/ Ip with all other settings kept consistent with those in Figure
H = E= CI h2(22) 0}. Denote =2 = S U Hzg U {0} and
H2(=) = h I +
31.3 (T)
T(t) Be,1
Be.2 tt I(x)
Be,
Suppose Assumptions 2.1-2.3 hold, for p being solution detH, ( 0 with mulitiplicity kp and dist S2.) > C for some positive constant C, we conclude that there exists ko eigenvalues of K KNTK,
We now present the results for the eigenvalues of the CK and NTK that detach from the bulk of their spectrum, fo- cusing on the non-trivial case where 0ke,1 (3r,1) is bounded away from zero. The following theorem establishes the asymptotic behavior of the isolated eigenvalues of K KcK,e. Theorem 3.1. (Isolated eigenvalues of the CK), Define the function h/ (2) = -1+ a1. EK C2)a())) and the at.1
XNTK 2...
NTK s+h,-1
outside SNTKE e such that
max OSISK,
INNK
(Be,p Be,00l + a.S.,
set H¹x 1- = {= E C2h)(z) = 0}. Denote
where the scalars 3r,0 and Be are defined in (9) and ko
is the multiplicity of zero as an eigenvalue of Hz(p). We
S1, = SU UU
denote H2(z) by replacing the a' o's in H1(2) with B's. For
p+ S being solution to det H2(2) =0 and pt - p e
and
H2 there are kj eigenvalues of K KTTK,
02, 023 11 [[(3II
022,2 tt T())
H (2)
XNTK 2 A NTK jtb
ag,
a41
where T is defined in (4) and
such that
I(2) diag(co
max OSiCk,
IX
NTK (Be, + Be,o)l + a.s.,
Under Assumptions 2.1-2.3, if H (p) has a zero eigenvalue with multiplicity kp and dist St. > C for some positive
where kt is the multiplicity of zero as an eigenvalue of Hz(++).
5

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
noise. Inspired by these observations, interesting ques- tion arises:
How can we understand the structure of the eigenvector (or eigenspace) associated with an isolated eigenvalue in high-dimensional regime?
-0.25
Rigenvector associated with the isolated eigenvalue of
We use Span(J) to denote the space spanned by Ja: a 1,.., K. Let PJ be the projection onto Span(J). The eigenvector associated with the ith isolated eigenvalue X is denoted by ú, This eigenvector can be decomposed into a signal part and a random part:
K CK,3
0.000
-0.025
K
Ja Ma
-0 050
ú; = Pyû; + (I, - 10--PJ)
PJ)
Wio
+ o
VaVa
000
800
1000
1200
Eigenvector associated with the isolated eigenvalue
(18)
K CXK,3-
where V. is random unit vector orthogonal to the ja. and its entries are identically distributed. The scalar Wia
0.10
à! + measures the cosine between é; and La and o
Vha We denote
da quantifies the extent of fluctuations in wia tua (wia 3
0.00
-0.10
If the isolated eigenvalue XCK is simple, it follows from Wia à! in that yma
200
400
300
800
(c) Eigenvector associated with the first eigenvalue of X X.
- [ [JJūūü ū,ū Jlao Tha
1
In ū,ū
Jo
Figure Visualization of the eigenvectors corresponding to the first eigenvalues of matrices KCK,,, K and X X under the setting of Figure (a), wi = |-06443, 0.005, 0.693) while in (c), W1 00.008 0.036, 0.008).
Wia
(19)
The
Thus, if p identified in Theorem 3.1 satisfies that H (p) has simple (multiplicity one) zero eigenvalue, we can explicitly evaluate wa Moreover, we say an eigenvector ù; is non-informative for clustering (i.e., does not contain any information about J) if max, ú, v that ù; becomes asymptotically orthogonal to Span(J). It is important to note that the eigenvector is not unique, and essentially, our analysis concerns the asymptotic behavior of the matrix I J ü,ūū _Jūūü J, as described in (19).
1_ = Oa.e. (1.,(1), meaning
Theorem 3.1 and Theorem 3.2 establish the conditions for the occurrence of isolated eigenvalues and provide their asymptotic positions.
3.2. Behaviors of the eigenvectors
Let Ay...., A be a group of isolated eigenvalues of ck that converge to the same limit 0z,1 og,,, where p is identified in Theorem 3.1. As per Lemma 2.4, the eigenspace
The preceding subsection studied the eigenvalues of the kernel matrices. We now turn our attention to the asymptotic properties of the eigenvectors associated with the isolated eigenvalues. Specifically, we provide a detailed analysis of the CK matrices, as the NTK matrices exhibit similar pattern.
Span E SSn-1 K KCKKú, = A4ti-1û;} (20)
is asymptotically equivalent to the eigenspace associated with eigenvalues of X X + t,, V V,VV that have a de- terministic limit p. This relationship also illustrated in Figure 3. Definition 3.3 below, we present precise math- ematical criterion for determining when an eigenspace is considered informative.
We begin with toy empirical example. Under the same setting as Figure 2, Figures 3(a)-(b) display the eigenvec- tors associated with the isolated eigenvalues of K KCK.3 and K KCK,3, respectively. Figure 3(c) shows the eigenvector corresponding to the leading eigenvalue of XX.
Definition 3.3. (Informative eigenspace). We say the eigenspace defined in (20) informative if there is a non zero matrix (JJ) depending on J such that
From these figures, we can reasonably infer that the eigen- vectors corresponding to the isolated cigenvalues (Figures 3(a)-(b)) are more informative, as they likely capture rele- vant information for clustering. In contrast, the cigenvector depicted Figure (c) can be reasonably regarded as pure
1 J A(J)I Oa.s. (1). || p
6

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Otherwise if AJJ 0, then the eigenspace is non- informative.
úg and Span(J). Furthermore, if (2(()) 0. then ||w approaches 0 by letting u, (51/ v/......... ./..../////// indi- cating that it can always be considered as non-informative eigenvector.
Denote by П. E ü,ü the projection onto the eigenspace defined in (20). Then we can write - as
The theoretical results presented above remain valid for the NTK, summarized in Theorem 3.7 and Corollary 3.8.
1 f 1 1x + + VA - -II,)= j 2mi Jon, + 0a.x. (1);
Theorem 3.7. Suppose the assumptions in Lemma 2.5 hold. Let X ..... j+hk be group of isolated eigenvalues of K KNTK, that converge to the same limit 3t.1P + 3r,0, and let II denote the projection onto the eigenspace spanned by the eigenvectors associated with these eigenvalues. Here, P is defined in Theorem 3.2. Suppose Assumptions 2.1-2.3 hold. For (p) we denote the left and right eigenvectors corresponding to O as (Vi,z)s and (Vr.)s respectively, bounded away from
(21) where Yo is an open disc such that p belongs to the interior of Tp and Ohp is positively oriented closed circle. Therefore, suffices to investigate the asymptotic properties of the right hand side (RHS) of (21). We establish the following Theorem 3.4, which provides the asymptotic behavior of eigenspace II, characterized by u' I, where and v non-random unit vectors.
where = 1....,kpp kp. If dist(p, O, then we have
Theorem 3.4. Suppose the assumptions in Lemma 2.4 hold, Let XÇ* ........K____ be group of isolated eigenvalues of K KCK, converging to the same limit 0ee, + og.o- where is defined in Theorem 3.1. For (p),), we denote the left and right eigenvectors corresponding to 0 as (Utp)) and r,o)) respectively, where i ranges from I to ke- If dist(p,Sn S1,t) is bounded away from 0, then for any non- random unit vectors u, R", we have
u II = -----hhh (2) (
+ +Oa.s. (1),
where
(V (V.D)((VLn)) LoD!
F(=)=L L (VLP)! ( (z) (Vro)
Corollary 3.8. Under the same conditions as those in The- orem 3.7, we have
u T HT
+ (1)
1 L P
1 - (OO)) ol.s...) (1).
where
P
Remark 3.9. For the NTK, analogous to Remark 3.6, p+ satisfies det H2(+) 0 and p+ + p E H2g then its associated eigenspace is non-informative.
hp F(p)= = - 1=1
U (UDD)
H (Urp)
Remark 3.10. We note that Wang et al. (2024) investi- gated how spiked cigenstructures in the input data propagate through the hidden layers of neural network by analyz- ing the spectrum of the CK, under the assumptions that the input data contains spiked eigenvalues and that the same activation function is applied at every layer. In contrast, our work examines both the CK and NTK. Our analysis demonstrates that isolated eigenvalues in the kernel matri- ces may arise from the underlying group structure of the input features, with the corresponding eigenvectors carrying useful information, even in the absence of spiked struc- ture in the original data as required in Wang et al. (2024). Additionally, our framework allows for different activation functions across layers and investigates their effects on the spectral properties of the kernel matrices.
By noticing that 2. ately derive the following corollary.
(2) r()), may immedi-
Corollary 3.5. Under the same conditions those in The- orem 3.4, we have
1 JJ J - t0...)
(22)
p
This result identifies the conditions under which the eigenspace corresponding to the isolated cigenvalues of the CK matrix are informative. Discussions on non-informative eigenspace can be found in Remark 3.6 below.
Remark 3.6. For p+ satisfying det (p+) = 0 and p+ P HP it follows that for any non-random unit vectors u, U E R"
3.3. Additional simulations and real data analysis
u II p+ v = (1).
(23)
In this subsection, we present additional simulations on both synthetic GMM and real data to further support our theoretical findings.
When Ao+t in Equation (20) has multiplicity one, ||w,|| can be regarded as a measure of the alignment between
7

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
of 1600 randomly selected images from each of the digit classes 1 and 7 in the MNIST dataset, with the class-specific mean subtracted from each group. Figure 5(a) displays the spectrum of the CK, obtained using three-layer neu- ral network. The eigenvectors associated with the isolated eigenvalues are shown in Figure 5(b). The spectrum reveals four isolated eigenvalues, with the eigenvector correspond- ing to the largest eigenvalue being informative, while the other three appear non-informative. These observations in line with our theoretical findings.
8
10
10
00
05
1.
1.
20
25
31
000
650
036
035
600
207
(a) Poly
(b) ReLU
0.56
10
250
500
700
1000
1250
1500
1750
2000
(c) Eigenvector associated with the largest eigen- value of K KeK, when all activation functions of are Poly.
005
0.00
0.81
885
0.0
(a) Spectrum K CK.3 estimated from 500 realizations
252
800
702
1000
1250
1500
1700
2000
-8.00
(d) Figenvector associated with the isolated eigen- value of K KCK, when all activation functions are ReLU.
10: 00:0
-8.00
160
3.04
Figure Eigenvalue histograms ((a)-bb) and eigenvectors ((c)- (d)) corresponding to the isolated (largest) eigenvalues of KCK.3 obtained with Poly and ReLU activation functions. Parameters
-8.55
settings: 2000, p 3600, c1 cs 0.2, and C. = (1 + 8(a 1)/ I for a = 3,4. The width di, da, da are identical those in Figure 1.
0.3
-0.
1009
2300
3008
(b) Eigenvectors associated with the isolated eigenvalues.
Figure 5. Eigenvalue histograms (a) and eigenvectors (b) corre- sponding to the isolated eigenvalue of CK.3- Activations=(Sin. ReIU//10, Sin]. The weights W1 Rdix>, W₂
Figures 4(a)-(b) visualize the eigenvalues of K KCK, under different activation function settings. In Figure 4(a), all acti- vation functions are set to Poly, consistent with those used in Figure 2, whereas Figure 4(b), all activation functions are set to ReLU. The ReLU activation functions have been normalized and centered satisfy the conditions specified in Assumption 2.3. Unlike the results shown in Figure 2(a), no isolated eigenvalues are observed when all activation functions are Poly. In contrast, when all activation func- tions are ReLU, an isolated eigenvalue emerges. Figures 4(c)-(d) show the eigenvectors corresponding to the largest eigenvalue in Figure 4(a) and the isolated eigenvalue in Figure 4(b), respectively. The former appears to be non- informative, whereas the latter is notably informative.
ddxxd: W E dd consist of i.d.. standard normal entries, where d1 2000, d2 ds 1000.
4. Theoretical insights for machine learning applications
It is known that the NTK theory leads to concrete conver- gence and generalization results (Bai & Lee, 2019) of neural networks. Let y be the true label vector of the training data and (t) be the prediction at time t. In an ultra-wide neural network with training loss 21yy- (t)11, the time evolution of the residual y ((() during early training is approx- imately described by the following ordinary differential equation (Jacot et al., 2018; Du et al., 2018; 2019):
These observations underscore the critical role of activation function selection in determining the spectral properties of the kernel matrix.
Next, we turn to real data analysis. The input data consists
dt (t) = (ark(y-g(())
(24)
8

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Based our theoretical results from Theorem 3.7-Remark 3.9, the first-order limits of entries in the isolated eigenvec- tors of NTK may or may not contain group features (i.e. informative or non-infoon -on-informative). When the eigenspace as- sociated with the largest isolated eigenvalue contains group features, DNNs tend to prioritize learning from this sub- space, Conversely, when the eigenspace lacks group fea- tures, DNNs instead prioritize learning irrelevant informa- tion, diverting attention away from effective group features. Moreover, Theorem 3e4-Remark 3.6 indicate similar phe- nomenon of the eigenspace during the DNN's initialization step.
References
Abbe, E., Adsera, E. B., and Misiakiewicz, T. The merged- staircase property: necessary and nearly sufficient con- dition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theoor, pp. 4782-4887. PMLR, 2022.
Ali, H. T., Liao, Z., and Couillet, R. Random matrices in service of ml footprint: ternary random features with no performance loss. In International Conference on Learning Representations, 2022.
Allen-Zhu, Z., Li, Y., and Song. Z. A convergence theory for deep learning via weer-praaettrizzaion.. In International conference on machine learning, pp. 242-252. PMLR, 2019.
5. Conclusion
This paper investigates the spectral properties of the conju- gate kernel and neural tangent kernel, revealing the evolu- tion of inherent group features through hidden layers. From Theorem 3.1 and Theorem 3.2, it can be seen that the oc- currence of the isolated eigenvalues depends on two key factors: (1) differences in the covariance matrices between different classes and (2) the choice of activation functions. The former shapes the vector t and the matrix T, while the latter determines the coefficients a's (3's). When iso- lated cigenvalues are present, these factors also determine the asymptotic behaviors of the corresponding eigenvec- tors. This conclusion is further supported by comparing the simulation results presented in Figures 1-4.
Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q.. Chen, G., et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182. PMLR, 2016.
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. On exact computation with an infinitely wide neural net. Advances in neural information process- ing systems, 32, 2019.
Finally, we discuss several potential directions for future work. Since current results rely on the assumption of GMM-diitributed inputs, it is important explore whether similar phenomena occur for more general distributions, particularly those with heavy tails. Additionally, extending our analysis to other neural network architectures represents another promising topic for future research.
Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional asymptotics of feature learn- ing: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35: 37932-37946, 2022.
Ba. J., Erdogdu, M. A.. Suzuki, T., Wang, Z., and Wu, D. Learning in the presence of low-dimensional structure: a spiked random matrix perspective. Advances in Neural Information Processing Systems, 36:17420-17449 2023,
Acknowledgements
Xiangchao Li and Xiao Han are co-first authors. Qing Yang is the corresponding author. This work was supported by NSF of China (No.1237) (37), the CAS Talent Introduc- tion Program (Category B), National Key R&D Program of China-2022YFa 1008000, NSF of China (No. 12101585) and the Young Elite Scientist Sponsorship Program by Cast (No. N...Y15520220125).
Bai, and Lee, J. D. Beyond linearization: On quadratic and higher-order approximation of wide neural networks. arXiv preprint arXiv: 191000009000000 2019.
Bai, Z. and Silverstein, J. W. Spectral analysis of large di- mensional random matrices, volume 20. Springer, 2010.
Impact Statement
Bai, Z. D. and Silverstein, J. W. CLT for linear spectral statistics of large-dimensional sample covariance matri- ces. The Annals of Probability, 32(1A):553 605, 2004.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Baik, J., Arous, G. B., and Péché, S. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643 - 1697, 2005.
9

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Belfer, Y., Geifman, A., Galun, M., and Basri, R. Spectral analysis of the neural tangent kernel for deep residual networks. Journal of Machine Learning Research, 25 (184):1-49, 2024.
Engel, A., Wang, Z.., Frank, S., Dumitriu, I., Choudhury, S., Sarwate, A., and Chiang, T. Faithful and efficient explanations for neural networks via neural tangent ker- nel surrogate models. arXiv preprint arXiv: 14585, 2023.
Sanaych-Georger F. and Couillet, R. Spectral analysis of the gram matrix of mixture models. ESAIM: Probability and Statistics, 20:217-237, 2016.
Fan, Z. and Wang. Z. Spectra of the conjugate kernel and neural tangent kernel for linear-wine width neural networks. Advances in neural information processing systems, 33: 7710-7721, 2020.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
Graves, A., Mohamed, A.r,, and Hinton, G. Speech recog- nition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649. Ieee, 2013.
Benigni, L. and Péché, S. Largest eigenvalues of the con- jugate kernel of single-layered neural networks. arXiv preprint arXiv:2201. 044553, 2022.
Gu, L., Du, Y., Zhang, Y., Xie, D., Pu, S., Qiu, R. C., and Liao, Z. "lossless" compression of deep neural networks: A high-dimensional neural tangent kernel ap- proach. arXiv preprint arXiv:2403.00258, 2024.
Bietti, A. and Mairal, J. On the inductive bias of neural tan- gent kernels. Advances in Neural Information Processing Systems, 32, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Brown, T. B. Language models are few-shot learners. arXiv preprint 11x::2005..41655 2020.
Couillet, R. and Benaych-Georges, F. Kernel spectral clus- tering of large dimensional data, Electronic Journal of Statistics, 10(1):1393 1454, 2016.
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, pp. 4376- 4386. PMLR, 2020.
Couillet, R., Liao, Z., and Mai, X. Classification asymp- totics in the random matrix regime. In 2018 26th Fu- ropean Signal Processing Conference (EUSIPCO), pp. 1875-1879. IEEE, 2018
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Damian, A., Lee, J., and Soltanolkotabi, M. Neural net- works can learn representations with gradient descent. In Conference Learning Theory, pp. 2022.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
PMLR,
Dandi, Y., Krzakala, F., Loureiro, B., Pesce, I., and Stephan, L. How two-layer neural networks learn, one (giant) step at a time. Journal of Machine Learning Research, 25 (349):1-65, 2024.
Liao, Z. and Couillet, R. On the spectrum of random features maps of high dimensional data. In International Confer- ence Machine Learning, pp. 3063-3071, PMLR, 2018.
Liao, Z., Couillet, R., and Mahoney, M. W. A random matrix analysis of random fourier features: beyond the gaussian kernel, precise phase transition, and the corresponding double descent. Advances in Neural Information Process- ing Systems, 33:13939-13955 2020.
Donahue, J., Krähenbühl, P., and Darrell, T. Adversarial feature learning, arXiv preprint arXiv:1605.091 2016.
Du, S., Lee, J., Li, H., Wang, L.. and Zhai, X. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675- 1685. PMLR, 2019.
Moniri, B., Lee, D., Hassani, H., and Dobriban, E. theory of non-linear feature learning with one gradi- ent step in two-layer neural networks. arXiv preprint arXiv:2310.07891, 2023.
Du, S. S., Zhai, X., Poczos, B., and Singh, A. Gradient descent provably optimizes over-parann werrpparamterrized neural networks. In International Conference on Learning Rep- resentations, 2018
Mousavi-Hosseinn A., Park, S., Girotti, M., Mitliagkas, I., and Erdogdu, M. A. Neural networks efficiently learn
10

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
low-dimensional representations with sgd. arXiv preprint arXiv:2209. /4863, 2022.
Xie, B., Liang, Y., and Song, L. Diverse neural network learns true target functions. In Artificial Intelligence and Statistics, pp. 1216-1224. PMLR, 2017.
Murray, M., Jin, H., Bowman, B., and Montufar, G. Char- acterizing the spectrum of the ntk via a power series expansion. arXiv preprint arXiv:2211.0784. 2022.
Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. 14548, 2020.
Nguyen, Q., Mondelli, M., and Montufar, G. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference Machine Learning, pp. 8119-8129 -129.. PMLR, 2021.
Yang, G. and Salman, H. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv: 10599, 2019.
Papyan, V., Han. X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of Sciences, 117(40):24652-22)
2020.
Pennington, and Worah, Nonlinear random matrix theory for deep learning. Advances in neural information processing systems, 30, 2017.
Radhakrishnan, A., Beaglehole, D., Pandit, P., and Belkin, M. Mechanism of feature learning in deep fully con- nected networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212.13881, 2022.
Shi, Z.,
Wei, J., and Liang, Y. A theoretical analysis feature learning in neural networks: Emergence from inputs and advantage over fixed features. arXiv preprint arXiv:2206.01717, 2022.
Silver, D., Huang,
A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panncershelvam, V., Lanetot, M., al. Mastering the game of go with deep neural networks and tree search. 2016.
nature, (29(7557):4844
Torfi, A., Shirvani, R. A., Keneshloo, Y., Tavaf, N., and Fox, E. A. Natural language processing advancements by deep learning: A survey, arXiv preprint arXiv:2003.01200, 2020.
Wang, 7. and Zhu, Y. Deformed semicircle law and centration of nonlinear random matrices for ultra-wide neural networks. The Annals of Applied Probability, 34 (2):1896-1947, 2024.
Wang, Z., Engel, A., Sarwate, A. D., Dumitriu, I., and Chiang, Spectral evolution and invariance in linear- width neural networks. Advances in neural information processing systems, 36:20695-2001 20728,, 2023.
Wang, 7.,, Wu, D., and Fan, 7. Nonlinear spiked covariance matrices and signal propagation in deep neural networks. In Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pp.
PMLR, 2024.

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
A. Appendix
In the later proof, the following lemmas are needed.
Lemma A.I. (Couillet & Senaych-Georgee 2016) Suppose Assumption 2.1 holds and let 2 E C be a point that is at least distance c > away from S. We have
G(=)ab = +ama())tn (0000(++0))))) (
- Enp G(=)abl Opp=))
a=1
Lemma A.2. Under Assumption 2, for any deterministic {uz (i with finite Euclidean norm, have
u G(2))) 00 a.s. for such that dist(S U [0), 2) >c
where c is a positive constant.
We postpone the proof of Lemma A.2 to the end of this section.
A.1. Proof of Theorem 3.1 and Theorem 3.2
Noting that the asymptotic spectral equivalents for the CK and NTK are identical up to change of the associated coefficients a's to f's, we only provide the proof for Theorem 3.1 here and the proof for Theorem 3.2 is exactly the same.
We devote the majority of this section to finding the isolated eigenvalues of X x xa A,V which, up to multiplication
by 1 and an addition of ag,o, constitute the eigenvalues of CK,e. In order to find the isolated eigenvalues of X X +
11, VAV" one only need to solve the determinant equation
det + az VAVT - -2In)= =0
(25)
for away from Si,e- An application of Lemma A.8 yields that (25) shares the same solutions
d -1 A + +I41)) =
(26)
According to Lemma 2.6 and the fact that
(
co (2)) = (i)g()am(())) = I(z),
p
p
we have
_ -
1
on. (0) (a Oa.s. Oa...(1).
p
P
Lemma A.1 gives
K Σ
+
G(2)
ma (1).
o=1
From Lemma A.2, it can be seen that the cross term
1 vp
G(()) vanishes as n + co. Therefore, we can write
LJGG(
G
T(2)
VTG())V=
[ 0 1 D cama
+ On.a. (1)
w t G(=)J
GG))
as well as
I + t A A.V G H1(2) + on. (1),
(27)
where
Hn H12
Ik a lag + ag T(T(2)
[2((2) 1}t (2)2)
H1(z)=
H
H 22
=
a T(2)
(28)
12

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Therefore, it suffices to find the solution of
H (z) =
(29)
on the real line by (27) and Lemma A.4.
first consider the case where (z) is bounded away from Using the Schur complement formula, we have
d 222det (H H 22)
(30)
11- det H 11 H - H 212 H2 112 det
Therefore, if satisfies that Pest(p,S1.t) > C for some constant C and makes H of the cigenvalue 0 denoted by kp, then according to (5), there exists at least one eigenvalue of X X 0 VAVV that converges to p almost surely.
H (p) a singular matrix, with the multiplicity
If (=) + O, studying (30) is not suitable because H = (2)=- tends to infinity. We instead consider the matrix (2) directly. If (29) has solution p+ such that p+ - E Hg, then p is a deterministic limit of certain eigenvalues of KcK,
Next, we investigate the number of eigenvalues of K CK, that converge to the limits identified above. By Lemma 2.6, we see that the eigenvalues of X x X asymptotically do not escape U {0}. Therefore, one may find a compact interval = [ai, bi] C (a, b) such that, for large enough n, X has no eigenvalue in I with probability one and
{ol det (p) = 0} n (a1, b1} = 0.
(31)
We Dj to denote the open disc centered at (a1 b1)/2 and of diameter (bı a1). Let
L1 = #( zeros of det T X + ag. + VAVT nI in DiJ. L2 = #(p1 D2 and H = =0}.
Note that the functions (el(X X + a V.V aI),, de) and del del.H, (2) are analytic Dj and non-vanishing on C,R By Lemma A.5 and the argument principle, we obtain
0_del(X x + az A 2I,)
L1 2mi JODg
V.V'V - =I.) O, + V
del(X X + + V
2mi f
a,dctG(z)
PODE
dot(I,, + 0,1 VAV
2ni JaDa detG((z
+ 0, +
A
1
0,detH L2
2ii Jang
det(1 1 + A.V G(=)V)
2Ti Jang det H (z)
almost surely, where aD₃ is isolated eigenvalue of X x + a VAV which has deterministic limit p. the same as that of H (p).. It is straightforward to see that if p+ defined in Theorem 3.1 satisfies h (p+) + O, then the multiplicity of p+ denoted by kt is equal to the multiplicity of as an eigenvalue of H1 (p+). the case that h (p) away from 0, we have that H, (p) has 0 eigenvalue of multiplicity kp by (30). This concludes the proof of Theorem 3.1.
positively oriented contour. Since both L₁ and L2 integers, the multiplicity of an
an eigenvalue
A.2. Proof of Theorem 3.4 and Theorem 3.7
For any closed interval C R defined above (31), we have
1 o u G]
2mi JaDs
almost surely. Lemma A.2 implies that, for any deterministic u and with finite Euclidean norm,
u. G(=+V u=Q,(=)V (1,, of G(=VV = oTQ,(==VV + Oa.s (1).
13

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Then by using Woodbury's identity, we obtain
1 of
u -'uu
u " + + - - vdz
2 doDg
1
f
u u'G(ooodz
2 JoD:
(32)
1
{
01, |u
A Gz))ooz
2Ti Jang
% u
Q (z)udz + Oa.s. (1):
2mi
JaD2
where I7 is the projection onto the eigenspace spanned by the eigenvectors corresponding the eigenvalues of K c that lie inside I. Thus what remains to find the limit of (32) is to obtain the deterministic equivalent of H (z) Since the zeros of h2(2) are away from Di, we obtain
H H (2)
H2) 12 12)
H
H
H
All notations presented here are defined in (28). Therefore, we can write
H (2)A, = [ Ax) H Ad
(2)A
H Az)
where
(=) 01.3 Hi (x)T Hr)
[H (x) a (2)t,
H (x)AA1) 00,2003
( H (2)T+ ar,t
1
oZ t [(2 H (2)tt' (
+
01,1
h1(2)
h1(2) 0z
(x) = 00,
HT
12(2) h1(2)
01,
Observing that u G((=VV - (=)J 0] + o..s. (1) (v G(=VV =
L L (2) 0] + on.o. (1) ) it becomes
evident that only H 1 (=) Arlu is needed. Then we deduce that
u - (z)
(z) (1).
We denote the spectral dccomposition H (z) as
H -2)==U,,AA
For E I such that (p) singular and kp denotes the multiplicity of 0 as an eigenvalue of (p), we denote the left and right eigenvectors corresponding to 0 as (Ur,)); and (U (Ur))): respectively, where ranges from to kp. For matrix (z) (M) denote Res(M) (Res(M (2)) Then have
lim H lim (z P)U U z-p
(U (UND)(UU
lim (z z<p
kp
(U (UI.)
lim
H.
14

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
where the last equation is obtained by applying the L. L'Hôpital's rule. Since
( (ULp) H. (p) and Hi(p)(U,,p))) =
obtain
(+)
= (m)m))
(3)0))
(..
It is easy to see that
u [1(2) H (x)
ae, (2)
01. tt {T)((200
p
a1,1
a11
= u (=))
(r)
0t5 tT ((2)r= (
- (1(3)
(x) (2)
( (1)
u
T
Q
Now one can conclude by (32) that
1 Σ
u
Res
(1)
p ZET
1
( (UDD)(UUD) (0 (2) ( (o.s.(1)
Thus the proof Theorem 3.4 is completed. Theorem 3.7 can be proved by the same lines as that of Theorem 3.4.
A.3. Proof of Equation (23)
Now, we consider the case that p+ + p HP with being at a distance from St and detH = 0. Suppose ||t|| is bounded away from 0, we note that lim H (2) 0, tt (22) and the RHS is rank-1 matrix. Hence there are K 1 eigenvalues of H (p+) converging to 0 almost surely. It is natural to see that the derivative H. (z)l is well defined and not close to 0. Based on the above facts and applying the L' L'Hôpital's rule again, we have
lim (z - (z) H (2)
lim (z (2) UU 1
K-1
(U (ULP):(UULP)
lim (2) lim
Σ
(z p+)
2-1n1 2-1p
(U H (U),(()
d=1
K-1
(U (Unps):(UL.p))
lim (2) 2-00
(UI,))) 10 H (3))==pp (Ur.p:)
0.
If ||t|| + 0, then p+ becomes a removable singularity for (2) (z), thereby ensuring that the residue remains zero. The conclusion can be obtained by repeating the proof of Theorem 3.4.
A.4. Proof of Lemma A.2
We divide the proof into three steps:
Step Fix v) > 0. For any 2 E C with 32 2 n, the expectation Eu G(() o(1).
15

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Step Convergence of u u (z))b Eu RuTGa (z))bb almost surely.
Step 3: For any such that di U {0), 2) > for some constant c, we also have u (2) - a.s.
We assume 3z 2 1 > 0 Rz = E > 0. For the case 32 S -1 < 0, one only needs to replace by In] in the following
proof. Define
X, = x - z,e,, G,(2) = (x) - =I,)
where ej E I2" denotes the vector with the kth element being and otherwise being Note that &/- = -Ej= Uyey. We can write
n G'G(=) E a G(:))ee
Decompose the term Eu (z))bee as = = -[[G(=))G((s)](se)) u (TG u
Since (z) is independent v/j, we have (2)(,) = (2)BB,, = 0. Let
X [1, 2j - 2 B5+1 Enl. G2(2) = (x] X, - 21 x-
EtrC a G,, -1
[rC اLا)
8y = -2 - -xx by=
where g(t) E (1,..., K} denotes the class to which vy belongs. All of these quantities are bounded absolute value by ? Then from Lemma A.6, we get for m 2 2,
(trCat))
EirC 90) (x,(()))
m
E
b
2. G,,(,),,,,,,, E 2J -
17
p
p
(33)
o ((m18) 7)
and
pm B(xJ x. trC (0 m 2.
(34)
By the Schur complement formula, obtain
v (G(2) G,(z)) - L6, - = Ju, 3 G, (z) where uj- = [[M,,.... 1 Mj+1 J. It follows from (33) and (34) that
1 = |Bbh (3, ) S by 1Bby b,)l
1
ا|u 1b 2 by 2 = 10,10
172p/
Thus we have
n2
Babg Σ
Baby BubyBruzl
Σ Ma 0
1/2p1/2 0.
(35)
ppp
We write x, = 1 1/2
vp 00)
where 2j = { Ve is random vector with i.id. standard Gaussian entries.
Denote
y - u]= [o) o()) (v/ 1- It is easy that ||,, S (2)x 2 O(n For 2 1, by
Lemma A.7, it is true that
2m
2m
u G,X
2m
E
y
2j
EE | 2,
[,
EE
yr 2
yj
pm
pm
1
m
P
2m
E
Σ
+
CE
(36)
C
los 12
1
<
CTE
ii=1
mm
pm
mm
16

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
where we the inequality
1 2m 1
Σ 1 -111,11
(m))
Write B,b,,]
E1y [ [ b,) X3 my It can be seen from (33), (34) and (36) that
x m3 Bl 2 (3) by G my s Bly5118 - bylluy G)()) byllu]G,(x)) 03
(1/ E1 - b Elyy an 12
El 22 0 npp
17pp3/2
This, in conjunction with (35), establishes Step
According the Borel-Cantelly lemma, it suffices prove that for any O, the probability P( ( Gz) > e) 1p-1199 to establish Step 2, where is positive constant. Let IEg denote expectation and EE denote conditional expectation with respect to the o-field generated by æi...., 2j. Using the Schur complement formula again, we write
u G(z)/b - Eu G(=)b Σ (B, BB-1)u G(2)gb G(z)vb
Σ
[ Ey-1)(m G())b} -u--uu (=))(=)((b - eybby)
/=1
(E, 1,1p,///0|3000,3y00000000,,,,,,3
jml
Σ [E3 E) + 7/2 + 743 + 734];
j=1
where 2b = [bh, ..., 1/ 16f41 v/nl
According to Lemma A.7, we have
E
Σ (E,, 13-1)791
S cE
Σ E B
1
+
E
Ebu 4
+
u
Eg/
17
1/
j=1
0
1ppp3
Similarly, we obtain
E
Σ(E, E
12/2
<
E x, G,
c
1pa
5=1
Denote r. = (z) is apparent that
According to Lemma A.1 and the identity
G5(2) G3 -3((3 2 2
G
17

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
get for 2
X,X_////00,,,//////0XXXxXXX,,,, "M ( (,B,)c((x33x,,,()) [ G,
E 1 am
S cB
$
(,0 " -
<CE
G,(2) (2)
+ B
2 3
0
where we use Lemma A.I and the observations that
(2(2) (2)
>x Ca (ma (z) ma (2)
Eab T
v
IrC + o(1),
2 -
plz - 2)
ma (2(2) m²(
C0
A 2/2 dv,(2) (2) 2
2
dist (s,t>(S,,))
Following a similar argument as in (36), we have for m 2 1,
E 2m 21 pm EE xr 2m
X
S
cE
rr
m
||,,
12m
0
1
p
pmd
Therefore, we have Σ) (R. - EEj-1)7s2 - 0 a.s. by showing that
E
Σ j=1
LE 1 (p
0
p
Similarly, for m 2 1, can be obtained that
B/yya| 2m
Bl
<
am v Bnigh
1
E [ly | im
0
(
(37)
and
E 1/y 2m B,Bju, x x,, G,,b, pmm 1 1m2m v E xz,
4m Ir 4m 0
(38)
1mm
One can conclude by Lemma A.7 that
2
E E 0
Σ E 1333
S Σ
E 733sl
+
Σ
122
and
Σ E, Eg-1)754
Se Σ
2
+
E El734
=0
1, =0(32pp00
1
/=1
Combining the results above, one can conclude that
n
P 2 E, 11-1) h01 752 + 733 + [31]
>
Σ 1=1
E - Ex-1) [751 + 732 + 733 + 731]
1=1
C
Σ j=1
EE R. )73k
A=1
1
2?
18

---

Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Hence, the proof of Step 2 is completed. Moreover, for any z with Rs
can obtain the result by replacing v by [Rt in the proof. Although we assume is fixed, the same conclusion also holds for r 7/2 converging
sufficiently slowly such 7/.. = (log n) r-)- when Rz is bounded away from 8 U {0}. We notice that u' (22))) is an analytic function any
open set O C C if dist (Speo(2 X X), Q) is away from O. Recall that dist(Spec( X X), S.S U (0}) + 0 almost surely, for E O with 0 I [2 S Th and (S U (0), R2) > e for some positive constant c. is clear that
u G(z){b - u G()) u G(= + im + GT(G(= + bb + a.s.
Therefore, we conclude Step 3 and complete the proof of Lemma A.2.
A.5. Auxiliary lemmas
This section contains several established results from previous work which needed in proof. Lemma A.3. (Theorem A.43 in Bai & Silverstein (2010)). Let A and B be Hermitian matrices, and let FA and F denote their respective empirical spectral distributions. Then
1
|FA
B
raak B). S
Y
Lemma A.4. (Theorem A.46 Bai & Silverstein (2010)). Let A and be two Hermitian matrices. Then
max (A((A) - (B) < A
Lemma A.5. Let f1, f2. be analytic on D, a connected open set of C, satisfying [f((2) S M for every n. and z in D, and [n(2) converges, as + 0o for each in a subsel of D having a limil point in D. Then there exists a function f. analytic in D for which fn(2) + f2) and fa (2) + (z) for all z E D. Moreover, on any set bounded by a contour interior to the convergence is uniform and fn (2) is uniformly bounded by 2M/c, where is the distance between the contour and the boundary of D.
The proof of Lemma A.5 can be found in Bai & Silverstein (2004).
Lemma A.6. (Lemma 2.2 in Bai & Silverstein (2004)). For z = [21, 2n] T with i..dd standardized random entries, C an n matrix, we have, for any m 2 2,
Cz - trCl S C((E|21 ([(B)))((CCC"")) + E|21 2 (CCC*)) E| T
Lemma A.7. (Burkholder's 's inequality.. Let { be complex martingale difference sequence with respect to the increasing a-field {F&). Then for any m 2 2,
m
Σ 18(1 P|Fx)
+ Cm Σ E|
I
Cm
Lemma A.8. (Sylverster's determinant identity). If A and B matrices of sizes m and then
det(Im AB) det(I, BA).
19


Reviews:
[{"review_id": "arvcLE4xck", "reviewer": "Reviewer_Hb59", "summary": "This paper studies outlier eigenvalues and eigenvectors of conjugate and neural tangent kernels for multi-layer fully connected neural networks at random initialization. The dataset can be a general Gaussian mixture model. This result shows how the information of the group features in the dataset propagates through the multiple layers via the outlier eigenvalues and corresponding eigenvectors.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our paper as well-organized and clearly presented, and we appreciate the constructive comments. Below, we provide point-by-point responses (marked with $\\Huge{\\cdot}$). The statements of weaknesses have been condensed to comply with character limitations.\n\n> Wang et al., 2024 in the reference ... paper?\n* We thank the reviewer for this important comment. The pioneering work by Wang et al.(2024) offers elegant results on the spiked eigenvalues and eigenvectors of CK. To clarify the differences between Wang et al.(2024) and our study, we provide the following comparisons.\n1. We allow for different population covariance matrices $C_a$ in GMM, without relying on the $\\tau_n-$orthonormal assumption introduced in Wang et al. (2024). Specifically, this assumption requires the difference between $tr(C_a)$’s to be $o(p^{2/3})$, whereas our study only requires this difference to be $O(p)$. \n2. We do not require $X^TX$ to contain isolated eigenvalues, as specified in Assumption 2 of Wang et al.(2024). Instead, the isolated eigenvalues of CK and NTK can arise from the group features of input data, i.e., through the terms $VA_{\\ell}V^T$ and $VB_{\\ell}V^T$ in our equations (5) and (8).\n3. They assume isolated eigenvalues are simple, focusing on single eigenvectors in their main theorems. In contrast, our work accommodates isolated eigenvalues with multiplicities greater than one and analyzes the corresponding eigenspaces. \n4. The two studies adopt different assumptions regarding the activation functions. Their work employs weaker conditions on the moments of the derivatives, whereas ours imposes weaker constraints on the expectations of the first and second derivatives.\n5. Given NTK's ability to approximate the generalization and dynamics of neural networks (NNs) on real data, our work explores the properties of NTK which are not addressed in Wang et al.(2024). \n\nAdditionally, we would like to point out a typo: the condition $trC_a^o=O(\\sqrt{p})$ in Assumption 2.1 is unnecessary. This correction, along with the aforementioned comparisons, will be included in the revised version.\n\n> Although the mathematical ... text? \n* For NTK: Let $\\boldsymbol{y}$ represent the label vector of the data. When the loss function $L(t)=\\|\\|\\boldsymbol{y}-\\hat{\\boldsymbol{y}}(t) \\|\\|^2 $, where $\\hat{\\boldsymbol{y}}(t)$ denotes the prediction at time $t$, the time evolution of the residual is approximately given by the following ODE: $$\\frac{d}{dt} \\hat{\\boldsymbol{y}} (t)= \\boldsymbol{K}_{\\text{NTK}} \\big(\\boldsymbol{y} -\\hat{\\boldsymbol{y}} (t)).$$ This indicates that NTK captures the dynamics of the training process in ultra-wide NNs. We draw the following statistical implications:\n1. Based on our theoretical results from Theorem 3.7 to Remark 3.9, the first-order limits of entries in the isolated eigenvectors may or may not contain group features.\n2. When the eigenspace associated with the largest isolated eigenvalue contains group features, NNs tend to prioritize learning this information. Conversely, if this eigenspace lacks group features, NNs instead prioritize learning irrelevant information, diverting attention away from effective group features.\n3. For non-isolated eigenspaces, we conjecture that their first-order limits contain significantly fewer group features, as observed in classical random matrices like the sample covariance matrix, where the entries of non-spiked eigenvectors are delocalized under mild conditions. Consequently, after effectively learning the group features captured by eigenspaces corresponding to isolated eigenvalues, NNs primarily shift their focus to learning from eigenspaces that lack first-order group features. In GMM, this implies that NNs begin leveraging more noise to learn $\\boldsymbol{y}$, leading to overfitting.\n\nThis potentially provides a perspective based on NTK to understand the phenomenon of overfitting. We will include this discussion in the revised paper.\n* For CK: We conducted an experiment on the MNIST data. A histogram showing the spectrum of the third layer's CK matrix and line plots of the eigenvectors for the top 5 eigenvalues can be found at the following link: <https://anonymous.4open.science/r/eig5492/README.md>.\nThe spectrum shows four isolated eigenvalues, with the eigenvector associated with the largest eigenvalue being informative, while the other three appear non-informative. These observations align with our theoretical findings from Theorem 3.4 to Remark 3.6.\n\n> A comprehensive literature ... study.\n* We will add all works mentioned by the reviewer, as well as additional relevant studies, including Yang and Salman (2019,arxiv), Bai and Lee(2020,ICLR), Hron et al.(2020,ICML), Papyan et al.(2020,PNAS), Yang(2020,arxiv), Wang et al.(2023,NIPS), Dandi et al.(2024,arxiv), and Engel et al.(2024,ICLR). Moreover, we will discuss potential extensions such as relaxing the distribution assumption and considering other architectures.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "No, this paper doesn't propose any method or evaluation criteria.", "theoretical_claims": "Yes, I checked the correctness of the proofs for theoretical claims. The results are basically based on previous works, e.g.Benaych Georges & Couillet, 2016.", "experimental_designs_or_analyses": "Yes, the numerical experiments of the paper validate the results of its theoretical analysis.", "relation_to_broader_scientific_literature": "This paper uses random matrix theory to understand the extreme eigenvalues and eigenvectors of empirical kernels in random neural networks.", "essential_references_not_discussed": "There are more references that should be mentioned or discussed in the paper:\n\n1. Ba et al 2023 (https://openreview.net/forum?id=HlIAoCHDWW) studied the Gaussian data with one spiked direction in the population covariance matrix and used the spike in conjugate kernel to study the feature learning.\n\n2. There are several papers that considered a similar Gaussian mixture setting and random neural networks: Liao, Couillet 2018 https://proceedings.mlr.press/v80/liao18a/liao18a.pdf, Couillet, Liao, Mai 2018 https://ieeexplore.ieee.org/document/8553034, Liao, Couillet 2019 https://ieeexplore.ieee.org/document/9022455, Ali, Liao, Couillet 2020 https://openreview.net/forum?id=qwULHx9zld. It would be better to have some discussions on these papers.\n\n3. Liao, Couillet, and Mahoney 2020 (https://papers.nips.cc/paper/2020/hash/a03fa30821986dff10fc66647c84c9c3-Abstract.html) studied the random Fourier feature model and the random feature regression model in this case.", "other_strengths_and_weaknesses": "## Strengths:\nThis presentation is clear and well-organized. The mathematical results are clearly presented.\n\n## Weaknesses:\n1. Wang et al., 2024 in the reference have studied the propagation of the spike eigenvalues and eigenvectors for conjugate kernel matrix through multiple layers. I do not know the novelty of this paper comparing with Wang et al., 2024. Wang et al., 2024 also have similar simulation for gaussian mixture dataset. There should be a clear comparison with this paper. Do the results of Wang et al., 2024 totally cover the results of CK in the current paper?\n\n2. Although the mathematical results presented in Section 3 are clearly presented, there is a lack of explanations regarding the statistical implications of these results. Can we derive any practical applications or insights for the machine learning community from these findings? Even some case studies or remarks should be better included in the main text.\n\n3. A comprehensive literature review should be included to provide context and establish the relevance of the work. Additionally, a discussion should be presented outlining the limitations of the current study.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQW3fFeQRn", "reviewer": "Reviewer_xuzj", "summary": "This paper analyzes the eigenvalues and eigenvectors of the Conjugate Kernel (CK) and Neural Tangent Kernel (NTK) for deep feedforward networks with random weights, in a high-dimensional setting where the input data come from a Gaussian Mixture Model (GMM). The authors show that, under certain assumptions, “spiked” or isolated eigenvalues of the CK and NTK can emerge outside the main (bulk) support of the limiting spectral distribution. They then precisely characterize the limiting positions of these outlier eigenvalues and analyze the asymptotic structure of the eigenvectors (or eigenspaces). The main conceptual point is that these isolated eigenvalues reflect group (or cluster) structure in the data—i.e. the eigenvectors encode the mixture components. Intuitively, the paper demonstrates how features of the data propagate layer by layer in a randomly initialized deep network, indicating how certain group structure is captured in these outlier modes. Overall, it provides theoretical insight into how CK/NTK spectra depart from their bulk distribution in the presence of class- or cluster-relevant structure.\n\n\n## update after rebuttal\n\nI am happy with the rebuttal responses and will maintain my originally high score 4 (Accept). **Important caveat**  I do agree with reviewer `Hb59` that a proper discussion of Wang et. al. 2024 is highly warranted given the high degree of similarity.   If the paper is accepted, Authors should be super clear about what is new or different in their results compared to the existing results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for his/her encouraging feedback. The recognition of our work's novelty, technical rigor, and clarity is highly motivating. We also greatly appreciate the insightful comments provided. In the following, we provide point-by-point responses (marked with $\\Huge{\\cdot}$). The statements of comments have been condensed due to character limitations.\n> The paper already cites much of the very relevant literature. But it ... two lines of theory.\n* We thank the reviewer for sharing these references, which will be incorporated into the revised version.\n\n> Data Model & Assumptions: ... distributions.\n* GMMs are widely used in machine learning analysis due to the technical convenience. In large-scale neural network models, mixture model behave asymptotically as if GMMs (Seddik el al. (2020, ICML); Couillet and Liao (2022, Cambridge University Press)). In fact, from the perspective of random matrix theory, many properties of Gaussian random matrices can be extended to random matrices with general distributions under certain moment conditions—a phenomenon known as universality (Ding and Yang (2018, AOAP)). To support our findings, we perform an experiment on the MNIST dataset. The visualization of the spectrum and eigenvectors of the CK matrix can be accessed via the following link:   <https://anonymous.4open.science/r/eig5492/README.md>. The spectrum reveals four isolated eigenvalues. Notably, the eigenvector corresponding to the largest eigenvalue contains group features, whereas the remaining three appear to lack such information. These empirical observations are consistent with our theoretical results presented in Theorem 3.4 through Remark 3.6.\n\n> Training: While ... some value.\n* We appreciate the suggestion to discuss training dynamics, and while we agree it would be valuable, space constraints prevent us from including it here. Instead, we kindly refer you to our detailed response to Reviewer Hb59’s second question, where this topic is addressed in detail.\n\n> As mentioned earlier, some discussion around neural collapse ... readers.\n* We acknowledge the importance of discussing the neural collapse phenomenon in relation to our study, as both lines of work consider input data with inherent cluster structures. Neural collapse illustrates that when training error reaches zero, classifiers tend to converge towards class means. In contrast, our research demonstrates that even without training, certain eigenvectors of kernel matrices (NTK and CK) derived from neural networks can encode group features within isolated eigenvalues. We will provide discussions on neural collapse and feature learning in the revised version.\n\n> The paper celebrates its key finding ... takeaways.\n* According to Lemma 2.4, the bulk eigenvalue distributions of the CK matrix exhibit similar curves across different activations, differing only by a constant scaling factor. However, Figure 4(b) shows an isolated eigenvalue for the ReLU activation, whereas Figure 4(a) does not display this phenomenon for the polynomial activation. This observation aligns with our findings on the influence of activations on the emergence of isolated eigenvalues. In the revised version, Section 3.3 will clearly elaborate on this point to highlight the effect of different activations.\n\n> My most important lingering question ... analysis?\n* Firstly, we would like to clarify that our analysis is based on the asymptotic spectral equivalence of kernel matrices, as established by Lemmas 2.4 and 2.5 in our paper. This equivalence allows us to operate within a low-rank perturbation framework, thereby circumventing the technical complexities arising from  dependencies introduced by nonlinear activation functions. Consequently, our results are derived using standard random matrix theory tools, without the need for free probability theory techniques.\n\n> The follow-up to the previous question ... kernel?\n* Our theoretical results hold for the GMM model. If the eigenspectrum of the covariance matrix is explicitly given without any class structure, it simplifies to a special case of GMM with only one group. Consequently, the asymptotic spectral equivalence holds for the CK and NTK matrices, as stated in Lemmas 2.4 and 2.5. This reduction allows us to focus on investigating the isolated eigenvalue and eigenvector of matrices of the form: $$X^TX+a\\mathcal{1}\\mathcal{1}^T+b\\psi\\psi^T+c\\mathbf{I},$$ where $\\mathcal{1}$ and $\\psi$ are two vectors. Under certain conditions on $(a\\mathcal{1}\\mathcal{1}^T+b\\psi\\psi^T)$, isolated eigenvalues may emerge. Regarding bulk eigenvalues and eigenvectors, the limiting spectral distribution is provided in Lemma 2.6. Further theoretical analysis, such as the rigidity of the bulk eigenvalues and the delocalization of eigenvectors, are beyond the scope of this paper. However, we believe that it is feasible to establish the local behavior of the bulk spectrum if the local laws for CK and NTK matrices can be derived.", "claims_and_evidence": "While this is a theoretical contribution and the support of claims is primarily theoretical, here are limited experiments that support the key claims:\n\n- Main Claim 1: “Certain eigenvalues of the CK/NTK can lie outside the bulk of the spectral measure in high-dimensional regimes.” Evidence: The authors present formal statements (Theorem 3.1 & 3.2) they show simulations that show outliers numerically.\n- Main Claim 2: “These outlier eigenvalues correspond to group or cluster structure in the data.”(Theorem 3.4 & Corollary 3.5).\nEvidence:  They provide numerical plots (eigenvectors strongly correlate with mixture class membership).\n- Main Claim 3: “The position of these isolated eigenvalues and the structure of their eigenvectors can be characterized by expansions related to the data covariance and the choice of activation function.”\nEvidence: The paper provides explicit formulas (involving terms α_ℓ,2, α_ℓ,3, etc.) for the CK and NTK spiked eigenvalues. The simulations with polynomial vs. ReLU activations illustrate how the location of outliers depends on activation.", "methods_and_evaluation_criteria": "the methods and evaluation approach (matching theoretical spectral predictions to empirical ones) are well-aligned with the paper’s theoretical goals:\n- The authors’ primary method is an asymptotic random matrix theoretical derivation of eigenvalue limits. They compare theoretical predictions with histograms of empirical eigenvalues from random networks.\n- The data are synthetic or follow a controlled GMM with different class covariances, which is reasonable for theoretical validation.", "theoretical_claims": "- The main theorems (Theorem 3.1 & 3.2) rely on finite-rank perturbation arguments from random matrix theory. The approach of showing that the empirical kernel matrix is well-approximated by a low-rank correction to a known limiting distribution is standard.\n- The authors reference prior results on deterministic equivalents for CK and NTK from (Gu et al., 2024) and from classical RMT references (e.g., Bai & Silverstein). The extension to identifying spiked eigenvalues and projecting onto the eigenspace (Theorem 3.4) is also sound\n- While I did not check all the proofs in depth, I did not spot an obvious algebraic or conceptual flaw, they seem sensible in the sense that they rely on well known established lemmas (like Woodbury identity, resolvent expansions, Schur complement formula, etc.).  However the proofs are quite technical and dense, and I might have missed something. \n- One area that I am less certain about:  In a multi-layer setting, different features are tied/coupled together, and they are not necessarily a Gaussian matrix to approximate their eignevalues. Can authors explain how they can seeming go around this dependencies to be able to apply free probability on deep representations? What are the key facts/lemmas that enable this?", "experimental_designs_or_analyses": "Overall, the experimental design matches the paper’s theoretical claims and is valid for a proof-of-concept demonstration:\n- The experiments primarily involve plotting eigenvalue histograms and the top eigenvectors for various settings (e.g. polynomial vs. ReLU activation). This is sufficient for a theoretical paper.\n- The dimensionalities (n, p) used are large enough (e.g. n=1200, p=600) to demonstrate the asymptotic effect, and the results show visually distinct outliers.", "relation_to_broader_scientific_literature": "The paper extends and complements a line of research studying  neural kernels in high dimensions using random matrix theory. \n\n- CK/NTK in Infinite Width: Prior works (Jacot et al. 2018;) derived closed-form expressions for infinite-width kernels but did not analyze a mini-batch of samples, and hence analyze spiked eigenvalues beyond the main spectrum. The present paper looks at how mixture-structure in data can cause outliers in large finite dimension.\n- Spiked Models in RMT: The approach of analyzing finite-rank deformations of a baseline limiting distribution is reminiscent of Baik et al. (2005) and other spiked covariance matrix models. Here, the baseline is the limiting kernel, and the finite-rank perturbation arises from class-structured data.\n- Empirical CK/NTK Spectra: Related to works by Fan & Wang (2020) or Wang et al. (2024), who studied the spectral distribution in large neural networks. This paper  approach is similar but focuses specifically on outliers and their eigenvectors in presence of cluster / class structure. \n- Feature Learning & Data Clusters: The demonstration that the outlier eigenvectors align with class membership echoes analyses in \nWorth improving: \n- “Neural Collapse”: I thin the connection of this result to neural collapse worth discussing. While the paper is about a random state / initialization, one could view this class-structure as a form of neural collapse or other clustering phenomena in wide networks, but from a pure RMT perspective. This might be interesting for some readers. \n- \"Feature learning\": because we know that classical NTK  is not leading to learning dynamics, it might be worth making some statements of how these results could affect feature learning in middle layers? In that sense, some brief mention and contrast with feature learning literature (Tensor Programs, muP literature) might also be worthwhile.", "essential_references_not_discussed": "the paper already cites much of the very relevant literature. But it might benefit from some other citation that are more broadly related:\n- Neural Collapse (Papyan et al., 2020) for how class means or class structure leads to low-rank phenomena in feature space. This might give the authors a link between their results and the class structure that emerges naturally during training in neural collapse. \n- Yang’s Tensor Programs (Yang, 2019–2021) might be relevant because it also addresses expansions of NTK under more general architectures. Although the paper’s setting is quite different, referencing it could highlight synergy between the two lines of theory.", "other_strengths_and_weaknesses": "Strengths:\n- Novelty: Detailed spiked eigenvalue analysis for CK/NTK in multi-layer random networks with GMM data is new and addresses a gap in the literature.\n- Technical Rigor: The paper’s theorems are grounded in well-established RMT methods (deterministic equivalents, rank-1 updates).\n- Clarity & Focus: the statements of main theorems and the structured proofs in the appendix are well written \n\nWeaknesses:\n- Data Model & Assumptions: Results rely on a GMM with somewhat idealized settings (independence, certain covariance scaling). If this is truly about initialization and the input is taken to be the real input, in reality, large-scale datasets do not always behave like mixtures of Gaussians. For example, the structure of the NTK and Conjugate kernels as revealed by lemma 2.4 to be a mere low-rank perturbation of the input at any depth, seems overly simplistic of the way that real networks shape and transform input distributions. \n-  Training: While the paper makes brief remarks on training, given that one of the key findings of the paper is on NTK, more exploration and discussion of training dynamics (though it is acknowledged as outside scope) would probably add some value", "other_comments_or_suggestions": "- As mentioned earlier, some discussion around neural collapse and feature learning literature would add value for some readers. Considering inputs to be clustered is a highly idealized and unrealistic initialization (as devil's advocate would ask: why would we need a neural network if the input is clustered?). Thus, making some connection to neural collapse would make the results more interesting for some readers. \n- The paper celebrates its key finding as finding connection between outlier eigenvalues and the activation itself, via the recurrent equations defined over moments of the activation. But they do not explore differences of activations from the point of view of their theory. For example in Figure 4 a& b for  poly and ReLU activation look largely & qualitatively the same, which seems to underserve the key focus of this paper, which is uncovering role of activation on the magnitude and position of eigenvalue outliers. My suggestion is that authors try to draw some implications from their theory about the activations that are not currently known, and show that that empirically holds. Many future readers may find these practical insights to be be the key takeaways of this paper, and without them, they will be left with no takeaways.", "questions_for_authors": "- My most important lingering question is, most classical results in free probability either work  for independent features, or if they are dependent but follow Gaussian distributuion , so their dependence is captured by the covariance. However, once we pass them through one layer of non-linear activations, neither of which is true. How can we get the eigenvalue distribution of the conjugate and tangent kernel here? Is the key the class-structure in the inputs and the resulting low-rank perturbation that enables the rest of the analysis? \n- The follow-up to the previous question is could your finite-rank spike analysis be extended to more general distributions than the GMM? For example,  what if instead of class-structure, we have an input whose eigenspectrum of covariance is explicitly given to us? Regardless of the spiked eigenvalues structure, Can a similar analysis reveal the bulk eigenspectrum conjugate and tangent kernel ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jyTf0NBuS1", "reviewer": "Reviewer_tzoT", "summary": "The paper explores the eigenvalue spectrum of the conjugate kernel (CK) and the neural tangent kernel (NTK) with random weights. The authors demonstrate the existence of isolated eigenvalues and present a theoretical approach to identifying where they lie and their possible impact on the model.\n\n## Update after rebuttal\n\nAfter the rebuttal period, I felt the authors expanded on their key points in a suitable way based on the criticism. However, I did not feel that they clearly outlined the important points and conclusion of the paper and how it builds on the current state of the art. I feel that with additional experiments and discussion related to their conclusions, the paper would be a stronger contribution. However, the work was well performed and so I am happy to raise my score by one point.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive questions. In the following, we provide point-by-point responses (marked with $\\Huge{\\cdot}$).\n> The paper doesn't explain why these results are novel or relevant. The results appear well validated and perhaps interesting. However, how can a machine learning practitioner, theoretical or otherwise, use these results to learn more about their networks/activation functions or training?\n* We thank the reviewer for this insightful comment. Below, we clarify the novelty and practical implications of our findings.\n1. Theoretical Insights into NTK and CK:\n   - Our theoretical results in Section 3 reveal that the first-order limits of entries in isolated eigenvectors, derived from both NTK and CK, may or may not encode group features.\n   - To empirically verify this, we conducted an experiment using the MNIST dataset. A histogram showing the spectrum of the third layer's CK matrix and line plots of the eigenvectors for the top 5 eigenvalues can be found at the following link: <https://anonymous.4open.science/r/eig5492/README.md>.\nThe spectrum shows four isolated eigenvalues, with the eigenvector corresponding to the largest eigenvalue being informative, while the other three appear non-informative. These observations align with our theoretical findings from Theorem 3.4 to Remark 3.6.\n\n2. Training Dynamics in Ultra-Wide Neural Networks (NNs):\n   - By analyzing NTK in ultra-wide NNs, we uncover deeper implications regarding the dynamics of training in such architectures. Let $\\boldsymbol{y}$ represent the label vector of the data. When the loss function $L(t)=\\|\\|\\boldsymbol{y}-\\hat{\\boldsymbol{y}}(t) \\|\\|^2 $, where $\\hat{\\boldsymbol{y}}(t)$ denotes the prediction at time $t$, the time evolution of the residual is approximately given by the ODE: $\\frac{d}{dt} \\hat{\\boldsymbol{y}} (t)= \\boldsymbol{K}_{\\text{NTK}}\\big(\\boldsymbol{y} -\\hat{\\boldsymbol{y}} (t)).$ This indicates that the NTK captures the dynamics of the training process in ultra-wide NNs. \n   - When the eigenspace associated with the largest isolated eigenvalue contains group features, NNs tend to prioritize learning this information. Conversely, if this eigenspace lacks group features, NNs instead prioritize learning irrelevant information, diverting attention away from effective group features.\n   - For non-isolated eigenspaces, we conjecture that their first-order limits contain significantly fewer group features, as observed in classical random matrices like the sample covariance matrix, where the entries of non-spiked eigenvectors are delocalized under mild conditions. Consequently, after effectively learning the group features captured by eigenspaces corresponding to isolated eigenvalues, NNs primarily shift their focus to learning from eigenspaces that lack first-order group features. In GMM, this implies that NNs begin leveraging more noise to learn $\\boldsymbol{y}$, resulting in overfitting. This potentially provides a perspective based on NTK to understand the phenomenon of overfitting.\n\n3. Impact of Activation Functions on Group Features Encoding:\n   - As demonstrated in Figure 4 of our paper, different activation functions yield distinct behaviors in the eigenvalues and eigenvectors. A carefully selected activation function (e.g., ReLU in Figure 4(b) and 4(d)) can effectively encode group features within NNs, facilitating the integration of group features from the initial stages.\n   - These findings suggest that careful selection of activation functions can significantly enhance a NN's ability to capture and represent group features. \n \n> What role do the authors expect the isolated eigenvalues play in the networks during training?\n* Please refer to our response to the previous question, specifically points 1 and 2, which address the role of isolated eigenvalues in networks during training.\n\n> How do you expect the results to change when applied to more architectures?\n* We believe that isolated eigenvalues and eigenspaces are likely to exist across various NN architectures (e.g., fully connected neural networks (FCNs) with dropout), provided the input data exhibits certain classification characteristics. Investigating these isolated eigenvalues and eigenspaces in different NN structures remains a largely unexplored problem. However, under current theoretical frameworks, it is challenging to directly prove the existence of isolated eigenvalues and eigenspaces in architectures different from FCN. For instance, whether the spectral equivalence described in Lemma 2.4 and Lemma 2.5 holds for other NN architectures remains unknown. We aim to address these related challenges in our future work.\n\nFurthermore, in the revision we will include more references on the spectra of NTK and CK matrices, such as Pennington and Worah (2017, NIPS), Yang and Salman (2019, arxiv), Hron et al. (2020, ICML), Murray et al.(2022, arxiv), Wang et al. (2023, NIPS), and Belfer (2024, JMLR).", "claims_and_evidence": "There are very few claims made in the paper rather the authors highlight specific results and theoretical insights. To this end, there is evidence to support their claims.", "methods_and_evaluation_criteria": "Very few benchmarks and datasets are used to demonstrate the paper's main results. It focuses far more on theoretical advances.", "theoretical_claims": "I did not see any obvious theoretical issues but did not check all equations closely.", "experimental_designs_or_analyses": "The authors do not present a variety of experiments or testable hypotheses.", "relation_to_broader_scientific_literature": "There are many references made to alternative research performed on the spectra of the NTK or CK matrices. The authors do not relate their work too extensively to this literature, mostly introducing their own ideas.", "essential_references_not_discussed": "More citations could be included regarding similar work studying the spectra of the NTK. However, it would not support the experiments or results of the manuscript.", "other_strengths_and_weaknesses": "The paper doesn't explain why these results are novel or relevant. The results appear well validated and perhaps interesting. However, how can a machine learning practitioner, theoretical or otherwise, use these results to learn more about their networks/activation functions or training?", "other_comments_or_suggestions": "I would suggest that the authors apply their analysis to more common network architectures and try to connect their results to understandable results in machine learning.", "questions_for_authors": "1. What role do the authors expect the isolated eigenvalues play in the networks during training?\n2. How do you expect the results to change when applied to more architectures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18eBjmJywN", "reviewer": "Reviewer_u6qk", "summary": "This paper analyzes the spectral properties of deep feedforward neural networks with random weights, focusing on Gaussian mixture model inputs. It rigorously examines isolated eigenvalues in the conjugate and neural tangent kernels, showing how they capture group features and evolve through hidden layers, influenced by covariance differences and activation functions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their kind and encouraging feedback on our paper: \"The work is highly thorough, and the theoretical contribution is solid. The precise characterization of the isolated eigenvalues is very interesting\". This positive recognition motivates us greatly. Below, we provide detailed point-by-point responses (marked with $\\Huge{\\cdot}$) to address all identified weaknesses. The statements of weaknesses have been condensed to comply with character limitations.\n> Regarding technical novelty,..., let me know.\n* Fan and Wang (2020) proved the limiting spectral distributions of both CK and NTK under the linear width assumption. Subsequently, Wang et al. (2024) established the asymptotic behavior of spiked eigenvalues and eigenvectors of CK. In this paper, while our theoretical results are derived using the standard random matrix framework based on the pioneering work of Gu et al. (2024), they reveal intricate phenomena involving the isolated eigenvalues and eigenspaces of neural networks. Specifically, as outlined in Section 3.2, the first-order limits of entries in isolated eigenvectors of both CK and NTK may or may not encode group features. To clarify the differences between Wang et al. (2024) and our study, we provide the following comparisons.\n1. We allow for different population covariance matrices $C_a$ in the GMM data, without relying on the $\\tau_n-$orthonormal assumption introduced in Wang et al. (2024). Specifically, the $\\tau_n-$orthonormal assumption requires that the difference between $tr(C_a)$’s be of order $o(p^{2/3})$, whereas our framework only requires this difference to be $O(p)$. \n2. We do not require $X^TX$ to contain isolated eigenvalues, as specified in Assumption 2 of Wang et al. (2024). Instead, the isolated eigenvalues of CK and NTK can arise from the group membership structure of the input data, i.e., through the terms $VA_{\\ell}V^T$ and $VB_{\\ell}V^T$ in our equations (5) and (8).\n3. They assume isolated eigenvalues are simple, focusing on single eigenvectors in their main theorems. In contrast, our work accommodates isolated eigenvalues with multiplicities greater than one and analyzes the corresponding eigenspaces. \n4. The two studies adopt different assumptions regarding the activation functions. Their work employs weaker conditions on the moments of the derivatives, whereas ours imposes weaker constraints on the expectations of the first and second derivatives.\n5. Given NTK's ability to approximate the generalization and dynamics of neural networks (NNs) on real data, our work explores the properties of NTK which are not addressed in Wang et al. (2024).\n\nAdditionally, we conducted experiments on the MNIST dataset under conditions that satisfy the linear width assumption. The empirical results support the existence of distinct isolated eigenvectors as revealed by our theoretical findings. Please click the following link to view the results: <https://anonymous.4open.science/r/eig5492/README.md>.\n\n> While this paper provides very complete results, ..., provide further insight.\n* By analyzing NTK in ultra-wide NNs, our theoretical findings provide insights into the dynamics of training in such architectures. Let $\\boldsymbol{y}$ denote the label vector of the data. When the loss function is defined as $L(t)=\\|\\|\\boldsymbol{y}-\\hat{\\boldsymbol{y}}(t) \\|\\|^2$, where $\\hat{\\boldsymbol{y}}(t)$ represents the prediction at time $t$, the time evolution of the residual is approximately given by the ODE: $$\\frac{d}{dt} \\hat{\\boldsymbol{y}} (t)= \\boldsymbol{K}_{\\text{NTK}} \\big(\\boldsymbol{y} -\\hat{\\boldsymbol{y}} (t)).$$ This indicates that the NTK captures the dynamics of the training process in ultra-wide NNs. We draw the following statistical implications:\n1. Based on our theoretical results from Theorem 3.7 to Remark 3.9, the first-order limits of entries in the isolated eigenvectors may or may not contain group features.\n2. When the eigenspace associated with the largest isolated eigenvalue contains group features, NNs tend to prioritize learning this information. Conversely, if this eigenspace lacks group features, NNs instead prioritize learning irrelevant information, diverting attention away from effective group features.\n3. For non-isolated eigenspaces, we conjecture that their first-order limits contain significantly fewer group features, as observed in classical random matrices like the sample covariance matrix, where the entries of non-spiked eigenvectors are delocalized under mild conditions. Consequently, after effectively learning the group features captured by eigenspaces corresponding to isolated eigenvalues, NNs primarily shift their focus to learning from eigenspaces that lack first-order group features. In GMM setting, this implies that NNs begin leveraging more noise to learn $\\boldsymbol{y}$, resulting in overfitting.\nThis potentially provides a perspective based on NTK to understand the phenomenon of overfitting.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I have checked all of the proofs.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The most relevant work is (Gu et al., 2024), which serves as a complementary study to this paper.", "essential_references_not_discussed": "NO.", "other_strengths_and_weaknesses": "Strengths:\n\n1 This paper studies the Stieltjes transforms of CKs and NTKs using the deterministic equivalents provided by previous work (Gu et al., 2024). The work is highly thorough, and the theoretical contribution is solid.\n\n2  The precise characterization of the isolated eigenvalues is very interesting.\n\nWeaknesses:\n\n1 Regarding technical novelty, the spectra of CKs and NTKs with linear width have been well studied by (Fan & Wang, 2020). This paper, however, explores the case of extremely large width. The prior width setting is more practical and extends beyond lazy learning, with the development of a new theoretical framework. Moreover, the deterministic equivalents of CKs and NTKs were already provided in previous work (Gu et al., 2024), and this paper examines the same NN model. Standard random matrix tools can easily derive results for the Stieltjes transforms, isolated eigenvalues, and eigenvectors. The technique used here is standard. If there is any new technical contribution I missed, please let me know.\n\n2 While this paper provides very complete results, the relevance between the theoretical findings and learning tasks is weak. Particularly for readers unfamiliar with RMT, the theoretical results may not offer novel insights.  For example,  (Gu et al., 2024) applies their theoretical results to design new compression methods. For this paper, my suggestion is to discuss  the impact of the isolated eigenvalues on training dynamics or generalization performance. It could provide further insight. \n \nOverall, I am positive about this paper, but I encourage the authors to further explore the theoretical results in the context of learning.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the outlier eigenvalues and eigenvectors of conjugate and neural tangent kernels for fully connected deep neural networks having random weights, with input data drawn from a general Gaussian mixture model.

The authors have done a commendable job during the rebuttal, and the majority of reviewers are now **convinced of the significance and merit of the work**.

As noted by Reviewer Hb59 and others, there remains room for improvement, particularly in terms of:
1. Clarifying the differences between this paper and Wang et al., 2024; and
2. Emphasizing the practical implications of the proposed theoretical analysis for ML applications.

Nevertheless, given the importance of the problem and the rigorous and technically solid contributions, I believe this paper will be a valuable addition to the ICML community and a useful reference for future work.

As a result, I recommend acceptance of this paper to ICML 2025.
Please ensure that the additional numerical results, clarifications, and discussions provided during the rebuttal phase are included in the final version.
Decision: Accept (poster)