Divergence of Neural Tangent Kernel in Classification Problems
Authors: Zixiong Yu
Songtao Tian
Guhan Chen
Venue: ICLR 2025 Poster (2025)
Keywords: neural tangent kernel
neural network
cross-entropy loss
classification problem
Primary area: learning theory
TL;DR: This paper investigates the divergence of NTK in fully connected and residual networks during classification tasks, showing that NTK theory is not applicable under the cross-entropy loss function as the empirical NTK fails to converge over time.

Abstract: This paper primarily investigates the convergence of the Neural Tangent Kernel (NTK) in classification problems. This study firstly show the strictly positive definiteness of NTK of multi-layer fully connected neural networks and residual neural networks. Then, through a contradiction argument,  it indicates that, during training with the cross-entropy loss function, the neural network parameters diverge due to the strictly positive definiteness of the NTK. Consequently, the empirical NTK does not consistently converge but instead diverges as time approaches infinity. This finding implies that NTK theory is not applicable in this context, highlighting significant theoretical implications for the study of neural networks in classification problems. These results can also  be easily generalized to other network structures, provided that the NTK is strictly positive definite.

Published as a conference paper at ICLR 2025
DIVERGENCE OF NEURAL TANGENT KERNEL IN CLAS- SIFICATION PROBLEMS
Zixiong Yu" Noah's 's Ark Lab Huawei Technologies Ltd. Shenzhen, Guangdong, China mrrixiiong2@huawer com
Songtao Tian
Department of Mathematical Sciences
Tsinghua University
Haidian District, Beijing, China
t tiansongtao 20200 t www.gggw........o.eoooooooo
Guhan Chen
Department of Statistics and Data Science Tsinghua University
Haidian District, Beijing, China
chen chen-gh23@maile gh23@mailee wtw.isss.ssssissse... tsingnua..dde eduu.
ABSTRACT
This paper primarily investigates the convergence of the Neural Tangent Kernel (NTK) in classification problems. This study firstly show the strictly positive definiteness of NTK of multi-layer fully connected neural networks and residual neural networks. Then, through a contradiction argument, it indicates that, during training with the cross-entropy loss function, the neural network parameters diverge due to the strictly positive definiteness of the NTK. Consequently, the empirical NTK does not consistently converge but instead diverges as time approaches infinity. This finding implies that NTK theory is not applicable in this context, highlighting significant theoretical implications for the study of neural networks in classification problems. These results also be easily generalized other network structures, provided that the NTK is strictly positive definite.
INTRODUCTION
Recently, neural network have achieved remarkable performance in various fields, such as computer vi- sion, natural language processing, and generative models. the field of computer vision, Krizhevsky et al. (2012) proposed AlexNet, which significantly outperformed traditional methods in the ImageNet competition using deep convolutional network. Subsequently. He et al. (2016) introduced ResNet, which addressed the degradation problem in training deep network by incorporating residual blocks, significantly improving model performance. In the field of natural language processing, Vaswani et al. (2017) proposed the Transformer, which greatly enhanced self-attention mechanism. In the field of generative models, Kingma & Welling (2013) introduced the Variational Autoencoder (VAE), enabling efficient training of generative models. However, despite the excellent performance of network in practical applications, there is still no complete theoretical explanation for why network perform so well.
toseenee----eeuncce tasks through the
The theoretical properties of network have been subject of study for a long time. In terms of the expressive power of network, Hornik et al. (1989); Cybenko (1989) proposed the well-known universal approximation theorem, which demonstrates that neural network with a sufficient number of parameters can approximate any continuous function. Recently, many works have continued to explore the expressive power of network in broader and more structured contexts, such (Cohen al., 2016; Hanin & Sellke, 2017: Lu al., 2017), among others. However, these works focus solely on the model's expressive capabilities of network and neglect their statistical properties, thus failing to fully explain the excellent performance network.
First author. Co-first author. iCorrespondigg author.

---

Published as a conference paper at ICLR 2025
Regarding the statistical properties of network, many studies have focused on the generalization ability of network. Within the static non parameete regression framework, Bauer & Kohler (2019); Schmidt- Hieber (2020) investigated the minimax optimality of network when different Hölder function classes are used as the true target functions. At the same time, the dynamic training of network is equally important. Du et al. (2018); Allen-Zhu et al. (2019); Chizat al. (2019) studied the performance of sufficiently wide network on the training set after gradient descent and stochastic gradient descent in regression problems. Building on this, Jacot et al. (2018) explicitly proposed the concept of the Neural Tangent Kernel (NTK) to more precisely characterize the performance of sufficiently wide network during gradient descent.
In the NTK theory framework, research can be typically divided into two steps: studying the convergence of the NTK, i.e., whether the neural network kernel can converge to the NTK; and investigating the performance of the corresponding NTK regressor within the kernel regression framework. The former establishes the validity of the NTK theory, while the latter elucidates the properties of network. For instance, Arora al. (2019); Lee et al. (2019); Hu et al. (2021) verified the NTK convergence of various networks under the MSE loss function. Furthermore, Lai et al. (2023a); Li al. (2023) demonstrated the uniform convergence of the NTK for fully-connede directly trained with MSE. Additionally, many work have investigated the generalization ability of the NTK regressor in MSE regression problem (Lee et al., 2019; Hu et al., 2021; Suh et al., 2021; Lai et al., 2023a; Li et al., 2023; Tian & Yu, 2024). However, for networks trained using the cross-entropy function, which is common in classification problems, the property of the empirical NTK is still an open question.
networks
This article primarily investigates the convergence of NTK in classification problem. As mentioned above, the convergence of NTK has been thoroughly studied in regression problem with the MSE loss function. In classification problems, the cross-entropy function more suitable for handling categorical label data, and thus it has wider range of applications compared to MSE. The convergence of NTK during training with the cross-entropy loss function, which is necessary, however, has hardly been researched. According to our results, under the cross-entropy loss function, the NTK of multi- layer fully connected neural network and residual neural network will no longer uniformly converge to NTK but instead diverge as time approaches infinity. This implies that NTK theory is no longer applicable in this case. From this perspective, our work has significant theoretical implications.
1.1 RELATED WORKS
The concept of NTK was first introduced by Jacot et al. (2018) to approximate the training process of wide network using kernel regression. There has been substantial research on the convergence of NTK in regression problems, with early works focusing on the pointwise convergence of Neural Network Kernels (NNK) to NTK (Du et al., 2019: Allen on this, many have proven the uniform convergence of NTK, meaning that as the network width approaches infinity, NNK uniformly converges to NTK across all possible samples and at every point in the training process (Lai et al., 2023a; Li et al., 2023; Lai et al., 2023b; Chen et al., 2024). For NTK in classification problems, due to the complex nature of the cross- cross-entropy loss function, many rescarchers have used MSE loss substitute (Vyas et al., 2022; Hron al., 2020). Some studies also examine the generalization ability of neural networks for classification problems under the cross-entropy loss function from different perspectives. Recent works, such as Ji & Telgarsky (2019); Taheri & Thrampoulidis (2024), have explored the dynamics of neural networks under the regime where the network width m + do and the training time T' is fixed and finite. These studies demonstrate that under such conditions, the NTK regime holds during training. In contrast, our work examines the case where m is fixed and T - oc. We show that in this setting, the NTK exhibits divergence, preventing uniform convergence to kernel predictors and revealing distinct overfitting dynamics in classification tasks.
al., 2019; Arora et al., 2019). Building
1.2 CONTRIBUTION
In this article, we study the convergence of NTK in classification problems for fully connected neural network and residual neural network. We point out that when using the cross-entropy loss function commonly used in classification problems, the NTK of network cannot achieve uniform convergence over time and across all training samples.
2

---

Published as a conference paper at ICLR 2025
First, we investigate the strict positive definiteness of NTK for fully connected network and residual network. We consider network defined on a general compact domain, i.., imput samples x' E X' C R4 convert the NTK of fully connected network and residual Through appropriate transformations, network into a recursive form and an arc-cosine kernel form, respectively, and then prove their strict positive definiteness. After proving the strict positive definiteness of NTK, analyze the dynamic properties of network and use proof by contradiction to demonstrate that under the cross-entropy loss function, the NTK of network cannot achieve uniform convergence.
To the best of our knowledge, our article is the first to theoretically prove the divergent property of NTK in classification problems. This aligns with the experiment results and reflects the limitations of earlier theoretical work on related issues. Overall, our article demonstrates that using NTK to study the properties of network in classification problems not good choice. Our results can also be easily generalized to other network structures, provided that the NTK is strictly positive definite.
2
PRELIMINARIES
2.1 NOTATIONS
For two sequence {r,] and {yn). we denote by In, O(yn) if there exists some positive constant
C' such that |z, S C7 yn holds for any large enough For index sequence i 1.... n, we write as E [n] for notational simplicity, We denote the sample marix as X (of 22 x22' and Y (y1, "..-.. yn,² For a function f(-) Rd I, denote f(X) represent the entry-wise application the function to each element in X. That is, f(X) = f(z))) ... ff........ fT This means the function / is applied individually
each sample in the sample matrix. We denote by [n] the sequence 1,2, ... n. In this paper, we consider the binary classification problem. We assume the pairs of training data (z, y) comes from X' {0, 1}. where X' is a compact subset of Rd The label comes from {0, } with unknown distribution. It is worth noting that, we do not make requirements on the separability of data, well as other characteristics.
2.2 NEURAL NETWORK
Fully connected network As simple but representative kind of network, we consider fully connected neural network with multi-hidden layer:
(z) x;;
2 (x) V mm (w) f(x: 0) ((L+))
,1= 1,2,
(2.1)
Residual network The residual network is also kind of widely used network, which features skip connections as following.
a 0 (x)
(Ax + b);
1710
(0)()) = V mi
a (w-la (0)))++0)))))) +b0), 1,2,,....
(2.2)
a (x) = (1-1) + +ay (x) +d 1=1,,....,
f(:;0) = (((+1) (1)
In both two models, a is the entry-wise ReLU function as (x: = max(0, x), and we use 0 to denote the vector that the parameters above flatten be. For 1 E [Z.]. the dimension of matrices W and 60) are pmmxm-- and g1 respectively. Additionally, the dimension of W(L+1) is IR xms, Since residual networks introduce skip connections compared to fully connected networks (the third equation in eq. (2.2)), the width of each layer in a residual network the same, that is.
mg my mi, where weights (1)) Rmixm and bias d()) E R The parameter in
3

---

Published as a conference paper at ICLR 2025
eq. (2.2) is chosen scaling factor. Additionally, since the input dimension and width may also differ, an additional linear transformation is applied at the input layer (see the first equation in eq. (2.2)), with weights A Rm and bias b R At initialization, for the fully connected neural network, the parameters of W and b(0) are independently and identically distributed (i.id.) as standard Gaussian variables No, 1). For the residual network, all parameters are initialized as i.idd random variables from (0, 1), except for ((1) and d which are initialized to zero. This exception made solely for the convenience of NTK calculation and is not the main focus of this paper.
Loss function For classification problem, we choose to apply the commonly-used and representative Cross-Entropy Ioss function, although our proof can be generalized to a class of loss function as mentioned in Remark That
(())
Σ Lys In(o;) + (1 ys) In(1 o:)]
(2.3)
((2g - 1)f(z;;)))
where {o.).-. and function is defined by
1
1 M ((r)==n(11)))
(2.4)
Under such setting, o can be regarded the "output probability". since it transform f(x; 0) and then compare to the label yi- In this way, the network function f(x; 0) is expected to be positive correlated to the probability that the label y of be label 1 instead of O. We use u, denote:
Ui == 10; Vil-
(2.5)
Then we have
if y; = 0; We see u, [0, 1]. Under the Cross Entropy loss function, the gradient descent optimization process. There an simple way to represent the gradient by u. The dynamic euqation is:
02 (1 11-0 if y; = 1.
U1 1 + e(2y-1)f(=;;))
(2.6)
gradient flow approximate
n
_ 0: = VoC(O,) = Σ
((2): - (z),:))))) - 1)
di
1=1
(2.7)
Vol(x;; (2): -1)-- 1)ui-
NEURAL TANGENT KERNEL
We first introduce the concepts of NNK and NTK. Both can be used study the dynamics of neural networks during training and have therefore been widely discussed in previous literature.
Neural Network Kernel The Neural Network kernel (NNK), also called the empirical NTK, is the inner product of the derivatives of the neural network with respect to all parameters. We denote NNK by K((s)) ) IRd Rdd + R... Specifically, the expression is given by
x') = (V, 0t), (=f(x'00,).
(3.1)
In equation 2.7, know the dynamic of parameters during training process. Since f(x; 0.) is function that decided by 8, its dynamic can also be derived by:
d di
00,
f(z; 0,) [Vo 0.)
at
Vos 0z) (os((;;))) (2mr 1)us.
(3.2)
Σ
Ke(2, xx) (2 - 1)44
4

---

Published as a conference paper at ICLR 2025
[of 8.)] (of 0c)] Note that equation 3.2 has similar form to kernel logistic regression. Unlike traditional kernel regression, the kernel function in 3.2 is not fixed during training. Instead, it changed over time.
where (2 24)
Neural Tangent kernel To better understand the dynamic characteristic of network output function, many previous work (Jacot et al., 2018; Allen- et al., 2019; Arora et al., 2019) systematically studied the properties of NNK, and showed that the network width m comes to infinity, NNK will converges to a fixed kernel both at initialization and during the whole training process, which is the so-called Neural Tangent Kernel (NTK):
Ke(2,z)) E. K (x, 2').
(3.3)
Morcover, Lai et al. (2023a); Li et al. (2023) further shows that the convergence from NNK to NTK is uniform over all possible input vector and all time point. This phenomenon helps lot in the learning of generalization ability network under the frame work of NTK theory.
In regression problems, the convergence of NNK is built the convexity of the MSE loss function. However, in classification problems where Cross Entropy is commonly used as the loss function, we show that this convergence longer holds. We will elaborate this point in the subsequent sections, which also reflects the limitations of NTK theory.
3.1 STRICTLY POSITIVE DEFINITENESS NTK
In order to ensure the divergence relationship between NNK and NTK, strictly positive definiteness of the kernel function is erucial. We first explicitly recall the following definition of positive definiteness to avoid potential confusion.
Definition 1 (Strictly positive definiteness). A kernel function ((::-) X'xX' + R is strictly positive definite over domain for any positive integer n and any n different points Z1...., In E X, the (x),,
smallest eigenvalue Amiu of the matrix K (X, X)
is positive.
To derive the positive definiteness of NTK of fully connected network and residual network, we first demonstrate the their expression in the recursive form or explicit form, respectively.
NTK of Fully connected network We first present the recursive formula NTK of fully connected network. For fully connected network given by equation 2.1, the NTK denoted by K can be defined as following as shown in Jacot et al. (2018). We first define
(1, = (x,
(x. x') (0,2") + 1,
(3.4)
and
((41) (I,2') = (0) (z, 15 (z, x') + 10+1) (x,
(3.5)
where
(2, (o
+1),
(3.6)
(0+1) (x,x²) = =0)))so0(((()))0(((
for any / E 1.... L. Then the formula of KFC is
KF (=,''' ((++))
(3.7)
NTK of Residual network We can provide the NTK expression for residual neural network 2.2, based on the the specific expression of NTK under the network structure without bias term which is given by Huang et al. (2020). Let x x' dd be two samples. Introduce the following functions:
1 (Ir arccos u), ku(u)=
1 (u(T arccos u) + v1 - u2
R1(u)
π
and
i == (2,
x(x)=
(3.8)
| 2
5

---

Published as a conference paper at ICLR 2025
The NTK of L hidden layers ResNet, denoted K is given by
K Ros (x, (x,x') |ل||| |
o(z1) (()l=)""'((())))) (00 ((d) + (l(((x),
(3.9)
where
,(4) x (Huang al., 2020) defined
(x, = c Σ B14 a2) (
1
( + (Ki-1 + 1) Ko (n+tast
(3.10)
where
Ki Ki-1 + a² +0?)(=1m Ki Ki-s (atag)=r); Ko = xTx'; x';
Bt Bi+1 1 a²ro Ki (= BL+1=1, Ct = a²
for 1 E [L]. In the above equations, K₁ and Bj are abbreviations for <'(,,2') and Bi (x)
respectively.
For the NTK defined above, we have the following proposition on their positive definiteness, which is proved in Appendix:
Proposition 1 (Strictly positive definiteness of NTK). The NTK of fully connected network 2.1 and residual network 2.2 is strictly positive definite.
The strictly positive definiteness has long been topic of interest in NTK theory (Jacot al., 2018; Zhang et al., 2024; Nguyen et al., 2021). The strictly positive definiteness of NTK of fully connected network defined on the unit sphere g was first proved by Jacot et al. (2018). Recently, Lai et al. (2023a) proved the strictly positive definiteness of NTK for mee-hiddenllayr biased fully connected neural networks on R, and Li al. (2023) generalized it multiple layer fully connected NTK on Rd (d 2 1). In these works, the bias term 6d) is omitted simplification of the network structure. In this work, we adopt a commonly used fully connected neural network with bias terms and use recursive form to prove strietly positive definiteness. As to the NTK of residual network, use equation 3.9 to transform the input vector from Rd to 3 and then prove the strictly positive definiteness of the dot-product kernel ((())
4
MAIN RESULTS
4.1 DIVERGENCE OF NETWORK
At standard NTK initialization, the NNK will converges in probability as width comes to infinity (Arora al., 2019), which is evidently independent of the loss function and training method. In regression problems, NTK is also proved to be convergent during the training process, which forms the theoretical basis for studying the generalization ability neural networks NTK theory (Suh et al., 2021; Li et al., 2023; Lai et al., 2023a). This phenomenon is because the parameters will not deviate far from their initial values no matter how long the network is trained, which is known as the so-called NTK regime (Allen-Zhu et al., 2019). However, it no longer holds when training with the cross-entropy loss function. We denote by (t) as the minimum eigenvalue of the NNK matrix: (o) Ami (X)
Theorem Fix the training samples (1 [e)n Consider fully-connected network and residual
network with cross entropy loss function in classification problem. If Su(() is consistently lower bounded by some postive constant C during training, then the network output function will comes to infinity the sample points { [eEn) i.e.,
lim f(()))) +00.
(4.1)
This theorem reflects the divergence property of network in classification problems. We attempt to understand the theorem above from a less technical perspective and in a simple, nuu-tto-understand manner. If have that the NNK always maintain uniformly strictly positive definiteness, i.e.,

---

Published as a conference paper at ICLR 2025
(t( >>> O, then each input vector 2; in the sample set, the output function value of network will tend to infinity, which means that some parameters of network diverge during the training process as time changes. We demonstrate the following direct corollary of Theorem 1:
Corollary 1 (Failure of the NTK regime). Assume the conditions in Theorem holds. For any initialized parameter 8, after training have that
lim 10, 00lloc = 00
(4.2)
11000
Note that in Theorem 1, we do not impose any requirements on the width of network, meaning that the aforementioned divergence holds regardless of the width m. Of course, we have not forgotten that all of this is based on the condition (t) > C >> 0. This condition will be satisfied if NNK uniformly converges to NTK, which will be used the proof by contradiction of Theorem 2.
4.2 DIVERGENCE OF NTK
In this section, will discuss the divergence of NTK in classification problem and demonstrate main results. In infinite time, the NTK continues to evolve and will not converges to a fixed NTK like in the regression problem case. Now demonstrate Theorem 2 (Divergence of NTK of fully connected network and residual network) ). Given samples {Z,, y1) W= and consider the training process under cross-entropy function, have
main theorem.
Let λo == Amin KR (0,,(XXX)))) For any initialized parameter Og, there exists x, E X, such that
Ao
sup K FC (2,2(1zz)))) -((0)--(M (FC((,))) 2
(4.3)
2n
t20
Let Ào == Amiu (K (X, X)). For any initialized parameter Ub there exists x, E X, such that
Ag
sup (R,ss (x, x²) K (x, (2)l 120
(4.4)
2n
To see the significance of this result, we can compare it with regression problems: In regression, regardless of the value of n. as long as m is sufficiently large, we can achieve uniform convergence of NNK to NTK, then approximate neural networks using kernel regression, and thus obtain properties of generalization ability (Lai et al., 2023a; Li et al., 2024). However, in classification problems, NNK cannot uniformly converge to NTK. Although when is sufficiently large, the lower bound of the gap between NNK and NTK, 2n is small, this small lower bound still makes it difficult for us to approximate neural networks using kernel regression. The theorem indicates that for both fully-connected and residual network, the NTK does not remain fixed during training when using the cross-entropy loss function. Instead, it continues to evolve, suggesting that the NTK theory, which works well for regression problems, does not directly apply to classification problems. This divergence implies that the NTK classification tasks cannot provide a static approximation of the training dynamics, thereby necessitating new approaches understand and analyze the behavior of neural network.
Aa
5 PROOF SKETCH
We present the sketch of our proofs in this part and defer the complete proof to Appendix. In this section, we denote the network output function fNN : X' C T - R and denote the corresponding NNK and NTK as (.'(.,.) and K(:-). respectively. We also fix samples [(r,, [ej)) To simplify the notation, in the proof sketch we do not deliberately distinguish the symbols for different network structures. Namely, the above symbols represent both fully connected neural networks and residual neural networks. We assume the training samples of the neural network trained using the cross-entropy loss function.
We can finish the proof of Theorem 2 by contradiction. For the sake of contradiction, we firstly assume that during training, NNK is invariant:
sup sup|KT"(z, K(z, x')1 om (l).
(5.1)
2, 120
7

---

Published as a conference paper at ICLR 2025
This implies that also have the relationship between the minimum eigenvalues
sup/Amin (x, Amia (K(X, xx)] - sup |X," (X, K(XX, 0m (1). (5.2) :0:
t20
Recall that the strictly positiveness of NTK has been demonstrated in Section 3.1, we know NNK is also consistently strictly positive definite during training. Namely, there exists a positive constant C
such that
inf Amin (KT (X, 120
(5.3)
Since NNK involves the derivatives of the network with respect to the parameters, it can be used to analyze the dynamics of the network function during training. In equation 2.5 we define u E R" which represents the residual and also reflects the output value of the network function on the training set. We construct
V(u)= = Σ In(1 (()) and V = V(((t)
(5.4)
For the scalar Vi, we have the dynamical equation
d di V2 = -u" -i==="(((diaanl (t)diag(2Y - -1), )ddagg22 - 1)t(t).
(5.5)
Since NNK is strictly positive definite in equation 5.3, we know that V3 is monotonically decreasing. Through some technical analysis (details in the appendix), we finally get V. - 0 as co. Therefore, as L + oc, we have
(t) - O, and (x:) + co.
(5.6)
Combined with the specific network structure, we can derive that
lim 11000
sup (2:)
(x, oo,
(5.7)
which indicates that NNK diverges during training. Since the assumption of uniform convergence of NNK to NTK leads to NTK being bounded, this results in contradiction, and we finish the proof.
6 NUMERICAL EXPERIMENTS
6.1
SYNTHETIC DATA
We conducted experiments on a synthetic dataset, using the previously mentioned fully connected network and residual network architectures.
Divergence of the Fully Connected Network Function The fully connected network has three hidden layers, with an input dimension of d = 2 and a width of m = 2000. For the training set, we generate input vectors uniformly distributed on the unit sphere Sd. i.e., the points far]eejo are (cos 0g, sin 0.), where 0, . The labels of the points are (0, 1, 0, 1, 0, 1), respectively. We shuffle the labels to eliminate any potential influence of data separability. The learning rate is set to 0.1, and the network is trained for 10, 000 epochs. The output values of the network at the 6 training points are plotted during training, and the final result is shown in Figure 1.
Divergence of the NNK during training In this section, we examine the behavior of the NTK both at initialization and during training to highlight the impact of the cross-entropy loss function. At initialization, we plot the NTK function of network with different width by fixing = (1, 0) and varying the polar angle x' from -- to म. The result is shown in in Figure 2, which demonstrates 2000. The results show that the NNK
that the NTK converges at t = 0. particularly when approximately converges to the NTK, aligning with previous findings in NTK theory. We then 2000 using the cross-entropy loss function and plot the
train the network with width of network's output at the same points. Figure presents the behavior of the NTK across different epochs, highlighting its divergence during the training process.
8

---

Published as a conference paper at ICLR 2025
Pont
Pont
nt 4 Po
Network Output Value
loga0 Epoch
Figure I: Divergence of the fully connected network: The plot shows the output values of the network at the 6 training points over the training process. Despite starting from a shuffled label configuration, the network function diverges at all six points training progresses.
NNK ot. 200
NNK am 0,m 1000
NNK at 0, - 2000
50
3.0
NNK Voluo
20
1.5
2.0
1.3
1.5
-3
-1
-3
-2
-3
PolarAngle ot Irput
PolarAngle of mput
Poto Angle Input
Figure 2: Convergence of the fully connected NTK: The NTK functions initialization for networks with different widths (m = 200, m 1000, and m = 2000). The results show convergence of the NTK 0, with the convergence becoming more apparent as the network width increases. especially when = 2000.
Epoch 0
Epuch 1000d
NNK Value
Polar Angle Input
Figure Divergence of the fully connected NTK: The behavior of the NTK at different epochs during training for network with width m 2000. The plot highlights the divergence of NTK over course of training.

---

Published as a conference paper at ICLR 2025
6.2 REAL DATA
We conducted an experiment on the MNIST dataset, using parity (odd or even) as the criterion for binary classification. We trained four-layer fully connected neural network the MNIST dataset. The network has a width of m = 500, with a learning rate of Ir 0.5, and was trained for epoch 100, 000. Since the dimension of MNIST dataset of NTK like in Figure 4. Therefore, we pick three points and show that the empirical NTK value disconverges on these points, which is sufficient confirm our results. The result is shown in Figure 4. The values of the first 10 epochs are discarded here in order to focus more on the behavior during training and avoid the experimental results being affected by initialization.
784, is hard plot the value
Cabeec hamole:
Figure 4: The NTK values the MNIST dataset for three selected points. The blue lines represent the original NTK values, while the red dashed lines show the smoothed values. The NTK values computed over 100, 000 epochs for four-layer fully connected neural network with width of m = 500 and learning rate Ir = 0.5. These points were chosen to illustrate the disconvergence of the empirical NTK values.
7 CONCLUSION
In this paper, we investigated the behavior of NTK of fully connected network and residual network in classification problems, particularly focusing on the convergence properties when training with the cross-entropy loss function. Our study reveals that unlike in regression problems, where NTK convergence has been well-documented, NTK in classification tasks does not uniformly converge and instead diverges as training progresses. This divergence indicates limitation in applying NTK theory to classification problems and suggests that NTK does not provide a static approximation of the training dynamics in these scenarios.
Our analysis highlights the need for new theoretical tools and frameworks to better understand and analyze neural network in the context of classification. We have shown that the standard NTK regime fails in classification problems, which implies that current theories might not fully explain the generalization properties of network trained such tasks. This opens up new avenues for research in understanding the complex dynamics of neural network beyond the NTK framework, especially when dealing with classification problems where cross-entropy loss is prevalent.
ACKNOWLEDGMENTS
All three authors of this paper have either participated in discussions at currently studying in the Department Statistics and Data Science at Tsinghua University. We would like to express our advisor, and fellow students for their guidance, training, and
sincere gratitude to the institution, valuable exchanges. We also extend our thanks to the AI Data Technology Laboratory of Huawei Noah's 's Ark Lab for kind suggestions and support.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
Benedikt Bauer and Michael Kohler. On deep learning as remedy for the curse of dimensionality in nonparametrie regression. 2019.
Guhan Chen, Yicheng Li, and Qian Lin. On the impacts of the random initialization in the neural tangent kernel theory. arXiv preprint arXiv:2410.05626, 2024.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on learning theory, pp. 698-6288 728.. PMLR, 2016.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4))303-315 1989.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675-1685 PMLR, 2019.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint 110:0:1810 2018
Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv: 1710.1 1278 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 770-778, 2016.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nngp and ntk for deep attention networks. In International Conference Machine Learning, pp. 4376-4386. PMLR, 2020.
Tianyang Hu, Wenjia Wang, Cong Lin, and Guang Cheng. Regularization matters: A nonparamet- ric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics, pp. 829 837. PMLR, 2021.
Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks? neural tangent kernel perspective. Advances in neural information processing systems, 33:2698-2177 27099,2799710 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and
generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. arXiv preprint arXiv: 1209, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 13Xiv:1312.611 2013.
arXiv preprint
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.

---

Published as a conference paper at ICLR 2025
Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks arXiv preprint arXiv:2302.0593. 2023a.
Jianfa Lai, Zixiong Yu, Songtao Tian, and Qian Lin. Generalization ability of wide residual networks. arXiv preprint arXiv:2305.18500 2023b.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. Statistical optimality of deep wide neural networks. arXiv preprint arXiv:2305.026577 2023.
Yicheng Li, Weiye Gan, Zuoqiang Shi, and Qian Lin. Generalization eroor curves for analytic spectral algorithms under power-law decay. arXiv preprint arXiv:2401.01555 2024.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhigiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. Advances neural information processing systems, 30, 2017.
Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning, pp. 1119. PMLR, 2021.
Johannes Schmidt-Hieber Nonparametric regression using deep neural networks with relu activation function. 2020.
Namjoon Suh, Hyunouk Ko, and Xiaoming Huo. A non-parametric regression viewpoint: Generaliza- tion of overparametrized deep relu network under noisy observations. In International Conference Learning Representations, 2021.
Hossein Taheri and Christos Thrampoulidis. Generalization and stability of interpolating neural
networks with minimal width. Journal of Machine Learning Research, 25(156):1-41 2024.
Songtao Tian and Zixiong Yu. Improve generalization ability of deep wide residual network with suitable scaling factor. arXiv preprint arXiv:2403.04545 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding generalization in deep learning. arXiv preprint 2rXiv:2206.10112 10012, 2022.
Shao-Qun Zhang, Zong-Yi Yi Chen, Yong-Ming Tian, and Xun Lu. A unified kernel for neural network learning. arXiv preprint 2rXiv:2403.17467, 2024.
12

---

Published as a conference paper at ICLR 2025
A FURTHER NOTATIONS
We first demonstrate the definition of Gaussian process.
Definition 2. A stochastic process (z), E R is called a Gaussian process if for any finite set of points 21, T2, ... In in Rd the random vector (f(x)), (x2)..... f(rn) follows multivariate normal distribution. Specifically, Gaussian process f is completely specified by its mean function m(x) = [x)) and covariance function K (xrxx') = [ and is denoted as:
f ~ (P(m())) ((x, x").
(() = 0, the process is called centred Gaussian process.
For the notational simplicity, we denote a centred Gaussian process GP(O, K) directly by f K.
B THE STRICTLY POSITIVE DEFINITENESS OF NTK
In this section, we demonstrate the proof of Proposition 1. In the following two subsections, we respectively prove the strictly positive definiteness of the NTK for fully connected network and residual network. For the sake of writing and referencing convenience, we have divided Proposition into Proposition? and Proposition
B.1 FULLY CONNECTED NETWORK
The proof of the strictly positive definiteness of NTK roughly similar to that in Jacot et al. (2018). However, since our network structure includes several bias terms, which is different from Jacot et al. (2018), we provide the whole proof here to ensure completeness.
Proposition 2. The NTK of fully connected network 2.1 strictly positive definite on X.
Proof of Proposition 2. Recall that the Hadamard product of positive definite matrices is still positive definite, and the sum of positive definite matrices is also still positive definite. By the recursive form of NTK, the result following from Lemma 4.
The following lemma is provided to ensure the recursive proof between layers can proceed.
Lemma 1. If kernel function K1 Rd x Rd - TR is strictly positive definite, then the kernel function K₂ defined by K2(x, x') = E (f()))) (f(x))) ]] is also strictly positive definite.
Proof. We prove the lemma by contradiction. Suppose that there exists set of {2;}i=ı... C Rd such that the Gram kernel matrix satisfies that K2(X, X) is strictly positive definite. Firstly, it is direct to verify that K2(X X, x) is positive definite. For any E IR", we have
u K2(X, X)u Ey~K, ( (((())) 0.
(B.1)
Since we suppose that (2(,, X) is not strictly positive definite, it means that there exists a vector Md E R and un * 0 such that up" (X,XX))uu X)uu 0. Thus we have
w K2(X,, X X) = Ey~K, 11 (f())))
(B.2)
namely, of (f())) = O holds almost surely. However, f(X) is n- Gaussian variable with strictly positive definite Covariance matrix, and thus is non-degenerate. We define g(v) = E (g((), where un, is the i-th entry of up. Without loss of generality, we assume that von # 0. If of (x())) 0 holds almost surely,
have g(v) 0 almost everywhere with respect to Lebesgue measure. We fix vector t E R and define the new function by gn (v,)) = g((w=, )) En= ( ( (v.) Since is non-constant, we have my (Aw) 0 where my denotes the d-ddimensional Lebesgue measure and A = [v,19n. (v,) * 0}. Then m., (v)g(v) > 0}) = -Jx fgn-1 (vnn An ddddo, 0 Therefore, prove the lemma by contradiction.
n
13

---

Published as a conference paper at ICLR 2025
Lemma 2. Let f [[-1,1] - R be continuous function with the expansion
f(u)
Σ anu" 7=
Wus 1E1-1,11]
(B.3)
and k(x, 1)) f(x,, y) be the dot-product kernel on Sd. Then an for all n Z 0 and there are
infinitely many an > O, then k is strictly positive definite on $4 == {[2) 22, ... ,t) E S [. > 0}.
Proof. Let 21. 2n $4 be different. The kernel matrix is
x)x =
Ma.
Mn = (112,
(B.4)
==
Since ((x,,y)) 1, we have
Mn - Ins
(B.5)
which is strictly positive definite. In this way, we finish the proof.
Lemma The coefficients of Maclaurin expansion of ko(u), KY ku( are both non-negative and
infinitely many terms are positive.
Proof. A direct calculation leads to that
80
(2n)
ko(u)
2n+1
(B.6)
4th
(n!)2(2n +
and
K2 (1)
π
2
(2n) !
2n+1
+1+
2 (-1)m-1(211) (2n)!
("(n))?(2n - 1) (-1)
π
2
4th (n!)² (2n + 1)
n=0
n=1
1
1
(2n)!
u 2n+2
(B.7)
2m
(n(n)) + 1)! 2n +1
Lemma 4. The kernel function (2 is strictly positive definite on X.
Proof. We do transformation from Rdd to Sd to finish the proof. For Rd., we define 3 = (z. 1), which means that add an entry 1 to the end of the vector æ. Define Φ: Rd - s by d(x) |
we define a kernel function K₁ (1x x²) = (ar x²) on $ S and a kernel function K₂ (ar x²) Ef-k, (f = fx1 ((x, 2")),)), then K2 is strictly positive definite on S by Lemma and Lemma 2. Actually, then have that
(x,) x') = 2 || (2(((x), s(x 1=1
(B.8)
Therefore, we can directly verify that (2) is strictly positive definite on X.
B.2 RESIDUAL NETWORK
For residual neural networks, since only the first layer in the standard residual network structure includes bias term, we transform the original kernel into an dot-product kernel on the sphere and then take a Maclaurin expansion to complete the proof.
Proposition 3. The NTK of residual network eg. (2.2) is strictly positive definite on X.
Proof. Note that in Lemma 4, we accomplish the proof through a transformation from Rd - S We can also define the NTK of ResNet in similar mannar. It is well known that the sum of positive definite kernel and a strictly positive definite kernel is still strictly positive definite. Therefore, to derive the strictly positive definiteness of K definiteness of K (x.(..) on sd, which is shown in Lemma
only need to consider the strictly positive
14

---

Published as a conference paper at ICLR 2025
Lemma 5. ( is strictly positive definite on $4 when L 2 2.
Now we start prove Lemma We first prove that Lemma 6. (i(--)) is strictly positive definite on $4.
Proof. We have known that 1 is dot-product kernel. If we denote by u = xTx Then we have Ki x') = u + (²).
(B.9)
Thus (i(-,) is strictly positive definite on 8 through Lemma 2 and Lemma 3.
Then one can easily prove Lemma 5 recursively based on the result of Lemma 6.
C DIVERGENCE OF NNK UNDER INFINITE TIME
C.I THE DYNAMIC OF u
Before the characteristic of NNK, we first demonstrate the dynamic property of the auxiliary variable
defined in 2.5, which pivotal in our analysis. For (21, "2. un 12" we have the dynamic equation:
-e - (3fe(
(2y: - 1) d dt f((x)
dt (t)
= [1 + el2ut (1)(())))
(C.1)
=-u (1 uz) Σ (2y) K),, 1)ug.
we define the matrix
KT = 1) (z,, 23 (2, -1)
d2i - 1) (X,, X)diag(2Y 1)
(C.2)
=[K], K, K,, K.)
then derive a more explicit dynamic equation
d dt
(t()) - 21. [K]]
(C.3)
is worth noting that K" is also a symmetric matrix as K,(X,X)
C.2 THE EXPLICIT FORMULA OF NNK
In this subsection, we introduce the explicit formula of NNK of the fully connected network.
We denote by D 1(W()) + b0) 0), such that have
(x) = V d (0000010 + (00)
(C.4)
Then the explicit formula of NNK of multi-layer fully connected network is
L+1
K2 x') =
Σ
(wwo (x))(()) + Veo Vnof(x))
(C.5)
15

---

Published as a conference paper at ICLR 2025
where the explicit formula is
VV fw))(x), vwof(z))
L+1
w(-D((
a (x))
II
2
w ((
(r-1)
a
(1(-))
m4-1
my-1
r=++1
L+1
L+1
2
D(r1))
a (x),
w(r)
((r-))
a()))(+'))
m
m,-1
r=++1
+ (x)
(C.6)
and
L+1
bmm (xx), Vonn
II Lr=l+1
2 m,-1
w(r)
((r-))
2
WirDD(r- ((r-1)
(r-1)
my-1
(C.7)
C.3 DIVERGENCE OF NNK UNDER INFINITE TIME
We first introduce new notations. By Proposition 1, we know the NTK is positive definite and thus the kernel matrix K (X, X) is positive definite for any samples X. Denote Xo == Amin [K (XX, X)) to be the minimal cigenvalue of NTK Gram matrix. Also, Let An(t) = Amiu (X,
Fully Connected Network. We proof the theorem by contradiction. We first assume that there a kind of parameter initialization such that
Proof of Theorem 2. Part 1:
Ao
sup (2(,, - KFF KFC 120
(x1, (g), - 2n
(C.8)
holds for any re, E X. Proposition ensures that Ao 0. Through Lemma we have
Ao > inf (t) 2 120
(C.9)
2
By Lemma 8, we have (2y; - 1)fe((;; + too as t + oc.
(L) Recall that fz(2;) w (2) (z,)) = m2 W (1) (a:), where ajt (a:) and W w (xx) denotes the j-th entry of a (1) ai
(x;) and (((++)(
(zx)) Therefore, there exists an index jo [mr.], such jo.t ( (x:) is infinity, as time t comes to infinity.
that the limit superior of w (L+1) (L)
(L) We first consider the case that the limit superior of jo. (z.) is infinity. In equation C.5, directly we
can see,
V win ((), V Vwn 2 (a(l) (z)
(C.10)
oft (xx)a
(L) (D) (1) (x.) 2 20 (1. +0.
yields the result that K2 (2(2,, diverges to infinity, which contradicts to the assumption in equation C.8.
16

---

Published as a conference paper at ICLR 2025
We then consider the other that the limit superior of W (L+1) just
( (a
(1) 0) is infinity. To
work on this, we focus on the bias term in the L-th layer, i... 6(4)
K((1,)) 2 oun ban f(x), Vocs 2 bes f(z), Vouf(z))
<[VV 2 w((+1) ( (2)
[ 2 w(++1) D
(C.11)
2 (wiltr") Jont
(z,) Ju.t > 0)) oc.
mL
It also contradicts to the assumption equation C.8. Therefore, finish the proof.
Part Residual Network. The proof is also accomplished through contradiction. We conduct it in similar mannar in the case of fully connected network. We first assume that there is kind of parameter initialization such that
Ao
sup (e(T,, - Res (Ii xr)) S
(C.
2n
120
holds for any 2y X. Proposition ensures that Ao >> 0. Through Lemma 7 we have
Ao 2
inf (t 120
(C.13)
By Lemma 8, have (2y: (z.) + +0 co. Recall the structure in equation 2.2 of residual network, the network output function f(x; 0) can be actually decomposed as f(x; 0) (((+)) a (x)
L
(Ar + b)
Σ
(vWl
122 (0 + +b00) + )(0)
mo
(C.14)
Since for any in training set, we have either of the limit superior. Namely, we have either
| /|
00 or ( (x:²
+ oc in the sense
lim sup 10000
W (L+1) oo, or
lim t0000
a (z) =0.
(C.15)
the former case holds, we have
Of((z)) (f{(z;)
((L+1)
(C.16)
dd(L) dd(L)
m.
in the sense of the limit superior. And thus the NNK will not converges to the NTK.
If the latter case holds, also have
0f((z.) 0f((z.) @W(L+1) OW(++))
- -1(
20,
(C.17)
in the sense of the limit superior.
Therefore, the theorem is proved through contradiction.
Lemma 7. Consider fully connected network 2.1 and residual network 2.2. Suppose the NNK uniformly converges to the NTK over the samples, as the network width m comes to infinity, namely,
A
sup sup [K((z, x1) K (214, 120 ije(n]
(C.18)
2n
then postulate a lower bound of No:
Ao
inf (o( 2
(C.19)
20
2
17

---

Published as a conference paper at ICLR 2025
Proof.
[No(t) Xpl - | X) XXI, 5 S| XXI
A (((141 (((,),))))KK, 2
(C.20)
Thus the lemma is proved.
Lemma 8. Consider fully connected network 2.1 and residual network 2.2. If we have a positive constant lower bound C of >n(1) during training, then the network networkfunction will comes to infinity at the sample points {ar (eel)) Le.,
lim 1 +00. +00
(C.21)
Proof. Since K diag(2Y - X))aaggX)) 1)-1, we know K share the same cigenvalues as Kr(X, X). Therefore, we have Amin (KT) )a(t). Define function V(r) - In(1 rz). where (21, ,TT)" E [0, 1)" Apparently, have V(r) 2 0. Given the dynamic of equation C.3, we can derive the dynamic of V(u):
dt V(u) = Σ
OV du,
Dr, r dt
dui
(C.22)
u,
u [K] K"u 0.
According the monotone convergence theorem, we know V2 V (u) converges as co. We can also prove that the limit of Vú is exact zero. We give the proof by contradiction.
First, we assume that ooo Vn V, > holds. Then for any > there exists 4 0, such that for any t > to. we have V, I Vi - V, + holds. We define the following set:
Ve r3 = [0, e e (V. -m [0, - e
)
(C.23)
max Ti S (V-tee) = = 1) - i= 111,,.-
E
We can verify that VVV,,, V, + C Pg. Therefore, we have u E I' when t to. Given that I'g is a compact set, we define M minrer, ||r|| > O. When E Pe, we can derive that
K" r 2 Amiu (K") ||||1 Amin (K") 2 MC 0.
(C.24)
then there is contradiction that
d di
V(u) - --u KMMC
11 s Vn + 1' -MCdt
(C.25)
10
V. - MC(1 - (o) + -00.
Thus, we prove that the limit of V, is zero. Based on the limit of V,, we can also get the convergence result of u;:
(uu) - 0
n
(C.26)
*< 4 -nn11 ui)
In u Vw)) 0.
Therefore, we have lim,-+ (2y; - (x;) = too as the time comes to infinity holds for any i [n]. In this way, finish the proof.
18

---

Published as a conference paper at ICLR 2025
Remark 1. The result of Lemma 8 can be extended to general loss functions. Specifically, the influence of the loss function is reflected in equation (C.22). Let Let((x) be a convex loss function with (x) < O. which appears in training as ((2y; 1)f(z;)) Under these conditions, the proof of Lemma 8 still holds. We only need to define
al (2y - 1x)( 0f(x:)
and set V(u(-)) = I(-) as replacemment. This demonstrates that our proof is general under these assumptions.
19


Reviews:
[{"review_id": "6l4k59yYyG", "reviewer": "Reviewer_wRPu", "summary": "The paper considers the convergence of the neural tangent kernel of fully-connected ReLU networks and ResNets for classification problems trained with cross-entropy loss functions.  The key result of the paper is a proof that, during training (in the gradient-flow approximation), the network parameters diverge due to the strict positive definiteness of the NTK, implying that many results from NTK theory may not be applicable in this context.", "questions": "See previous sections.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The result appears to be sound and certainly has relevance insofar as it demonstrates the non-applicability of many NTK results in a common scenario (training with cross-entropy).  The presentation is mostly readable and the derivation appears to be correct, though admittedly I did not dive too deeply into the appendices, and certainly the material appears novel (at least to me).", "weaknesses": "While I don't question to novelty of the result in literature, I don't necessarily find it too surprising.  To truly minimize the cross-entropy would require $f(x_i,\\theta) \\to +\\infty$ for $y_i=+1$ (and similarly for the other class).  Without regularization and given infinite time, recalling that the NN is a universal approximator in the infinite-width limit, you should expect precisely this, in which light theorem 1 (and subsequently corollary 1 and theorem 2) is to be expected.  Nevertheless it is important to see this intuition proven formally, so I do not consider this a serious criticism.\n\nMinor points - the presentation of the paper could certainly be improved, for example:\n\n- line 163: you don't optimize the parameters with gradient flow.  Gradient flow is used to (approximately) analyze the optimization process.\n- line 185: this is badly written.  You already defined $K_t$ in (3.1) - why are you repeating it in (3.3) as though it was new?\n- line 283: \"...will converges to NTK with probability as width comes to infinity...\" Do you mean \"will converge with probability 1\" or \"will converge in probability\"?\n- Theorem 2: (4.4) is precisely equivalent to (4.3).  Are these meant to refer to different networks (ie should (4.4) be modified to refer to ResNet)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWvkAQaBKY", "reviewer": "Reviewer_Nv3G", "summary": "This paper investigates the divergent properties of neural network kernels (NNK, empirical NTK) in classification problems using the softmax loss. Specifically, it demonstrates that the NTKs of fully connected neural networks and ResNets are strictly positive, and that model outputs and parameters will diverge as long as the NNK is bounded below by a positive constant during training (Theorem 1 and Corollary 1). Additionally, it is shown that, unlike in regression problems, the NNK is not fixed during training (Theorem 2).", "questions": "- The obtained results rely on the positivity of the NNK. Could you clarify the required conditions (e.g., number of neurons) for this assumption?\n\n- Could you discuss the relationships with the related papers mentioned in the weaknesses?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proves the divergence of NNK during training, indicating the inapplicability of NTK theory for regression problems.\n- The paper is well-structured and easy to read.", "weaknesses": "- The divergence result of the NNK is not particularly surprising, as it is evident that parameters and outputs must diverge to minimize standard classification losses such as logistic, softmax, or exponential losses. For example, refer to the following papers: \n\n[D. Soudry, et al] THE IMPLICIT BIAS OF GRADIENT DESCENT ON SEPARABLE DATA. ICLR 2018.\n\n[Z. Ji and M. Telgarsky] The implicit bias of gradient descent on nonseparable data. COLT, 2019.\n\n- Given the scale of $\\lambda_0$, which typically degenerates as $n\\to \\infty$, I am curious about the significance of the deviation $\\lambda_0/2n^2$, which goes to $0$ as well.\n\n- The results do not address the convergence of gradient descent. If the positivity of the NNK (Eq. (5.3)) holds, convergence can be shown as in NTK theory, but this requirement seems redundant for classification problems. The NTK separability assumption, a weaker condition than positivity, is sufficient for proving the convergence of gradient descent. See the following papers for details:\n\n[A. Nitanda, G. Chinot, and T. Suzuki] Gradient descent can learn less over-parameterized two-layer neural networks on classification problems. 2019.\n\n[Z. Ji and M. Telgarsky] Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. ICLR 2020.\n\n[H. Taheri and C. Thrampoulidis] Generalization and Stability of Interpolating Neural Networks with Minimal Width. JMLR 2024.\n\n- These related works are not discussed in the paper. \n\n- (Minor) \nA few typos: (1) Line 110: $f(x_1, f(x_2),$ (2) In the second statement of Theorem 2: Should this be the result for $K_t^{Res}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RTfIFF3kol", "reviewer": "Reviewer_rp1n", "summary": "This paper explores the convergence properties of the Neural Tangent Kernel (NTK) in classification tasks, with a focus on multi-layer fully connected neural networks and residual neural networks. The authors establish that the NTK of both types of networks is strictly positive definite, a key characteristic for comprehending the training behavior of neural networks. Using a contradiction argument, the authors demonstrate that when employing the cross-entropy loss function, the parameters of the neural network tend to diverge due to the strictly positive definiteness of the NTK. The results imply that NTK theory may not hold in the context of classification problems, leading to significant theoretical implications for understanding the dynamics of neural networks.", "questions": "1. In Theorem 1, it is assumed that the minimum eigenvalue of the NNK matrix has a lower bound. However, the eigenvalues of the NNK Gram matrix approach zero as the number of training examples increases, as demonstrated in the two papers below. Can this observation offer any insights into NTK theory for classification problems?\n\n[1] Lili Su and Pengkun Yang. On learning over-parameterized neural networks: A functional approximation perspective. In Advances in Neural Information Processing Systems, pp. 2637–2646, 2019.\n\n[2] Atsushi Nitanda and Suzuki Taiji. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. In International Conference on Learning Representations, 2021.\n\n2. Could the scope be broadened to incorporate other classification loss functions in order to gain a more comprehensive understanding of NTK behavior?\n\n3. What will happen if the data is completely separable or follows a Gaussian mixture in classification problems?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper provides valuable insights into the behavior of the NTK in classification problems, highlighting a significant divergence from established understandings in regression tasks.\n\n2.\tBy demonstrating that NTK does not uniformly converge in classification scenarios, the authors identify a critical limitation of NTK theory.", "weaknesses": "1. The claim that current NTK theories might not fully explain generalization properties in classification tasks could be seen as an overgeneralization without comprehensive evidence. A more nuanced discussion on the conditions under which NTK theory might still apply could add depth to the argument.\n\n2. Although the paper recognizes the need for new theoretical tools, it doesn't provide specific directions or approaches for developing these frameworks. Offering some initial ideas or suggestions could help guide future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qi4u497cbY", "reviewer": "Reviewer_jqsc", "summary": "This paper showed that the multi-layer fully connected neural networks and residual neural networks satisfy the NTK in classification problems. Based on this building, this paper shows the positive definiteness and the convergence of NTK.", "questions": "Nothing.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This paper is easy to follow.", "weaknesses": "The main contributions of this paper are trivial since the derivation in Eqs. (2.7)-(3.2) is common-used. Besides, the finding that the empirical NTK does not consistently converge but instead diverges as time approaches infinity has been observed by existing studies.\n\n\n----\n\nI am very sorry that I failed to upload the complete review comments due to my negligence. I would like to add now:\n\nThe main claim of this paper is that the neural network parameters will diverge due to the strict positive definiteness of NTK. This is an interpretation, and the core technology behind it is the estimation of strictly positive definite NTK eigenvalues. This is an existing conclusion [1-2]. Therefore, the technical contribution of this paper is not novel, but more like a reinterpretation.\n\nWhen people talk about a new interpretation of a formula, they should consider what the new interpretation means. I found that there is a statement, \"this finding implies that NTK theory is not applicable in this context, highlighting significant theoretical implications for the study of neural networks in classification problems.\" Unfortunately, this paper does not provide further analysis.\n\n[1] Q. Nguyen, M. Mondelli, and G. Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In Proceedings of the 38th International Conference on Machine Learning, pages 8119–8129, 2021. [2] Shao-Qun Zhang, Zong-Yi Chen, Yong-Ming Tian, ​​Xun Lu. A Unified Kernel for Neural Network Learning. arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)