Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks
Authors: Chaoyue Liu
Han Bi
Like Hui
Xiao Liu
Venue: NeurIPS 2025 poster ()
Keywords: Activation function
condition number
NTK
wide neural network
Primary area: theory
TL;DR: we indentified a new and interesting property of nonlinear activations: better feature separation for similar inputs and better NTK conditioning

Abstract: Nonlinear activation functions are widely recognized for enhancing the expressivity of neural networks, which is the primary reason for their widespread implementation. In this work, we focus on ReLU activation and reveal a novel and intriguing property of nonlinear activations. By comparing enabling and disabling the nonlinear activations in the neural network, we demonstrate their specific effects on wide neural networks: (a) *better feature separation*, i.e.,  a larger angle separation for similar data in the feature space of model gradient, and (b) *better NTK conditioning*, i.e.,  a smaller condition number of neural tangent kernel (NTK). Furthermore, we show that the network depth (i.e., with more nonlinear activation operations) further amplifies these effects; in addition, in the infinite-width-then-depth limit, all data are equally separated with a fixed angle in the model gradient feature space, regardless of how similar they are originally in the input space. 
   Note that, without the nonlinear activation, i.e., in a linear neural network, the data separation remains the same as for the original inputs
   and NTK condition number is equivalent to the Gram matrix, regardless of the network depth. Due to the close connection between NTK condition number and convergence theories, our results imply that nonlinear activation
  helps to improve the worst-case convergence rates of gradient based methods.

Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks
Chaoyue Liu
Elmore Family School of Electrical and Computer Engineering
Purdue University cyliu@purdue.ed
Han Bi
Like Hui
Department of Physics and Astronomy Purdue University bi bi530purdue... edu
Halıcoolu Data Science Institute
University of California San Diego 1hui mrii@ucsd.com edu
Xiao Liu
Rosen Center For Advanced Computing
Purdue University
1iu4201@purdue edd
Abstract
Nonlinear activation functions are widely recognized for enhancing the expressivity of neural networks, which the primary reason for their widespread implementa- tion. In this work, we focus on ReLU activation and reveal a novel and intriguing property of nonlinear activations. By comparing enabling and disabling the nonlin- ear activations in the neural network, we demonstrate their specific effects on wide neural networks: (a) better feature separation, i.e., larger angle separation for similar data in the feature space of model gradient, and (b) better N7K condition- ing, i.c., a smaller condition number neural tangent kernel (NTK). Furthermore, we show that the network depth (i.e.. with more nonlinear activation operations) further amplifies these effects; in addition, the all data equally separated with fixed angle in the model gradient feature space, regardless of how similar they are originally in the input space. Note that, without the nonlinear activation, i.e., in linear neural network, the data separation remains the same as for the original inputs and NTK condition number is equivalent to the Gram matrix, regardless of the network depth. Due the close connection between NTK condition number and convergence theories, our results imply that nonlinear activation helps to improve the worst-case convergence rates of gradient based methods.
seree--wwtthhhhe-epeee limit,
Introduction
is well known that nonlinear activation functions increase the expressivity of neural networks, which is the primary reason of their widespread implementation. A nonlinearly activated neural network can approximate any continuous function to arbitrary precision, as long as there are enough neurons in linear neural network, the network reduces
in the hidden layers [13 [2], while without to linear models of the input. In addition, deeper neural networks, which have more nonlinearly activated layers, have exponentially greater expressivity than shallower ones [32
28
29
22
34
indicating that the network depth promotes the power of nonlinear activation functions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this paper, we reveal novel and interesting effect of nonlinear activations that has not been previ- ously noticed, despite their widespread application: the nonlinearity leads to larger data separation in the feature space of model gradient, and helps to decrease the condition number of neural tangent kernel (NTK). We also show that the depth of the network further amplifies these effects, namely, a deeper neural network has better feature separation and smaller NTK condition number, than a shallower one. While distinct and independent from the property of increasing expressivity, this property of nonlinear activations resembles the former in the manner that the effects vanish in the absence of nonlinear activations: removing the nonlinear activations in a neural network, the data separation and NTK condition number reduce to values observed in linear model. Hence, the effect purely attributes to the presence of nonlinear activations.
Specifically, we first show the better separation phenomenon, i... improved separation for similar data in the model gradient feature space. We prove that, for a wide ReLUU network f. any pair of data input vectors x and z that have similar directions (i.e., small but non-zero angle 0in between and z) become more directionally separated in the model gradient space (i.e., model gradient angle between V (x) and V fz) is larger than Onn) with high probability of random initialization. We also find that deeper ReLU networks result in even better feature separation, i.e., larger S. Ultimately, in the sinnit--wiith--th--deplpl limit, all data are equally separated with angle 75.5 in the model gradient feature space, regardless of the input angle fin as long as 6,n. is non-zero. Numerical simulation also show that the better separation phenomenon generalizes to other commonly used nonlinear activation functions, including GeLU, tanh, etc.
We further show the better NTK conditioning property of nonlinear activation, i.e., smaller NTK condition number. We prove that, as a consequence of the better feature separation, the NTK condition number of a wide ReLU network is strictly smaller than that without the nonlinearity. when the training dataset is not degenerate (i.e., no pair training inputs are parallel). Moreover, with larger depth, the NTK condition number becomes smaller. The intuition is that, if there exists a pair of similar inputs x and z in the training set (i.e., the angle between x and z is small), which is usually the case for large datasets, then NTK of linear neural networks must have close-to-zero smallest eigenvalues, resulting in extremely large NTK condition numbers. The activation makes these similar data more separated, hence it helps to increase the smallest eigenvalues of NTK, which in turn leads to smaller NTK condition number. We further show that, in the infinite- seeee-wiitthhhhh then--ehh---hhh limit, the NTK condition number of ReLU network converges to fixed number m44 which is independent of the data distribution and much smaller than typical NTK condition numbers.
Connection with optimization theory. While there could be multiple implications of the above property in various aspects, here we present its connection with existing optimization theories. Recent optimization theories showed that the NTK condition number k, or the smallest eigenvalue of NTK, controls the theoretical convergence rate of gradient descent algorithms on wide neural networks [ 20). Combined with these theories, our findings imply that: (a), the activation function has the effect of improving the corst-case convergence rate of gradient descent, and (b), deeper wide ReLU networks have faster convergence rate than shallower ones. Previous works often focus accelerating convergence via a better function of x. while assuming x is given and fixed, Our findings provide different perspective of achieving acceleration by tuning itself. Experimentally, indeed find that deeper networks converge faster than shallower ones.
Contributions.
We summarize our contributions below. We find that:
Nonlinear activation functions induce better separation between similar data in the feature space of model gradient. larger network depth amplifies this better separation phenomenon.
Nonlinear activations have the effect of decreasing the NTK condition number. A larger depth of the network further enhances this better NTK conditioning property.
This better NTK conditioning property leads to faster convergence rate of gradient descent. We empirically verify this on various real world datasets.
The paper is organized as follow: Section 2 describes the setting and defines the key quantities and concepts, and analyzes linear neural networks as the baseline for comparison; Section 3 4 our main results on the better separation and better conditioning of nonlinear activation, respectively; Section discusses the implication on theoretical convergence rates; Section Section/]comme concludes the paper. Proofs of theorems and main corollaries can be found in the appendix.
discuss

---

1.1
Related work
NTK and its spectrum have been extensively studied [19 5 21
25 4 6 since the [SS] shows that the NTK spectrum of an infinitely wide ReLU network asymptotically exhibits power law. Its distribution further shown be similar that of Laplace kernel [1 6], and can be computed (TO]. Nguyen, Mondelli, and Montufar (25) analyzed the upper and lower bounds for the smallest NTK eigenvalue in O() and S2(), respectively. With the assumption of spherically uniformly distributed data where the spectrum of (elementary-wise
T0.
36
discovery of constant NTK for infinitely wide neural networks [17].
power of the Gram matrix becomes simplified, (23). utilizing Hermite polynomials and power series expansion of NTK, provides the order of the smallest eigenvalue of the NTK of two-layer ReLU network in the infinite width limit. Under the same data setting, []] computed the NTK eigenvalues for the two-layer ReLU network. Relying on the values of off off-diagonal entries of the NTK matrix in the infinite depth limit, another work (36) analyzed the asymptotic dependence of the NTK condition number on the network depth L for ReLU networks, which shows decreasing trend as T. increases, consistent with our result.
In contrast prior works, comparison between scenarios with and without ReLU, at any finite depth without data distribution assumption. Note that, without an assumption on data distribution, NTK spectral analysis becomes much harder and many Cana-distributionn-deeeddd results may not hold any more. Moreover, at finite depth, off-diagonal entries of the NTK matrix has converged and typically quite different from its infinite depth limit, which makes analysis even harder.
able distill the effect of ReLU activation function via sharp
We
of prior work (2) which has results of similar flavor. It shows that the depth of linear neural network may help to accelerate optimization via an implicit preciinditinn of gradient descent. We note that this prior work is in orthogonal direction, as its analysis is based on the linear neural network, which activation-free, while our work focus on the better-conditioning effect of activation functions.
2 Setup and Preliminaries
Notations for general purpose. We denote the set 11.2....., by [n]. We use bold lowercase letters, e.g.. v, to denote vectors, and capital letters, e-g-, A, to denote matrices. Given a vector, denotes its Euclidean norm, Inner product between two vectors is denoted by {+ .). Given a matrix A, we denote its i-th row by A its j-th column by A.j. and its entry at i-th row and j-th column by Agj- We also denote the expectation (over distribution) of a variable by [B[+] and the probability of
an event by Pl.]. For a model f(w x) which has parameters w and takes x as input, we use V to denote its first derivative w.r.t. the parameters w, ie., ff = /Ow.
coully)oonnetted) neural network. Let x E Rd be the input, my be the width (i.e., number of neurons) of the 1-th layer, W(I) E mpmxxmi-1,1 E [T. + 1), the matrix of the parameters at layer 1, and (2) be the activation function, which is applied element-wise. A (fully-connected) neural network f. with L hidden layers, is defined as:
(x) x
(x) -Voo (W- vI (1,2, ,4,,
Vmm
f(x) = -W(L-1)
where Co = -( (o For the special case of ReLU activation function: o(2) =
max (0, 2).
We also denote (x0((x) A yer
(x). Following the NTK initialization
ymi
scheme (17), these parameters are randomly initialized i.i.d. according to the normal distribution
(O 1). The scaling factor VCa/ / /mi is introduced normalize the hidden neurons g We denote the collection of all parameters by w.
Remark 2.1. In this paper, we consider the bias-free setting where no bias term is included when computing the hidden neurons Eg.77). In fact, the bias term can potentially lead different results, as have been noticed in [16].
3

---

Without loss of generality, we set the layer widths as
mg d, ML+1 = 1, and my = m, AI E [r].
(2)
and call m as the network width. In the rest of the paper, we typically consider wide neural networks, i.e, networks with large widths m and fixed depths L.
Linear neural network. For comparison purpose, we also consider linear neural network f. which is the same as the neural network defined above, except that the activation function is the identity function (=) = 2 and that the scaling factor of Eq. is 1/vm.
Model gradient feature and neural tangent kernel (NTK). Given model f (e-g.. neural network) with parameters w, we call the the derivative of model f with respect to all its parameters as the model gradient feature vector V f((;; x) for the input The NTK K. is defined as
K(W;:1, = [V](w; (f(;:x)))
(3)
where X₁ and X2 are two arbitrary network inputs. For a given dataset D = {(x; (1) 1=)) there is gradient feature matrix F such that each row (w. V[(w;x) x:) for all E [n]. The n x NTK matrix K ((w) is defined such that its entry Ki (w). w. 4. E [n]. is (,, x,). It is easy to see that the NTK matrix
K(w) = F(w F(w)F(w))
Note that the NTK for linear model reduces to the Gram matrix G' E Rdxd, where each row of the matrix X an input feature x,, i.e., X, = =x²".
As pointed out by (1 19 17 a neural network with large width m is approximately a linear model on the model gradient features (woo:
/(w;x) f(wo x) V[[(w: - wo) + O(I//m).
(5)
Hence, the training dynamics of a wide neural network is largely controlled by the model gradient features of the training samples. We will see that the model gradient angle, i.c., the angle between the model gradient features of an arbitrary pair of inputs, key quantity that measures the mutual relations between training samples and is closely related to the NTK condition number and convergence rate.
Definition 2.2 (Model gradient angle). Given two arbitrary inputs x, z Rd, define the model gradient angle as the angle between the model gradient vectors V (x() and (f(z):
(Vf(x), Vf(z)) [V<x||||><>
6x, 2) A arccos
Condition number. The condition number x of a positive definite matrix A is defined as the ratio between its maximum eigenvalue and minimum eigenvalue:
K Amaz (A)/ Amin (A).
(6)
In the rest of the paper, we specifically denote the NTK matrix, NTK condition number and model gradient angle for the neural network as K, and é, respectively, and denote their linear neural network counterparts K, and é. respectively. We also denote the condition number of Gram matrix G by Ku-
2.1
Without nonlinear activation: the baseline for comparison
To distill the effect of the nonlinear activation function, we need a activation-free case as the baseline for comparison. This baseline is the linear neural network f. with the same width and depth as f.
Theorem 2.3. Consider a linear neural network f. In the limit of infinite network width m + co and network initialization Wo, the following relations hold:
for any input x E Rdd IV x) = (L + 1)|
for any inputs x, z Rd, o(x, Pen (x,

---

This theorem states that, without a nonlinear activation function, the model gradient map Vf : :x: H fx(x) does not change the geometrical relationship between any data samples. For any imput pairs, the model gradient angle e remains the same as the input angle 0in. Therefore, it is not surprising that the NTK of a linear network the same as the Gram matrix (up to constant factor), as formally stated in the following corollary (which can also be consistently obtained using Theorem of (17).
Corollary 2.4 (NTK condition number without activation). Consider a linear neural network f. In the limit of infinite network width Moreover, R = Kg-
and at network initialization, the NTK matrix K (L+
This corollary tells that, for linear neural network, regardless of its depth I,, the NTK condition number & is always equal to the condition number Ki of the Gram matrix G. Therefore, any non-zero
deviations, 5Φ _ Φ 8m from the input angle 6.n. and 5K _ rig from the Gram condition number Ko. observed for nonlinearly activated network f. should be attributed to the corresponding nonlinear activation.
3 Better separation in model gradient space
In this section, we show that the nonlinear activation function helps data separation in the model gradient space. Our theoretical analysis will focus on the special case of ReLU, and the results will be numerically verified on other nonlinear activations as well. Specifically, for two arbitrary inputs x and z with small O.n (x, z), we show that the model gradient angle ((x, 2) is strictly larger than O,n z). implying better angle separation of the two data points in the model gradient space. Moreover, show that the model gradient angle @xx, z) monotonically increases with the number of layers L. indicating that deeper network (more ReLU nonlinearity) has better angle separation.
First, we introduce an auxiliary quantity, L-embedding angle (x) z), which measures the angle between two hidden vectors (x) and ax (z) at infinite width, and an auxiliary function [0, - [0, T)) with g(z) = arccos -== cos + sin.). We also denote the (-fold composition of g(-) as gt Please see Appendix A for the plot of the function and detailed discussion about its
properties. As highlight, g has the following property: g is approximately (but less than) the identity function (2) = z for small z, i.c., < 1.
The following lemma gives the relation between the model gradient angle 6 of any two inputs and their original input angle Oin, via the embedding angles 000) and the function g.
Lemma 3.1. Consider the ReLU network defined in Eq. 1 with L hidden layers and infinite network width. Given two arbitrary inputs x and the angle o z) between the model gradients V fx)
and V (z) satisfies
L-1 a1 0n)) (x, +0
(((1)) vml
cos cos 2) I
Σ
cos 0 (x,
with g (x, = ( (x, z)). Moreover, [f VL | + O Jmm ( for any x.
Better feature separation. Comparing with Theorem 2.3 for linear neural networks, we see that the nonlinear ReLU activation only affects the relative direction, but not the the magnitude, of the model gradient. Lemma ggives giies the relation between şi and the input angle O.n. Figure & as function of O.., for different network depth L.
The key observation is that: for relatively small input angles (say 0n I 30, which is actually not quite small), the model gradient angle is always greater than the input angle Onn This suggests that, after the mapping Vf x x fx(x) from the input space to model gradient space, data inputs becomes more (directionallyy separated, if they are similar in the input space (i.c., with small O..). Comparing to the linear neural network case, where Φ(x, z) = 8sn (x, z) as in Theorem 2.3 we see that the ReLU nonlinearity results in a better angle separation d(x. z) > o(x, z) for similar data.
Another observation that: deeper ReLU networks lead to larger model gradient angles, when On < 30" This indicates that deeper ReL.U networks, which has more layers of ReL.U nonlinear activation, makes the model gradient more separated between inputs. Note that, in the linear network case, the depth does not affect the gradient angle ő.
5

---

100
4- 6p
network, ReLU network, L=3
20
ReLU network, 1-10
network,
30 60 90
120
150
180
Figure 1: Model gradient angles o input angle 6an (according to Lemma 3.1). Linear neural networks (black dash line), of any depth L, always have o 0in. ReLU neural networks with various depths have better separation & Oin for similar data (i.c., small 0.,)) Deeper ReLU networks have better separation than shallow ones for similar data. All neural networks are infinitely wide.
Very similar inputs (i.e., when 8in (x, z) < 1), especially those with different labels, are often hard to distinguish and is one the key factors that makes training difficult, because the decision boundary has to be fine-tuned to separate these very closely located inputs in order to make correct prediction. Hence, the regime of very small input angle (0in < 1) of particular interest for model training. The following theorem confirms the better separation in this regime.
Theorem 3.2 (Better separation for similar data). Consider two arbitrary inputs x, 2 E Rd, with small input angle 0 < 0r (x, z) & 1, and the ReLU network defined in Eq.(7). The model angle o(x, x, 2) is strictly greater than the inpul angle O.n (x, 2):
Φ(x, 2) 0: 0x.(x,2) 2).
(8)
with high probability of the random network initialization, the network width 12(1/Of))
The following corollary quantifies the better separation in this regime.
Corollary 3.3. With the same setting in Theorem 3.2 and with infinite width - but finite depth = 22(1/0in). cos Φ(x, z) = -1 - 2x Oin + o(0in) cos Oin-
Remark 3.4 (Separation in distance). Indeed, the better angle separation discussed above implies better separation in Euclidean distance well. This can be easily seen by recalling from Lemmal 3.7 that the model gradient mapping f preserves the norm (up universal factor L 1).
We also point out that, Figure I indicates that for large input angles (say Oin > 30") the model gradient angle is always large (greater than 30%). Hence, non-similar data never become similar in the model gradient feature space.
Better separation in infinite width and depth limit. Now, consider the infinite width and depth case. We took the infinite width limit a prior, this technically leads to the infinite width-theen-een limit. The following theorem shows that, no matter how similar two inputs originally are, as long as they are not parallel, their model gradient features eventually get wide separated in large depth.
Theorem 3.5. Consider the ReLU neural network defined in Eq. and two non-parallel inputs and z, x H z. In the sefnite-width-iiee tethh--hhe--pptttt limit, the model gradient angle Φ(x, converges to fix value arccos 7, regardless the inpul angle 0x.(x,2) z).
Remark 3.6. The limit point value arecos & is about 75.5", which means the inputs are quite well- separated the model gradient feature space, as network depth increase to infinity. Recall that, without the nonlinear activation, 6(x, z) Oin which can be arbitrarily small.

---

ReLU?
GeLUU
Tanh
130
40
60
100
Oa
120
180
180
120
Figure 2: Better separation for non-ReLU activation functions. Left: ReLU?, Middle: GeLU, Right: tanh. All plots are model gradient angle vs. imput Pm..
is interesting to observe that this limit point value is independent of the input angle, which means that the data points are mutually equally-separas in the limit. We will discuss its implications NTK in the following sections.
Beyond ReLU activation. Here, we numerically verify that the better separation phenomenon obtained for ReLU above also holds for other nonlinear activation functions. Figure shows the relations between the model gradient angle & and the input angle Ann for the following nonlinear activations: ReLU (i.e., (2) max(0, 2 }), GeLU and tanh. One easily that the better separation holds: for relatively small input angles 8.n (e.g., fin S 30%), Φ is always greater than Oni and for deeper networks,
even greater. Interestingly, we observe that the gradient angle converges to 90" for these activation functions indicating gradient features become orthogonal in the limit of I. oc, different from the 75.5" that obtain for ReLU networks.
We also show that the better separation generalizes beyond the NTK setting/regime. Please see Appendix B for more discussion.
4 Better NTK conditioning
In this section, show both theoretically and experimentally that, the nonlinear activation induces decrease in the NTK condition number k. Moreover, a neural network with larger depth L, which means more nonlinear activations in operation, the NTK condition number is generically smaller.
Connection between condition number and model gradient angle. The smallest eigen- value and condition number of NTK are closely related the smallest model gradient angle min;,j ix] Xx), through the gradient feature matrix F. Think about the case o(x,, x,) = (i.e., (x, ) is parallel to f((xx)) for some i, [ni] then F, hence NTK K, is not full rank and the smallest eigenvalue Amiin is zero, leading to infinite condition number s. Similarly, if min;,j ix] (Xx,, xj) is small, the smallest eigenvalue Aman (K) also small, and condition number is large, stated in the following proposition (see proof in Appendix C).
Proposition 4.1. Consider with d > n, is of full row rank. Suppose that there exist i, E [n] such that the angle o between vectors Bü and By. small, i.e., & & 1, and that there exist constant C e Bs < C for all l E [n). Then, the smallest eigenvalue Amin (A) O(o?), and the condition number k = (11)2)).
positive definite matrix A BB", where matrix B Raxd
O such that
Therefore, good data angle separation in the model gradient features, i.c., min;,jj[mi 6(x,, not too small, is necessary condition such that the condition number x is not too large. As is shown in the last section, the ReLU nonlinearity makes the samples more separated when mapped from the input data space to the model gradient feature space. Hence, it is expected that the NTK condition number will decrease in the presence
ReLU nonlinearity.
Smaller NTK condition number. Theoretically, we consider the infinite width limit. We require that the dataset is not degenerated, i.e., x; If x, for all j.j. This is mild and commonly used setting in the literature, see for example (9). We require that the weights of the first layer (1)) be trainable
7

---

55-
169
Laripwec:: ReL.
19851: hoLd
Unear
30
Synthdde: heL.
unsar
shear
F-MIST: Ro.
uno
Wodel (ww.eenn 20..
25
Condition numberr
20.
Network
Figure 3: Better separation (left) and Better NTK conditioning (right) of ReLU network on various datasets. Solid lines of ReLU networks, dashed lines are of linear neural networks for comparison. Left: Minimum ó (in degrees 9) vs. depth. ReLU network has better separation of model gradient feature depth increases. Right: NTK condition number has better conditioning of NTK as depth increases. Note that L 0 corresponds to the case of a linear model and linear neural network, and the NTK in this case is the Gram matrix.
depth. ReLU network
and fix the other layers in the following theorem. This is also common setting in literature to simplify the analysis [9].
Theorem 4.2. Consider the ReLU network in (4 in the limit m - do and at initialization.
Let the weights of the first layer W()) be trainable and fix the other layers. We compare the two scenarios: (a) the network with ReLU activation and (b) the network with all the ReLU activation removed. The smallest eigenvalue Amin K) of its NTK in scenario (a) larger than that scenario
(b): Amin (Ka) ) > Amiu (Kr), and the NTK condition number is in scenario (a) is less than that in scenario (b): Ki Rb- Moreover, for ReLU neural networks fa of depth L1 and f2 of depth L2 with L1 L2, we have Ksz < f2
This theorem confirms the expectation that the NTK condition number н should be decreased, as consequence of the existence of the ReLU nonlinearity. This theorem also shows that the depth of the ReLU network enhances this better NTK conditioning.
The high-level intuition behind the proof this theorem is that: the derivative of the ReLU function, d'(x) =(<=)=[(=20) [=20]]. resembles a binary gate which has open and close states. When ReLU are imple- mented, the model gradient map V
V fx(x) increases the directional diversity of the vectors ff(x), due to the high dimension of the model gradient space and the different activation patterns of the hidden layer for different samples Hence, it expected that the feature matrix F, as well as the NTK matrix K, is better conditioned.
In fact, fixing the weights of the top layer is not necessary and can be removed. We relax this requirement in Appendix FF In our experiments in Section [.1 where all layers are trainable, we observe the phenomena of better separation and better NTK conditioning.
NTK condition number in infinite depth. As consequence of the pairwise equal separaato result (Theorem (.55), the NTK matrix got simplified in the infinite depth limit. The following theorem shows that the NTK condition number converges to fixed value n+4 data distribution.
which is independent of the 3
Theorem 4.3. Consider the ReLU neural network defined in Eq. and dataset {(x,, v1) W)) Suppose that all data inputs are normalized ||x;|| = 1 for all i, and x; It x, for all # j. In the weithte-width-hhen-ee limit, the NTK condition number & converges to nt4
4.1 Experimental evidence
Here, experimentally show that better separation and better conditioning happen in practice.

---

inNNIST
Libriswwec
Leserment
Lecer reeel
Unea redel
Dipat-2
Depine
Beshw)
Depthot
Dasthi
Dep
Beshwil
Dopthe7
Dasthil
Deppiik Depmn-3
Rasher
Digtte))
Desthw23
Dipth-00
Szocha
Cocera
Epocha
Figure 4:
Training eurve of ReLU networks with different depths. On each of these datasets, we see that deeper ReL.U network always converges faster than shallower ones.
Dataset. We use the following datasets: synthetic dataset, MNIST [18). FashionMNIST (f-MNIST)
[35]. SVHN [24) and Librispeech [27]. The synthetic data consists of 2000 samples which are randomly drawn from a 5-dimensional Gaussian distribution with zero-mean and unit variance. The MNIST, f-MNIST and SVHN datasets are image datasets where each input is an image. The Librispeech is a speech dataset including 100 hours of clean speeches. In the experiments, we use subset of Librispeech with 50, 000 samples, and each input a 768 768-dimensional vector representing a frame of speech audio and we follow [3] for the feature extraction.
Models. For each of the datasets, we use ReLU activated fully-connected neural network ar- chitecture to process. The ReLU network has L hidden layers, and has 512 neurons in each of its hidden layers. The ReLU network uses the NTK parameterization and initialization strategy (see (17)). For each dataset, we vary the network depth L from 0 to 10. Note that L 0 corresponding to the linear model case. In addition, for comparison, we use a linear neural network, which has the same architecture with the ReLU network except the absence of activation function.
Results. For each dataset and given network depth I., we evaluate both the smallest pairwise model gradient angle ji]] Φ(x;, Xj) and the NTK condition number K, at the network initialization. We take 5 independent runs over random initialization seeds, and report the average. each run, we used A-100 GPU to compute the NTK, which took 4 ~ 10 hours. The results are shown in Figure 3 We compare the two scenarios of with and without the ReLU activation function. As one can easily see from the plots, ReLU network (depth L = 1,2, ... 10) always have better separation of data features (i.c., larger smallest pairwise model gradient angle), and a better NTK conditioning (i.e., smaller NTK condition number), than its corresponding linear network (compare the solid line and dash line of the same color). Furthermore, the monotonically decreasing NTK condition number shows that deeper ReLU network have better conditioning of NTTK
5 Optimization acceleration
Recently studies have shown strong connections between the NTK condition number and the thcoreti- cal convergence rate of gradient descent algorithms on wide neural networks 9 8
37 26 20
In g 20]. the worst-case convergence rate has been shown to be
L(wi) S - (11
Although x is evaluated on the entire optimization path, all these theories used the fact that NTK is almost constant for wide neural networks and an evaluation at initialization Wo is enough.
As smaller NTK condition number (or larger smallest eigenvalue of NTK) implies a faster worst- case convergence rate, our findings suggest that: (a), the ReLU activation function helps improve the worst-case convergence rate of gradient descent, and (b), deeper wide ReLU networks have faster convergence rate than shallower ones.
We experimentally verify this implication. Specifically, we train the ReI.U networks, with depth L ranging from to 10, for the datasets MNIST, f-MNIST, and Librispeech. For all training tasks, we cross-entropy loss the objective function and use mini-batch stochastic gradient descent (SGD)

---

of batch size 500 to optimize. For each task, we find its optimal learning rate by grid search. On MNIST and f-MNIST, we train 500 epochs, and on Librispeech, we training 2000 epochs.
The curves of training loss against epochs are shown in Figure We observe that, for all these datasets, deeper ReLUU network always converges faster than shallower one. This is consistent with the theoretical prediction that the deeper ReLU network, which has smaller NTK condition number, has faster theoretical convergence rate.
Trade-off between optimization and generalization. Although a faster convergence in terms of number of iterations for deep networks, Theorem 3.5 suggests, in the extreme of infinite depth L - 30, any non-parallel input pairs become equally separated in gradient features regardless of their original similarity. Even though not mutually orthogonal, this could also result in a trivial generalization: close to random guess for unseen data. The same consequence can also be obtained from [4], where they dropped the initial random guess value and obtained a zero prediction for unseen data.
As for finite depth, it is theoretically hard to predict what depth this trade-off starts happen. Under the same experimental setting as in Figure Table that the generalization performance starts decrease at depth I, suggesting optimization-generalinn
trade-off for large depth.
Table I:
Generalization dependence on ReLU network depth L. Test accuracies are reported after training convergence on MNIST.
Depth L
10
12
test accuracy
95.98
97.43
97.57
97.52
97.39
97.19
Conclusion and discussions
In this work, we showed the effects of nonlinear activation on better separation of similar data in feature space and on the NTK conditioning. We also showed that more sequential activation operations, i.e., larger network depth, amplifies these effects. As the NTK conditioning is closely related to theoretical convergence rate of gradient descent, our findings also suggest a positive role of activation functions optimization theories. A limitation of the paper that the theoretical analysis is only conducted on ReI.U activation, although results have been empirically verified for other nonlinear activations. For other activations, the analysis requires analytical expressions for integrations involved, which requires distinct type of analysis and we consider it as future work.
References
[1] Zeyuan Allen Zhu, Yuanzhi Li, and Zhao Song. "A convergence theory for deep learning via over-parameterizzaion"" In: International Conference on Machine Learning. PMLR. 2019, pp. 242-252.
[2] Sanjeev Arora, Nadav Cohen, and Elad Hazan. "On the optimization of deep networks: Implicit acceleration by overparameterizztionn.. In: International Conference on Machine Learning. PMLR. 2018, pp. 244-253.
[3] Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. "The convergence rate of neural networks for learned functions of different frequencies". In: Advances in Neural Information Processing Systems 32 (2019).
[4] Yuval Belfer, Amnon Geifman, Meirav Galun, and Ronen Basri. "Spectral analysis of the neural tangent kernel for deep residual networks". In: arXiv preprint arXiv:2104.0300 (2021).
[5] Alberto Bietti and Julien Mairal. "On the inductive bias of neural tangent kernels". In: Advances in Neural Information Processing Systems 32 (2019).
[6] Lin Chen and Sheng Xu. "Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS" In: International Conference on Learning Representations. 2021.
[7] George Cybenko. "Approximation by superpositions of a sigmoidal function". In: Mathematics
of control, signals and systems 2.4 (1989), pp. 303-314.
10

---

[8] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. "Gradient Descent Finds Global Minima of Deep Neural Networks". In: International Conference on Machine Learning. 2019, pp. 1675-1685.
[9] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. "Gradient Descent Provably Optimizes Over- paeeppprraeterizee Neural Networks". In: International Conference on Learning Representations. 2018.
[10] Zhou Fan and Zhichao Wang. "Spectra of the Conjugate Kernel and Neural Tangent Kernel for lincar-width neural networks". In: Advances in Neural Information Processing Systems 33 (2020).
[11] Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. "On the similarity between the laplace and neural tangent kernels". In: Advances in Neural Information Processing Systems 33 (2020), pp. 1451-1461.
[12] Boris Hanin and Mark Sellke. "Approximating continuous functions by relu nets of minimal 11278 (2017).
width". In: arXiv preprini
[13] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. "Multilayer feedforward networks are universal approximators". In: Neural networks 2.5 (1989), pp. 359-366.
[14] Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. "Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks?-Al A Neural Tangent Kernel Perspective". In: Advances in neural information processing systems 33 (2020), pp. (198-2709,
[15] Like Hui and Mikhail Belkin. "Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks". In: arXiv preprint arXiv:2006.0733
(2020). [16] Arthur Jacot, Franck Gabriel, Francois Ged, and Clement Hongler. "Freeze and chaos: Ntk views on dnn normalization, checkerboard and boundary artifacts". In: Mathematical and Scientific Machine Learning. PMLR. 2022, pp. 257-270.
[17] Arthur Jacot, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: Convergence and generalization in neural networks". In: Advances in neural information processing systems 31 (2018).
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. "Gradient- based learning applied to document recognition". In: Proceedings of the IEEE 86.11 (1998), pp. 2278-2324.
[19] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. "Wide neural networks of any depth evolve as linear models under gradient descent". In: Advances in neural information processing systems 32 (2019).
[20] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. "Loss landscapes and optimization in over- parameterized non-linear systems and neural networks". In: Applied and Computational Harmonic Analysis 59 (2022), pp. 85-116.
[21] Chaoyue Liu, Libin Zhu, and Misha Belkin. "On the linearity of large non-linear models: when and why the tangent kernel is constant". In: Advances in Neural Information Processing Systems 33 (2020), pp. 15954-15964.
[22] Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. "On the number of linear regions of deep neural networks". In: Advances in neural information processing systems 27 (2014).
[23] Michael Murray, Hui Jin, Benjamin Bowman, and Guido Montufar. "Characterizing the Spectrum of the NTK via a Power Series Expansion". In: International Conference on Learning Representations. 2023.
[24] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. "Reading digits in natural images with unsupervised feature learning". In: Proc. Int. Conf. Neural Inf. Process. Syst. Workshops (2011).
[25] Quynh Nguyen, Marco Mondelli, and Guido Montufar. "Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks". In: International Conference on Machine Learning. PMLR. 2021, pp. 81 19--891.
[26] Samet Oymak and Mahdi Soltanolkotabi. "Toward moderate overparameteriiatton:: Global convergence guarantees for training shallow neural networks". In: IEEE Journal on Selected Areas in Information Theory 1.1 (2020), pp. 84-105.
[27] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. "Librispeech: an asr corpus based on public domain audio books". In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEFE, 2015, pp. 5206-5210.
11

---

[28] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. "Exponential expressivity in deep neural networks through transient chaos". In: Advances in neural information processing systems 29 (2016).
[29] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. "On the expressive power of deep neural networks". In: international conference on machine learning.
PMLR. 2017, pp. 2847-2854.
[30] Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. "Deep information propagation". In: arXiv preprint arXiv:1611.0122
(2016).
[31] Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. "Theoretical insights into the optimization landscape of over- over-voeraaae perammetettee shallow neural networks". In: IEEE Transactions Information Theory 65.2 (2018), pp. 742-769.
[32] Matus Telgarsky. "Representation benefits of deep feedforward networks". In: arXiv preprint arXiv: 550..080910111 (2015).
[33] Maksim Velikanov and Dmitry Yarotsky. "Explicit loss asymptotics in the gradient descent training of neural networks". In: Advances Neural Information Processing Systems 34 (2021), pp. 2570-2582.
[34] Qingcan Wang et al. "Exponential convergence of the deep neural network approximation for analytic functions". In: arXiv preprint arXiv: 1807.00297 (2018).
[35] Han Xiao, Kashif Rasul, and Roland Vollgraf. "Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms", In: arXiv preprint 1708. 07747 (2017).
[36] Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. "Disentangling trainability and generalization in deep neural networks". In: International Conference on Machine Learning.
PMLR. 2020, pp. 10462-10472.
[37] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. "Gradient descent optimizes over- parameterized deep ReLU networks". In: Machine Learning 109.3 (2020), pp. 467-492.
12

---

A Properties of function g
Recall that the function g: [0, T) - (0, σ²) is defined as (see Lemma E.5
g(2) arccos
cos
sin
(10)
π
Figure shows the plot of this function. From the plot, we can easily find the following properties.
120
y=z function
0
120
150
Figure 5: Curve of the function g(0). As can be seen, g(0) is monotonic, and is approximately the identity function in the small angle region (0 < 90%).
Proposition A.1 (Properties of 9). The function g defined in Eq. 110 has the following properties:
1. is a monotonically increasing function;
2. g(2) S for all 2 E T); and g(2) if and only =
3. for any 2 E [0, r), the sequence (x)) is monotonically decreasing, and has the limit limy (2) =0
Proof. Part 1. First, we consider the auxiliary function ğ(2) Σ-Σ cos sin We see that
dō(2) - 2 sin 0, Vz E T).
dz
Hence, Í(z) is monotonically decreasing on [0, T). Combining with the monotonically decreasing nature of the arccos function, we get that g is monotonically increasing.
Part 2. It suffices to prove that cos z S ī(z) and that the equality holds only at 2 = For 2 = it is easy check that
g(2). as both z and sin are zero. For 2 E (0,
π 1/2), noting that
tan 2 z > 0, we have
1 (-2 + tan 2) cos > cos
ī(2)
cos
+ π
= cos π
(11)
π
For T/2, we have cos 2 0 < 1/x 9(T/2). For z E (T/2, we have the same relation
as in Eq. (T. The only differences are that, this case, cos < and tan z < Therefore, we still get ((z) > cos for (=/2, T).
Part 3. From part 2, we see that g(z) < z for all 2 E (0, m). Hence, for any (z) < (2)).. Moreover, since is the only fixed point such that g(z) 2, in the limit - Xo, g2()) - 0
worth note that the last property of y function immediately implies the collapse of embedding
vectors from different inputs in the infinite depth limit L, - This embedding collapse has been observed in prior works [28 30) (although by different type of analysis) and has been widely
discussed in the literature of Edge of Chaos.
13

---

100
80
80
60
40
40
10
20
10
9m
60
On
120
180
60
120
180
100
100
80
80
60
8
40
40
3
10
20
0x
0
Pm
180
Figure 6: Model gradient angle & vs. input i. in different scaling regimes o 0.01, 0.1, 1 and 10. The better feature separation always holds for similar data (when On is small, left end of each plot).
Theorem A.2. Consider a ReLU neural network. Given any two inputs x, 2 E Id,, the sequence of angles (00)) (x, 2) between their l-embedding vectors a (x) and a (z), is monotonically decreasing. Moreover, in the limit of infinite depth,
lim 01 0,
(12)
L-000
and there exists a vector such that, for any input the last layer L-embedding
a (x) ||x||a.
(13)
B Beyond the NTK regime
Here, we show that the better separation phenomenon still holds outside of the NTK regime. We consider different initialization scales initialization. 6 the model gradient angle á as a function the input 0 for different scaling regimes: ou 0.01, 0.1, and 10. It shows that the better separation phenomenon still holds for similar inputs at various network depths.
No o.). Note that o 1 corresponds to small
C Proof of Proposition 4.1
Proof. Consider the matrix and the vectors by A Br. E [n].. The smallest singular value square of matrix B is defined
T TBBTV
Ee uyball
min (B)
min v=0
min v-40
vTv v
|
14

---

Since the angle á between b, = Bg and b, By is small, let v' be the vector such that of =
|
o/j -bbb and U/ = 0 for all k & i.j. Then
Ex o bEll
|b
b by
a min (B)
VIbb;1? 2 + |byl b.
|/
//2b,12 2 + ||b 12
2/| |
b 1 + | 12 cos ()
[b. b
[b 1 + b 1| 6 O
Since A = BBT, the smallest eigenvalue Amin (A) A is the same as min (B).
On the other hand, the largest eigenvalue Amaz (A) of matrix A is lower bounded by tr( A)/n. Note that the diagonal entries Age = |b&l |. Hence, e E Amaz (A) S C. Therefore, the condition number Amar (A)/ (A)/Amin (A) = (21/6²²).
D Proofs of Theorems without (ReLU) activation
D.1 Proof of Theorem 2.3
Proof. First of all, we provide a useful lemma.
Lemma D.I. Consider matrix A Rm with each entry of A ii.d. drawn from (0, 1). In the limit of m + oc.
1 LAT A + laxds in probability.
(14)
We first consider the embedding vectors (!)) and the embedding angles 8(), By definition of linear neural network, we have, for all E [T.] and input x Rdd,
a0)
W Www(--... w(
(15)
m/
Note that at the network initialization entries of W() are i.d. and follows \(0, 1). Hence, the inner product
(x). az)) = x TWII ... w-(11) w (1) w(0 (1) (a) xT
m
where in step (a) we recursively applied Lemma D.1
times. Putting
get |ā (2) (x))l
||x||
for all E [L] By the definition of embedding angles, is easy to check that (1() (x, = Pin (x. 2).
for all E []].
Now, we consider the model gradient V and the model gradient angle Φ. As we consider the model gradient only at network initialization, we don't explicitly write out the dependence on Wo, and we write V f((o, x) simply as V fxx). The model gradient V can be decomposed as
0f(x)
fx() (v) v2f(x), .... Vx+I((x)), with v.f(x) ow vI E [L+1]. (16)
Hence, the inner product
L+1
(vf()), vf(2)) Σ (Vi((x), vif(z))
and for all / + 1].
L+1
L+1
(V)((x), vif(2)) (a(11)) (x), α (1- (z))
II
W(YT
II 1mt+1
- (b)
TT.
Vm
mm
1=++1
Here in step (b), again applied Lemmal D.I Therefore,
v fxx), ff(z)) (L
+ 1)x
(17)
Putting z = x, we get | to check that Φ(x, p.. (x
(T. + ||] By the definition of model gradient angle, it is easy
D
15

---

E Proofs of Theorems for ReLU network
E.1 Preliminary results
Before the proofs, we introduce some useful notations and lemmas. The proofs of these lemmas are deferred Appendix G
Given a vector v E R,, we define the following diagonal indicator matrix:
[[v20) diag (11 (0220):;"" ,I1220)),
(18)
with
1 2 0, 0 U2 < 0.
[1,200] =
Lemma E.1. Consider two vectors V1, V2 E Rp and p-dimensional random vector w ~
(0,, Denote as the angle between V and V2, i.e., cos
(21.Va) [ v Then, the prob-
ability
1
2() v2 2 0)
(19)
2
Lemma E.2. Consider two arbitrary vectors V1, V2 E RP and a random matrix W E R9*p with entries Wij iid. drawn from (N, 1). Denote as the angle between Vı and V2, and define u1 (WVi) and u2
1/2
v9 o (Woo((W)))))) Then, in the limit of q + co.
(u1, u2)
(( 0) cos 0 sin [valllvell-
(20)
Lemma E.3. Consider two arbitrary vectors V1, V2 E TRP and two random matrices E Txx and W E ROXy, where all entries Uzg. i [s] and E [g]. and Wa. & E [g] and I E [p), are i..dd drawn from (001)). 1). Denote 8 as the angle between V1 and V2, and define matrices A1 va (Ww and A₂
222
v2 U UWW Then, in the limit of q + the matrix /
A = π - 0
1axs.
(21)
Lemma E.4. Consider matrix B = AAT with A E B and random matrix W RIXP where all entries W are i...d. drawn from (0, 1). Define the tensor A' E Raxxxx such that A/y ==
v2Asal (W, Au2) Let B' E Igxx be the matrix such that each entry B'y = EEE, Al jks- Then, in the limit of - 00, the smallest and largest eigenvalues satisfy: Amin (B') Amin (B), and Amaz (B") < \max (B).
E.2
Proof of Lemma
Proof. The model gradient V (x()) is composed of the components Vif(x) A af Each such component has the following expression: for E [L + 1]
BW7, for / E [L 1].
vif(x) = a (1-1) (x)6 (x),
(22)
where
$ 1
(x)
2
WiL+1)
(a(L) W yau (x)20) ... (a
(23)
Note that in Eq. 22 vx)(x) outer product of column vector a (1-1) (x) E R m (m-11 d
if I 1, and m otherwise) and row vector 5L0) (x) Rixmi (m) 1 if L and my m otherwise).
First, we consider infinitely wide neural network & of depth L. We have the following lemma.
Lemma F.5. Consider a ReLU network poe defined in Eq. 7 with infinite width. For all E [T.), the following relations hold:
16

---

for any input E R [la"(x)) (x)] = ||x||;
for any two inputs x, z R., Q00) (1, 2) 9 (0(1-1) z)) Let g'(:) be the l-fold composi-
tion of g()), then
gl0 (x, 2) got (0in(x22))
(24)
We consider the inner product (z), (x)). for E [ + 1 By Eq.22)
we have
(z), V (x)) = (6 (z), 80h (x)) (1-11) (z), (1-1) (x))
(25)
For (a 11 (x) applying Lemma E.5 we have
{a (1-1)))) (z),a (x)) | |z| cos 0 1) (x, 2).
(26)
For (z), (x) by definition Eq. 23
have
1-11
(000 (2), 510 (x)) = 2
x W((+ (x000x000))
w(++) (1200020000000000000 (20 W(+1) ...I W(++I)T
Recalling that a ((O and applying Lemma
E.3 the the term A above, obtain
(z), (z), (x)) (11 (x, z) (8(1+1) (z), (0(
n
Recursively applying the above formula for l' 1,1 + 1, ... L, and noticing that 8(4+1) we have
(ll)) (x, z))
(2), (x))
II
(27)
V=l-1
Combining Eq.
25.
26 and 27 we have
L-1
(x,)(x22)
(V./ (z), Vi (x) = ||x||||||| cos (21 (x, 2)
(28)
1==--1
7
For the inner product between the full model gradients, we have
L+1
L-1
80') (xx z)
(V) (z). V (x)) E (z) Vt.
||x||| Σ
cos (
(x, z) P=t
II
(29)
Putting the above equation, we have g(0) (x1 (x, 2) = 0 for all E [T.], and obtain
| [x ||x| 2 + 1).
(30)
Hence, we have, for an infinitely wide neural network,
cos Φ (x, z) = (V (2) (x)
L-1
| x [f
= L 1=0
Σ
cos
( (x, z)
II
(1
g0)) (x, 2)/
V=1
(31)
Now, we consider the finitely wide neural network f. As have been shown by
17.9 9 21
1
(V)(z), Vf(()))((xx) (V) (x))
(32)
m
with high probability of random initialization of the network f. Letting z = x above, we also have
[ = || (x)
(33)
\m
Using the above two equations, we have
cos (xx z) fz)( vx(x)) = cos の0 (x, z) +
(34)
mm
with high probability of random initialization of the network f.
With bit of abuse of notation, refer the flattened vectors of Vif in the inner product.
17

---

E.3 Proof of Theorem 3.2
Proof. For simplicity of notation, we don't explicitly write out the dependent on the inputs x, z, and write ((0) _ g(0) xx z). and ø - <xx,2). We start the proof with the summation term on the R.H.S.
of Eq. 7 in Lemma 3.1
cos (
(00) /T)
1-1
L-1
cos (())
1
tan 0(0)) - _
II
(1 000)/7))
d==0
cos ((0)
1 1+ 3π
(0()))3 o(or1))3
(1 0(())
1=0
P==
P=1
(c) cos B(0)
[-1
(00) 3 + 0((10))a)
1++1
3m
[m
L-1
0(0) +
(0(0))2 + o((810))3)
3
F=1
COS (0)
L - (00)
(L - ()L | - 2) (010))2 0 ((0(0))3)
L+1
π
37
L f(0) + o(B(0))
cos ((0)
2
In step (a) above, use the relation (00 g(((1-1)), ), i... cos g(1) = c-cos (((-1) / sin 011-1))/ /T;; in step (b), used the fact that ((0) Oin (Theorem A.2 which stays small and used the Taylor
expansion of tan. step (c), we used the following lemma (proof is in Appendix G.7):
Lemma E.6. Given any imputs x, such that O:n (x,(x, z) < 1, for each I [L]. the l-embedding angle g() (x, z) can be expressed
g0' 2) 0m (x, 3π
(0 (x. 2))² ((0in(x,
By Lemma 3.1 there exists constant such that
cos (xx, < (1 (1- L
6m + o(fzn) cos Im + Vm
(35)
2x
When m
16m2 La cos? P.n we have cos Φ(xx z) < cos (x, 2). namely, @(x, z) > (x,
E.4
Proof of Theorem 3.5
Proof. We start the proof with analysis of the embedding angles g()) in the infinite depth limit.
First, by Lemma R.5 and Proposition A.1 we easily find that, for all input angle On, = p(0) * 0, 0())
monotonically decreasing and converges to zero: lim;-soo 00)) + 0. As the following analysis is independent of g(0), we will not explicitly write out the arguments and z.
Now, we analyze its convergence rate, utilizing Fq.(57). As we are considering the infinite depth limit and g(t) converges to zero, we can drop its o(.) term and rewrite Eq. 57 as:
d0()
(0(0)2
(36)
dl 3m
Solving this differential equation, we get (()) ((0) /1 + (30)-1)(1)1) and
lim ((1) .1 3m. (-000
(37)
18

---

By Eq. we get the following relation between ΦL and ØL+1:
(L + 2) cos ØL-1 = (1- / (L + 1) cos Ør + cos g(++1),
(38)
Rearranging terms, we get
(cos 0L+1 cos (b.. = 1-0))==-11 (L + 1)0(1)
ØL
L cos p(++1).
(39)
L+2
The right side of (3) converges to zero L co. Using Eq (7) and Lime cos f (++1)
we have
1
lim cos 0r. L-000
(40)
Hence, dL converges to arccos } = 75.5%, in the limit L -
E.5 Proof of Theorem 4.2
Proof. First of all, we note that in scenario (b), i.c., the network with all ReLU activation removed, the network simply becomes linear neural network (while with the same trainable parameters W(1) as the ReLU network in scenario (a)). By the analysis in Section 2.1 we can easily see that the NTK matrix in scenario (b) equivalent the Gram matrix G, and Ki fig. Hence, whenever comparing the two scenarios, it suffices to compare the NTK K (and its condition number x)) of ReLUU network with the Gram matrix G (and its condition number Kig). We prove the theorem by induction.
Base case: ReLU neural network of depth L 1. First, consider the shallow ReLU neural network
V2 v (WW).
f(W:x)=
(41)
mm
where W are the trainable parameters. The model gradient, for an arbitrary input can be written as
v fx()) = xx(x) E IR dx
(42)
where 6(x) E 1lxm has the following expression
2 =
8(x)
vd I
m
At initialization, W is a random matrix. Recall that the NTK K = FFT, where the gradient feature matrix F consist of the gradient feature vectors fx()x for x for the dataset. Applying Lemma D.I in the limit of m - co, we have that each entry Kiş is equivalent to Σx. A A'ar jki- with
A == V 12 20,) where X Ra is the matrix of input data. Then apply Lemma E.4 immediately have that
Amin (K) > Amin (G), Amaz (K) < >mma (G).
Hence, we have that Ka < KG-
In addition, note that this network has hidden layer, and that the "zero-hidden layer" network is just simply the linear model. For linear model, the NTK is simply the Gram matrix. Hence, for the base case, we have Kn < Kfz- = Ke, with network fi of depth 1 and network fa of depth 0.
Induction hypothesis. Suppose that, a ReLU network ft-1 of depth I. 1, its NTK condition number KL-1 is strictly smaller than Kg-
19

---

Induction step. Now, let's consider the two ReLU networks fo of depth L and fL-1. It is suffices 11,-1- The model gradients, for any given input x, can be written as:
prove that Rt.
(x) = xxx((x) Rdxm Vft-i(()) xxx.-1(x)) E Rdxm
where
2 W (was m
(x(x)
-V Vm m
6x-1(x) = =V IwrC-
w I
2122))20 2120)
m
Note that the matrix W(L) has different dimensions for ft and fL-1-
Using the same argument in the base case, well as applying Lemma D.1 8(x)'s s, we directly obtain Ky < KL-1-
when contracting the
E.6 Proof of Theorem 4.3
Proof. First, consider the normalized NTK matrix L+1 K. By Lemma 3.1 we have for it diagonal elements:
1
L L+1 (x,, x.)
V ff(x)
||x.|2
(43)
L+1
By Theorem 3.5 we have that, in the infinite depth limit, each of off-diagonal elements of the normalized NTK matrix converges to Namely,
1 4
K L+1
In + Jn:
(44)
4
where matrix Jn has its elements being ones. Therefore, lim T.TT
1 K has one eigenvalue
A 1 + T and all remaining eigenvalues X2 = Ag -...
3 Then its condition number is An 4.
F Relaxing the constraint on top layers
Theorem F.1. Consider L-layer ReLU neural network f as defined in Eq. in the infinite width limit m - 30 and at initialization. We compare the NTK condition numbers Ra and Ka of the two scenarios: (a) the network with the ReLU activation, and (b) the network with all the ReLU activation removed. Consider the dataset D = {(x1, V1), (X2, V2)} with the input angle 6sn between X1 and x2 small, 0mn < 1. Then, the NTK condition number Ka Kb- Moreover, for two ReLU neural networks fi of depth L, and fo of depth L2 with L, > Izz, we have K. < Kifz-
Proof. First, let's consider the scenario (a), i.e. the ReLU network. According the definition of NTK and Lemma 3.1 the NTK matrix K for this dataset D = {(x1, (x2, w2)3 is (NTK is normalized by the factor (T(T. 1)3):
[x (xx),
(VX(x)), V)(x2)) ) - f(x)))
-
X | X2
|
X2 cos Φ Xx
)
K
v)(x)))
||x]
cos
The eigenvalues of the NTK matrix are given by
1 2 ||
(k))
||x2|| + / Xx] || |x2 | +
|
1
X
2
cos
26)
(45a)
{²( 2 ||x||| /
1
|||||]
1|
+
||
|
cos
26)
(45b)
20

---

In the scenario (b), the ReLU activation is removed in the network, resulting in a linear neural network. In this case, the NTK is equivalent to the Gram matrix G, given by Corollary 2.4
We have ||x||| I1? x X2
x xg
)
||X I|? X2
||x
||||
cos 0in
||x2|
|||||
cos Psn
X
and its eigenvalues as
(G)
Xx2
12
x
||
X22
X
X2
cos 20
X2 [x |
X2
x
X2
|
1
X2| 22
cos 20
By Theorem 3.2 have cos Φ cos Iin. when am < and 0. 0. Hence, have the following relations
(G) (K))
d2 (K) Az))
which immediately implies Ria <
Kb-
When comparing ReLU networks with different depths, i.e., network fi with depth L1 and network f2 with depth L2 with L1 L2, notice that in Eq,445) the top eigenvalue λι monotonically decreases in d, and the bottom (smaller) eigenvalue X2 monotonically increases in ø. By the proof of Theorem 3.2 we know that the deeper ReLU network fi has a better separation than the shallower one f2. ie..
6121 Hence, we get
(1 > {1(Kg)) (2(K))) > (2
(46)
Therefore, we obtain Kif < K..z Namely the deeper ReLU network has a smaller NTK condition number.
G Technical proofs
G.1 Proof of Lemma D.1
Proof. We denote Aij as the (i,j)-th entry of the matrix A. Therefore, (A A) Ek= Aui First we find the mean of each (A"'))is Since Ai are i..d. and has zero mean, casily see that for any index k.
1, if
E E[AA:Any]
10 otherwise
Consequently,
EI( - A([-AA))sh A)úl =
11, if
0.
otherwise
That is [ TT A|
Id-
Now we consider the variance of each (ATA)ij: if + j we can explicitly write,
Var
(A A)
A Ag, A Akaj
m²
L
E [A A AkziA
m2
ky=1
m
[
Σ E AR Aks
m 2
As 54
E [4
A2 + Σ
A ][B [
m 2
ki
(m + 0)
m²
21

---

In the case of i = j. then,
m
m
Var [A (a) 1
2 (47)
Var
(A" A)ai
Var
A
(m 2)
m
m²
m²
m²
m
In the equality (a) above, we used the fact that A ~ (1).. Therefore, limm-+so Var 1 (A A)) =
O.
Now applying Chebyshev's inequality we get,
1 Pr( A A
Tal 22) Var( 1 A))
(48)
m
Obviously for any 2 as m > oc, the R.H.S. goes to zero. Thus, A + Iaxd, in probability.
G.2 Proof of Lemma
Proof. Note that the random vector w is isotropically distributed and that only inner products w Vi and w V2 appear, hence we can assume without loss of generality that (if not, one can rotate the coordinate system to make true):
Vi |1,00,, 0),
V2 = V2 (cos 0 sin 0, 0, 0.. ,)
In this setting, the only relevant parts of w are its first two scalar components wi and w2. Define w as
(wi, W2, 0, .0 V of + w3 sincooss,ssnu sin ,,,..)))
0)-
(49)
Then,
1 { dw
1
P((wTv vi 2 20 A (wTv2 2 20] = P((wTv, 2 0) (W V2 2 0)]
2TT J
2
2m
G.3
Proof of Lemma
Proof. Note that the ReLU activation function (2) can be written as 2Is20- We have,
(n uz) v [l I[W,,2..w.20,WWV
I (W:)) T ( www. W.Ve
90000
2 [v] wI [ (w w v22
21E
Note that the random vector is isotropically distributed and that only inner products wF'v v, and w V2 appear, hence we can assume without loss of generality that (if not, one can rotate the coordinate system make true):
Vi = v1 || (1, 0,0, -....,....))) .).
V2 V2 (cos 0, sin 0, ... .).
In this setting, the only relevant parts of w are its first two scalar components and w2. Define w
w = (1i, W2, O, ... .,, = / / + wy(cosw, sin w. 0,..-,.0
(50)
22

---

Then, in the limit of q + 00,
uu, uz) = ww pxx) 1w w V2) (.. [v] WIll
(WT v220 w [22]
= =/| Eu
||
cos cos(B - -w)d
2T ---
= 2ll 2 4m
(( 0) cos 0 + sin 0)
- =|v=llllvell
((=-0) (( - 0) cos 0 + sin 8)
π
G.4 Proof Lemma E.3
Proof.
2 Σ
A142
2. (U.x)
9 k=1
2- 2 w. /uw
(a)
Fu. (un") Wow. 200.w//ww2www0)
2 En- [un P[(w" v1 2 0) A (w v2 2 0)
(b) π Txx--
In the step (a) above, we used the fact that is independent of W, vi and V2. In the step (b) above, applied Lemma E. and used the fact that (0, [un?] Tsxs-
G.5 Proof of Lemma
Proof. Starting from the definition of the smallest eigenvalue, we have that Amin Ami satisfies
u B'u'u u ||u|2
Amin (B') min
uz0
Σi DE (E
min uy00
Σ u²
)2 Σ EL V2u, Aik (.
min
u/0
Σ1 uf
E 2W,
i== 2( )
min
2
Amin (B).
(51)
2i=1
In the inequality (a) above, we made the following treatment: for each fixed 1, we consider wiI[ A.20)) as the i-th component of a vector u/: by definition, the minimum eigenvalue of matrix B AAT
Amin (B) min ( Bu' |/1l < (u,) But//lul Vj:
(52)
u 70
moreover, this S inequality becomes equality, if and only if all u', are the same and equal to arg noo (u')² Gu'/ u/|l2. It is easy to see, when the dataset is not degenerate, for different j. u', are different, hence only the strict inequality < holds in step (a).
23

---

Continuing from Eq 51 54,51,
we have
((W 201)²
Amin (B')
min u
Amin (B)
Ei=1 "
1=1
Σ 2u
L. 203)
min uşE0
Amin (B)
2im1
2 Amin (B) Amin (B).
min
si=l
u
Ei=1
U
Therefore, we have that Amin (B') > Amin (B).
As for the largest eigenvalue Amaz (B'), can apply the same logic above for Amin (K) (except replacing the min operator by max and have < in step (a)) to get Amaz (B') < Amaz (B).
G.6 Proof of Lemma
Proof. Consider an arbitrary layer 1 E [T.] of the ReLU neural network f at initialization. Given two arbitrary network inputs x, z E Rd, the inputs to the I-th layer are a axl=)(x)) and (z)., respectively.
By definition, we have
(x) (x) v 2 (WL) a (z) = V (IVD
a
(53)
with entries of ( being i..dd drawn from (0, 1). Recall that, by definition, the angle between (x)))(x)) and (1- (((1)(())) (z)) is glt (x, Applying Lemma E.2 we immediately have the inner product
(x))
(x)
1
(IT - g(t (1-11 1) (x,, z)) cos 001 (x, z) + sin ((- (xx,))
π
x |la (1- 1) (xx) a (1 (1-1)
(54)
In the special case of z, we have (xx-1)(x, 2) 0, and obtain from the above equation that
|a (1) (x)]
||a (x)
(55)
Apply (5) back to Eq. (54). we also get
(a (z) a (x))
cos @ (x((xx))) (x,z) [(a(((()) (z)]
( (0--1) (x, 2)) cos (xx1)(xx,)))))))))))))) 2) + sin 01l (xx,))((()))))) 2))
π
(56)
That is (x, 2) = g(000-1) z)). Recursively apply this relation, we obtain the desired result.
G.7 Proof of Lemma E.6
Proof. By Lemma E.5 we have that
cos (0) (x, z) = (1
011 011-1)(x, 2)
1
cos (1- (x, 2) + - sin (1l-1) (x, (x,
cos 0(l-1) (xx
1 tan 0
00 1) (x,z) - g () 2)
cos 0ll (11-1) (x, z)
1
2))3 + to (10
(x, 2) (
3m
Noting that the Taylor expansion of the cos function at zero is cos 2 = 1 22 + o(23), one can easily check that, for all / E [L].
((l) (x, z) g(1 (x. z)
(g(11)((x (x, z))² + (10
(57)
3m
24

---

Note that 800 (x,z) < (11 (x,z) (x, =
o(1/L). Iteratively apply the above equation, one gets. for all
I E o(1/L),
1 1 (x)z))² 2))² + o (xxx))))))
8 (x, 2) = 0 (x, z)
(58)
3
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are accurate summary and high-level descriptions of the scope and the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the last section of the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

The full set of assumptions are explicitly stated in the theorem statements, and the complete proofs are included in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The experimental details that are needed to reproduce the experimental results
are provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: The datasets used are public accessible. The code reproduced following our description experimental settings Section 4.1 Moreover, the experiments only used to justify our theoretical statements.
be easily
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppp wttps/ccccccggguuddeesseeeeeeeee hwwpbll//gguidee/ddeeubbaaiiiaia
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The experimental details are provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This is theoretical paper. The experiments are used for verification of the theoretical claims. Average multiple runs are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resource information is provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ptattps://eeurtii hww.nnuu...../. httphh/ttt.e//aae.....p.////at/a
Answer: [Yes]
Justification:
The authors have read and complied the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is a theoretical work, and there is no socictal impact of the work per- formed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks, as it theoretical work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cited the datasets used in the experiments. The paper does not other existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification:
The paper is not using LLMs for any purpose.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurrs http//rrppsscccooofferreeeee//// Cc/Conffrrecces/2225//II. for what should or should be described.
32


Reviews:
[{"review_id": "kAFbMXN34j", "reviewer": "Reviewer_ndzW", "summary": "The paper discusses properties of neural tangent kernels and the effects of non-linear activation functions compared to linear activations. The field of discussion lies in the application to hyper-wide neural networks and examines the relation between network depth and width and the angular separation of internal representations in model gradient space. The key findings suggest that increasing the network depth significantly enhances the feature separation observed in wide models with non-linear activations. In particular, the authors show that in the infinite-width-then-depth limit, all data are equally separated with an angle ∼75.5 degrees.\n\nThe authors support their claims with detailed mathematical analysis under standard assumptions for wide network limits (e.g., random initialization, asymptotic regimes). They also validate these insights empirically using synthetic and real datasets like MNIST and LibriSpeech. Their findings offer a new perspective on why non-linear activations are crucial for effective learning in deep models.", "questions": "While ReLU is studied in depth, can the theoretical framework be extended to other activation functions and their derivatives? Would the conclusions (and numerical limits) still hold? If not, what aspects of the analysis break down?\n\nWhat are the practical takeaways for model design or training procedures, how your findings can be used in certain settings?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper presents solid theoretical analysis with lemmas and theorems grounded in random matrix theory and kernel methods. Experimental results align well with theoretical predictions and are reproducible via detailed descriptions in Section 4.1.\nThe paper is well-written, logically structured, and accessible to readers familiar with NTK and wide network theory. The claims made in the abstract and introduction match the contributions in the body of the paper.\n\nThe insights into how non-linearities improve optimization dynamics via NTK conditioning and representation learning via feature separation are valuable for understanding modern current DL systems. While not  proposing a novel method, it offers interpretability tool and define specific numerical limits for parameter spaces.\n\nFocus solely on ReLU activations without considering a broad array of possible functions around may miss some insights.", "was_revised": true, "final_justification": "The paper contains interesting results though for a limited scenario. During the rebuttal authors addressed the challenges of generalizing theory, and taking into account they did experiments to empirically observe  similar behavior on other activations, I put my score as slightly above the borderline.", "rebuttal": "We thank the reviewer for the comment. We address your concerns and questions below.\n\n**W1** : “*Focus solely on ReLU activations without considering a broad array of possible functions around may miss some insights.*” and **Q1**: “*... can the theoretical framework be extended to other activation functions and their derivatives? Would the conclusions (and numerical limits) still hold?*\"\n\n**A**: In fact, we considered other activation functions including ReLU², GeLU and Tanh (See Figure 2 and discussions at Line 238). Numerical evaluations show that similar results hold for these activation functions. We believe the results holds more generally. The difficulty for theoretical analysis of these non-ReLU activations is to carry out the integrations similar to those in Line 652 and Line 661. Nevertheless, these integrals can be estimated numerically, as we did in Figure 2.\n\n\n**Q2**: “*What are the practical takeaways for model design or training procedures, how can your findings be used in certain settings?*”\n\n**A**: We consider our paper as a theoretical work that provides insights and understandings of deep learning. Two possible practical implications include: (1) we may need to rethink the current NTK-based optimization theories which often assume a fixed condition number, our paper suggests the activation function can influence this condition number hence the convergence rate. (2) Another implication of our better separation theorem could relate to the origin of adversarial examples where the adversarial and clean examples can be considered as two similar inputs. Our theory indicates that, due to the activation function, they are quite largely separated in the gradient feature space which may attribute to the observed distinct predictions between adversarial and clean examples."}, {"review_id": "7lMxvaxMek", "reviewer": "Reviewer_p5SR", "summary": "This paper studies the effect of non-linear activation functions on the NTK and its associated feature map $x \\mapsto \\nabla_\\theta f_\\theta (x)$ in infinite-width neural networks. \nBased on the fact that feature norms are (virtually) unchanged by linear and ReLU networks, the authors focus on the study of the separation angle between data points and between their features. \n\nThey show that using linear activation functions results in a Gram matrix which is (virtually) unchanged across the hidden layers: the angles between data do not change and the associated NTK has a condition number that corresponds to the Gram matrix of the original data. Conversly, the use of activation functions increases the angles between data with small angles, and improves the NTK condition number. \n\nMore specifically, for ReLU networks, the paper provides a recursion relation for the angle between data points. This recursion allows the authors to study the infinite-width and infinite-depth networks, showing that: \n- All data pairs have an angle that converges to $\\sim 75,5$ degrees. \n- When the data lie on a sphere, the condition number will converge to $\\frac{n+4}{3}$. \n- The condition number decreases with the depth of the network. \n\nThe connection between the condition number and the training speed allows them to discuss implications for convergence speed.  Various experiments support these theoretical findings.", "questions": "- Fast convergence  can occur with a decrease of generalization; for instance, a NTK Gram matrix equal to the identity matrix would lead to fast convergence but no generalization capability. Could you discuss this trade-off explicitly? Specifically, how does generalization behave in Figure 4 (the experiments that illustrate the increase of training speed) ?\n- Relatedly, the experiments results provide an interesting fact that is not discussed in the paper: in the ReLU setting, angles converge to around 75.5 degrees, whereas with other activation functions used in Figure 2, it seems that the angles converges to around 90 degrees, and this could potentially reduce generalization properties. Could you provide the generalisation for these activation function in the setup of Figure 2 and discuss about this depending of what you obtain? Also can you report the limiting angles somewhere?  \n- In corollary 3.3, I suggest you could explicitly mention an insightfull consequence: $L$ should be of order $\\frac{1}{\\theta_{in}}$ for $\\Phi(x,z)$ to be non trivial. Do you agree with this statement? \n- Proof clarifications: \n    - Isn’t Lemma 4.1 a consequence of the Law of Large Numbers, given that each coordinates is the average of m independent variables? \n    - While the result is correct, and equation (40) captures the intuition, I believe it’s inaccurate to claim “solving the above equation (40), we found that …”. One should keep all approximations orders and prove that the l.h.s. is of order $\\frac{1}{L^2}$, thus implying that $1-4\\cos(\\Phi_L) \\to 0$ as $L\\to\\infty$. Do you agree ?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is clearly written and easy to follow, with an equality well-written and well-organized appendix. \n- It provides a fairly comprehensive study of the effect of ReLU activation function  on the NTK with no bias terms. A clear study of these neural network which lies at the boundary between the order and freeze phase was to my knowledge lacking. Its investigates both the separation angles and the features norms, provides recursive relations on angles, analyzes limits and variation of the condition number. The proofs feature interesting random matrices and geometric flavours, taking full advantage of the specific activation function they consider, the ReLU. I particularly like the result concerning the limiting angle,  and whose value is non-trivial (i.e. neither 0 nor 90 degrees).  I think this is one of the main new contributions of the paper, along with the decreasing property of the condition number and its explicit limiting value. The paper thus provides interesting quantitative results on top of the qualitative results. \n- Empicial results show that these theoretical insights can be extended to other activation functions. \n\n\n\nWeaknesses: \n- Link with previous literature is incomplete: \n    - Reference [1] shows that a transition occurs for the NTK between a “freeze” (order) phase, where the scaled NTK converges to a constant, and a “chaos” phase where the NTK looks more like a Dirac Kernel. It was shown that, in term of their characteristic value, ReLU networks without bias lie on the boundary between order and chaos. The current paper extends the study of this phenomena by looking specifically at what happens at the boundary, but does not explicitly mention this connection. \n    - I believe that the linear NTK has been studied before or is classical knowledge (it results directly from the recurrence relation in Theorem 1 of [2]). Check if it was not provided somewhere else.  \n    - The improved conditioning property (though without quantitative results provided in the paper) between linear and non-linear activation were already known. Specifically, the linear NTK has infinite condition number when the number of data points exceeds the dimension. Whereas, as shown in  Appendix A.4 [2], the NTK restricted to the sphere is positive-definite as soon as the activation function is non-polynomial. \n    - Could you discuss also relation with [3]. \n\n- It would be interesting to explicitly discuss the role of bias, as they can significantly affect the behaviour of the NTK. \n- The paper focuses exclusively on the speed of convergence, and does not discuss the generalization performance, despite a trade-off between these two effects. Empirical results exploring generalisation behaviour would be give valuable additional insights.\n\n\n\n[1] - Freeze and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts, Jacot, Gabriel, Ged, Hongler (https://proceedings.mlr.press/v190/jacot22a/jacot22a.pdf)\n\n[2] - Neural Tangent Kernel: Convergence and Generalization in Neural Networks, Jacot, Gabriel, Hongler, ArXiv Version https://arxiv.org/pdf/1806.07572\n\n[3] - Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension Karhadkar, Murray, Montúfar https://arxiv.org/abs/2405.14630", "was_revised": true, "final_justification": "The authors have addressed all my points, in particular those about generalization, relation with previous works, and certain technical details. After reading the other reviews and the authors’ responses, I confirm my score.", "rebuttal": "We thank the reviewer for the insightful and detailed comments. We’d like to address your concerns and questions below.\n\n**W1a**: “*Reference [1] shows that a transition occurs for the NTK between a “freeze” (order) phase ... and a “chaos” phase. The current paper … looking specifically at what happens at the boundary, but does not explicitly mention this connection.*”\n\n**A**: We thank the reviewer for providing this related work. We will cite it in the revision and clarify this connection. Basically, our theory only considered the bias-free setting, which is widely adopted by many prior theoretical works, such as [Du et al. 2019]. After reading [1], we agree that the bias makes a big difference (bigger than we previously thought) and it is interesting to extend the analysis to include the bias term. We also realize that this extension is technically non-trivial (just like observed in [1], different scaling $\\beta$ of the bias leads to distinct results for NTK). We would like to consider it as a future work.\n\n[Du et al. 2019]: Du et al. Gradient Descent Finds Global Minima of Deep Neural Networks. ICML 2019.\n\n**W1b**: “*I believe that the linear NTK has been studied before or is classical knowledge (it results directly from the recurrence relation in Theorem 1 of [2]).*”\n\n**A**: While the linear NTK result, i.e., Corollary 2.3, can be a direct consequence of Theorem 1 of [2], the feature angle separation theorem, Theorem 2.2, does not follow [2]. We didn’t find any prior work having the result of Theorem 2.2, although it is not technically hard. Hence, we put it in the Preliminary section and do not claim it as a contribution. In addition, we will clarify the connection of Corollary 2.3 and Theorem 1 of [2] in the revision.\n\n**W1c**: “*The improved conditioning property…between linear and non-linear activation were already known. Specifically, the linear NTK has infinite condition number when the number of data points exceeds the dimension. Whereas … the NTK restricted to the sphere is positive-definite as soon as the activation function is non-polynomial.*”\n\n**A**: The known result is only for the special case of an infinite linear NTK condition number. In addition, it also requires the data lies in a unit sphere. Our result not only applies for more general settings (e.g., linear NTK is also positive-definite), but, more importantly, focuses on the better separation of gradient features, which is not known from prior works. We’d like to cite [2] and add a discussion about the connection.\n\n**W1d**: “*Could you discuss also relation with [3].*”\n\n**A**: The paper [3] provides both upper and lower bounds for the smallest eigenvalues of NTKs for both shallow and deep NN. First, while these bounds are insightful and valuable, it is different from our results as it is not clear in [3] what effect the activation function has on the smallest eigenvalue. In addition, [3] obtained a linearly increasing upper bound for the  smallest eigenvalue: $\\lambda_{min}< L$. However, note that when depth $L$ increases, NTK accumulates (an addition of $L$ positive definite terms), also resulting in an linear increasing top eigenvalue, hence condition number is still independent of depth $L$. Therefore, we also don’t see a depth-dependent NTK condition number in [3], which is also different from ours.\nMoreover, we conjecture that the bounds obtained in [3] is not tight enough to extract the effect of non-linear activations and depth. As we argued above, upper bounds of [3] would leads to a depth independent bounds for condition number, which hides the depth dependent nature of the true value of condition number, due to a upper bound that is not tight enough. \n\n**W2**: “*It would be interesting to explicitly discuss the role of bias, as they can significantly affect the behaviour of the NTK.*”\n\n**A**: Yes, we agree that it is interesting to discuss the role of bias. With a quick trial, we found that the analysis with different scaling $\\beta$ of the bias is technically non-trivial. Similar to the distinct results observed in [2] for different $\\beta$, we believe the analysis of bias could potentially lead to new observations, which will be quite interesting. We consider it as a future work.\n\n**W3**: “*The paper … does not discuss the generalization performance... Empirical results exploring generalisation behaviour would be give valuable additional insights.” and  Q1: “Fast convergence can occur with a decrease of generalization…. Could you discuss this trade-off explicitly? Specifically, how does generalization behave in Figure 4 (the experiments that illustrate the increase of training speed) ?*”\n\n**A:** We agree there is often a trade-off between optimization and generalization, especially when depth is large. We will add discussions about this trade-off in the revision.\n\nAs our theory suggests in Theorem 3.5 and Theorem 4.3, in the extreme case of infinite depth, any input pairs become equally separated in gradient features regardless of their original similarity. Even though not mutually orthogonal, this could also result in a trivial generalization: close to random guess for unseen data. This somehow can also be obtained from [Wang et al. 2020], where they dropped the initial random guess value and obtained a zero prediction for unseen data. We will cite this paper in the revision.\n\nAs for finite depth, there is no theory that clearly suggests at what depth this trade-off starts to happen. In our experiments in Figure 4, we can slightly see this trade-off. For example, for MNIST, the test accuracies for different depths $L$ are:\n\n     95.98% (L=1),  97.43% (L=3),  97.57% (L=6),  97.52% (L=8),  97.39% (L=10),  97.19% (L=12)\n\nAbove, we can see a decreased generalization starting from a certain depth (in this case around $L=8$).\n\n[Wang et al. 2020]: Wang et al. Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? — A Neural Tangent Kernel Perspective. NeurIPS. 2020.\n\n\n**Q2**: “*... in the ReLU setting, angles converge to around 75.5 degrees, whereas with other activation functions used in Figure 2, it seems that the angles converges to around 90 degrees, and this could potentially reduce generalization properties. Could you provide the generalisation for these activation functions in the setup of Figure 2 and discuss this depending on what you obtain? Also can you report the limiting angles somewhere?*”\n\n**A**: As mentioned above, very deep ReLU NNs already have a reduced generalization performance and converge to random guess at infinite depth, hence we don’t think the difference between $75.5$ degree for ReLU and $90$ degree for others make much difference in generalization. \n\nWe will also add the (numerical-based) observations about the limiting angles of other activation functions in the revision.\n\n\n\n**Q3**: “*In corollary 3.3, I suggest you could explicitly mention an insightful consequence: $L$ should be of order $1/\\theta_{in}$ for $\\phi$ to be non trivial. Do you agree with this statement?*”\n\n**A**: Yes. We agree. We will update it in the revision.\n\n\n**Q4-1**: “*Isn’t Lemma 4.1 a consequence of the Law of Large Numbers …*”\n\n**A**: There is no Lemma 4.1 in our submission. We’d like to ask which lemma you refer to. We are happy to make further clarification after.\n\n**Q4-2**: “*I believe it’s inaccurate to claim ‘solving the above equation (40), we found that …’  …Do you agree?*”\n\n**A**: Thanks for pointing out this issue. Yes, you are correct. We will update our proof in the revision. Basically, based on Eq. (40), we can see that $1/4$ is an attraction point for the series $\\cos \\phi_L$ (look at the sign of $Delta \\cos \\phi_L$). With a bit of effort we can show that the series converges. Then, suppose $\\cos \\phi_L$ converges to a point not equal to $1/4$, the l.h.s. of Eq, (40) will be at the order of $1/L$, which results in a non-converging $\\cos \\phi_L$, contradictory. Hence, we can show that the series converges to $1/4$."}, {"review_id": "zL3yghocnV", "reviewer": "Reviewer_sTQk", "summary": "This paper presents simple results on the tangent feature map of deep neural networks: (1) non-linear activations cause the angle in tangent feature space to be larger than input angles, and (2) this effect is amplified with depth. Importantly, this effect is not observed in linear networks. A natural consequence of this increased angle is improved conditioning of the NTK. This phenomenon is also empirically verified on datasets using finite width neural networks.", "questions": "1. Can the authors comment on finite width effects?\n2. Further, instead of increasing depth, could similar results be derived as a function of width for a fixed depth?", "limitations": "Yes, the authors have addressed limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The result is simple and presented in a straightforward and easy to follow manner.\n2. It provides a novel perspective and concrete explanation (as far as this reviewer is aware) for the utility of non-linear activation functions. \n3. Theoretical results seem sound and are numerically verified as well.\n\nWeaknesses:\n\n1. The theory depends on infinite width of the network, and is restricted to ReLU activations.\n2. The section on optimization acceleration could benefit from more quantitative analysis. For example, a quantitative convergence improvement as a function of depth, informed by the main result would be insightful. Especially if this quantitative analysis is corroborated by the increasing depth experiment that is presented. Currently, the faster convergence for higher depths is qualitative and does not build on the main result in a theoretical capacity.\n3. Some of the writing could be slightly improved", "was_revised": true, "final_justification": "After going through the detailed discussions between the reviewers, AC, and the authors, I will maintain my original score. While there are some concerns regarding scope of settings in which their NTK results hold, I believe that the results in the current state of the paper are significant and novel enough and hence justify my score of 4.", "rebuttal": "We thank the reviewer for the comment. We address your concerns and questions below.\n\n**W1**: “*The theory depends on infinite width of the network, and is restricted to ReLU activations.*” and **Q1**: “*Can the authors comment on finite width effects?*”\n\n**A**: Our theory is conducted in a sufficiently large but finite width setting (See for example Theorem 3.2). Basically, our theory is based on the well-established large-width NN analysis (such as [8,9,15,17,19] in the reference list of our paper), where the NN deviates from an infinitely wide NN slightly (in the order of $O(1/\\sqrt{m})$). While there is not much theory for smaller network width, we temporarily only consider this  sufficiently large but finite width setting. \n\nIn fact, we considered other activation functions including ReLU², GeLU and Tanh (See Figure 2 and its discussion). Numerical evaluations show that similar results hold for these activation functions. The difficulty for theoretical analysis of these non-ReLU activations is to carry out the integrations similar to those in Line 652 and Line 661. Nevertheless, these integrals can be estimated numerically.\n\n\n**W2:** “*The section on optimization acceleration could benefit from more quantitative analysis. For example, a quantitative convergence improvement as a function of depth, informed by the main result would be insightful. … Currently, the faster convergence for higher depths is qualitative and does not build on the main result in a theoretical capacity.*”\n\n**A**: We agree that a quantitative result on the optimization acceleration would be more interesting. While we can get a quantitative result in the simple setting of data size equal $2$ based on the quantitative result of feature separation – Corollary 3.3, the major difficulty for general case is that the connection between the smallest eigenvalue of NTK and data feature separation is still not quantitatively clear, therefore, the quantitative result of feature separation downgrades to a qualitative result of NTK. We conjecture that a sophisticated analysis involving random matrix theories can achieve this goal. We consider it as a future work. \n\n\n**Q2**: “*Further, instead of increasing depth, could similar results be derived as a function of width for a fixed depth?*”\n\n**A**: Lemma 3.1 and Theorem 3.2 show that, with a fixed depth, the width contributes minimally compared to a width-independent term (see for example Eq. (7)). Our paper currently focuses on the leading term (i.e., the width-independent term). As for the width-*dependent* term, we believe it also depends on the initialization seed, in which case we only obtain an upper bound."}, {"review_id": "IEyPrjpg3g", "reviewer": "Reviewer_L1pt", "summary": "This paper reveals that non-linear activation functions not only enhance expressivity but also improve feature separation and NTK conditioning in wide neural networks. These effects are amplified with network depth, facilitating convergence of gradient-based methods. In contrast, linear networks lack these advantages, preserving the original data geometry and exhibiting poorly conditioned NTKs.  While the theoretical analysis focuses on ReLU, empirical results with other activations support the claims.", "questions": "What are the specific technical challenges in extending the theoretical analysis to other activation functions? Is it because, for most non-linear activations, it is difficult to obtain a closed-form expression for the entries of the corresponding NTK? If so, have you considered analyzing the Hermite coefficients? Perhaps the higher-order coefficients could be bounded or even omitted.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1 This paper is technically sound, The claims are well supported by theoretical analysis on ReLU and by empirical observations on other activations.\n\n2 The paper is well-written and clearly presented, making its theoretical insights and empirical findings easy to follow.\n\n3 The results are important for understanding the role of non-linear activations. The paper provides a comprehensive theoretical analysis demonstrating that the ReLU activation improves  feature separation and NTK conditioning. To the best of my knowledge, these findings are new.\n\nWeaknesses\n\n1 The title should be specific to ReLU activations as the main contribution of this paper is the theoretical findings, which is only conducted on ReLU.\n\n2 This paper demonstrates that as depth increases, the condition number of the NTK matrix with ReLU improves. However, many previous works [1] [2] show that as depth increases, the NTK degenerates. That is, the infinite depth limit of the NTK regime is trivial.\n\n3 To be honest, I acknowledge that the findings of this paper are novel, but not entirely surprising. While the theoretical analysis provides valuable insights, it is limited to ReLU activations. It seems that extending the results to other widely used activation functions may involve technical challenges. If the analysis could be extended to arbitrary non-linear activations, it could offer valuable guidance for the design of new activation functions.\n\n[1] Hayou S, Doucet A, Rousseau J. The curse of depth in kernel regime.\n\n[2] Bietti A, Bach F. Deep equals shallow for ReLU networks in kernel regimes.", "was_revised": true, "final_justification": "The rebuttal by the authors has addressed mostof my concerns. Thus, I raise my score", "rebuttal": "We thank the reviewer for the comment. We address your concerns and questions below.\n\n**W1**: “*The title should be specific to ReLU activations as the main contribution of this paper is the theoretical findings, which is only conducted on ReLU.*”\n\n**A**: We thank the reviewer for this important suggestion. We will update the title in the revision to include ReLU. Please also note that we present numerical results for other types of activation functions in section 3 (seen in Figure 2).\n\n**W2**: “*This paper demonstrates that as depth increases, the condition number of the NTK matrix with ReLU improves. However, many previous works [1] [2] show that as depth increases, the NTK degenerates. That is, the infinite depth limit of the NTK regime is trivial.*”\n\n**A**: Yes. As our Theorem 4.3 suggests, the NTK becomes trivial for the infinite depth limit, which is consistent with [1,2]. But our main focus is on the finite depth scenario, where NTK and gradient features may not be trivial. At a practical depth, a better feature separation can make classification tasks easier as there is less need to fine-tune the decision boundaries. We focus on how different depths lead to different levels of gradient feature separation and how it affects the NTK matrix.\n\n\n**W3**: “*.... If the analysis could be extended to arbitrary non-linear activations, it could offer valuable guidance for the design of new activation functions.*” and **Q1**: “*What are the specific technical challenges in extending the theoretical analysis to other activation functions?... have you considered analyzing the Hermite coefficients? Perhaps the higher-order coefficients could be bounded or even omitted.*”\n\n**A**: Yes, the technical challenge is to carry out a few integrations for other activations (similar to the ones in Line 652 and Line 661). Instead, we numerically estimated them and showed the results for non-ReLU functions in Figure 2.\n\nWe appreciate your valuable suggestion on bounding or omitting higher-order Hermite coefficients. We will try it."}, {"review_id": "MPMyz8h7uK", "reviewer": "Reviewer_gxGU", "summary": "The authors investigate the advantages of using ReLU activation in the Neural Tangent Kernel (NTK) regime of fully connected neural networks. They demonstrate that ReLU improves data separation in the gradient\nfeature space, leading to better conditioning of the NTK compared to linear activations. This improvement becomes more pronounced as network depth increases. Their findings are supported by experiments on both synthetic and standard benchmark datasets.", "questions": "See above.", "limitations": "1. What are the practical implications of your theoretical findings?\n2. Could you clarify how the separation property of the data points relates to their corresponding labels?\n3. Figure 2 provides numerical evidence that the separation phenomenon also holds for various non-linear activation functions. However, the behaviour differs across ReLU², GeLU, and Tanh. Can your theoretical framework be extended to accommodate other non-linear activations and account for the observed differences?", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe authors propose a new direction for investigation, focusing on characterizing the angles between data points in the NTK feature space. Additionally, they examine the spectral properties of the NTK, with particular attention to its condition number. Their theoretical insights are supported by a thorough experimental evaluation, showing agreement between empirical findings and theoretical predictions.\n\nWeaknesses\n\nThe main limitation of the paper lies in the narrow scope of the comparison. Specifically, the focus is on ReLU versus linear networks. The theoretical analysis emphasizes that ReLU networks yield greater angle\nseparation for similar data points in the NTK feature space. However, the impact of non-linear activations in neural networks is already well established, making this result less surprising. A more compelling \ndirection would have been to explore and compare a broader range of activation functions.\n\nIn this context, the setup of the trainable networks is rather restricted and lacks practical applicability. Specifically, in Theorem 2.2 (linear network), the feature space is attributed to the gradient at initialization.\nIn Theorem 4.2 (non-linear network), only the first layer is trainable, while all subsequent layers remain fixed.\n\nThroughout the manuscript, the intuition underlying the separation property is not clearly expressed. In fact, for similar data points, increased separation may be undesired, especially when the labels are the same. Yet, the manuscript does not explore how this separation behaviour is influenced by the labelling of the data.\n\nThe spectral properties of the NTK have been extensively studied in prior work. This manuscript focuses specifically on the condition number of the NTK, which is closely tied to its smallest eigenvalue. However, the comparison is again limited to the simplistic case of linear networks. It remains unclear what the practical implications of this comparison are. How does this insight benefit neural network training or inference? \n\nMoreover, while the manuscript claims that the condition number is small (line 57), it appears to be quite large when applied to larger datasets.\n\nProposition 4.1 has limited applicability, as it is demonstrated only in the context of small datasets where the number of samples n is less than the dimensionality d.", "was_revised": true, "final_justification": "While the authors’ response only partially addresses my concerns, I think the paper is worth publishing, and I maintain my original score.", "rebuttal": "We thank the reviewer for the comment. We address your concerns and questions below.\n\n**Re**: “*The main limitation of the paper lies in the narrow scope of the comparison. Specifically, the focus is on ReLU versus linear networks.… A more compelling direction would have been to explore and compare a broader range of activation functions.*”\n\n**A**: We’d like to point out that our goal is to extract the influence by the existence of non-linear activations. The only way to fulfil this is to directly compare non-linear activated NN with linear NN. Comparing among different non-linear activations does not work for this goal. Analogously, please think about another important property of non-linear activations – increasing expressivity of NNs: with non-linear activations NN can potentially approximate any continuous function; without it is basically a linear model. \n\nIn addition, we indeed considered other activation functions, as shown in Figure 2. Again, these need to be compared with linear NNs in order to see the influence of these activations.\n\n**Re**: “*... the impact of non-linear activations in neural networks is already well established, making this result less surprising.*”\n\n**A**: We didn’t find any similar work in the literature, and we don't think our result is previously known. We’d like to ask the reviewer to provide additional clarification and references. \n\n**Re**: “*... in Theorem 2.2 (linear network), the feature space is attributed to the gradient at initialization.*”\n\n**A**: Due to the constancy of NTK [Jacot et al. 2018] and transition to linearity [Liu et al. 2020], the gradient features during and after training keep the same as at initialization. Hence, an analysis at initialization is enough.\n\n[Jacot et al. 2018] Jacot et al. Neural tangent kernel: Convergence and generalization in neural networks, NeurIPS, 2018.\n\n[Liu et al. 2020] Liu et al. On the linearity of large non-linear models: when and why the tangent kernel is constant. NeurIPS, 2020.\n\n**Re**: “*In Theorem 4.2 (non-linear network), only the first layer is trainable, while all subsequent layers remain fixed.*”\n\n**A**: It is often an acceptable setting in theoretical works to fix upper layers to simplify analysis. See for example [Du et al. 2019]. Indeed, we have a section in Appendix F to relax this constraint on upper layers, and similar results also hold. In addition, in our numerical results, all layers are trainable and results are consistent with our theory. Hence, we believe the result should still hold when the constraint is removed.\n\n[Du et al. 2019] Du et al. Gradient Descent Provably Optimizes Over-parameterized Neural Networks, ICLR, 2019.\n\n\n**Re**: “*In fact, for similar data points, increased separation may be undesired, especially when the labels are the same.*” and **Q2**: “*Could you clarify how the separation property of the data points relates to their corresponding labels?*”\n\n**A**: We don’t quite think  increasing separation is undesired when the labels are the same. In fact, for the training data in the same class, whether there is more or less separation among them does not really matter. The classification task is to find decision boundaries between different classes. Therefore, as long as the feature separation is increased for those with different labels so that the decision boundaries need less fine-tuning, the optimization becomes easier. \n\n**Re**: “*The spectral properties of the NTK have been extensively studied in prior work. This manuscript focuses specifically on the condition number of the NTK, which is closely tied to its smallest eigenvalue.*”\n\n**A**: As we also mentioned in the related works, there are extensive studies on NTK spectrum and its smallest eigenvalues. Our work differs from the prior works in the sense that we are the first to investigate the influence of the existence of non-linear activations. We also have the better separation phenomenon (Section 3) which cannot be found anywhere else. Our analysis techniques are also different.\n\n**Re**: “*while the manuscript claims that the condition number is small (line 57), it appears to be quite large when applied to larger datasets*”\n\n**A**: We apologize for not making it crystal clear.  Generally, the condition number is extremely large due to an infinitesimal smallest eigenvalue, and is far greater than the data size $n$. In Line 57, we say it is small, meaning it becomes much smaller. We will make this point clear in the revision.\n\n**Re**: “*Proposition 4.1 has limited applicability, as it is demonstrated only in the context of small datasets where the number of samples n is less than the dimensionality d.*”\n\n**A**: When $n$ is greater than $d$, the smallest eigenvalue of linear NTK becomes zero and its condition number is infinite. The better separation, Theorem 3.2, and better NTK conditioning, Theorem 4.2, still hold.\n\n**Q1**: “*What are the practical implications of your theoretical findings?*”\n        \n**A**: We consider our paper as a theoretical work that provides insights and understandings of deep learning. Two possible practical implications include: (1) we may need to rethink the current NTK-based optimization theories which often assume a fixed condition number, our paper suggests the activation function can influence this condition number hence the convergence rate. (2) Another implication of our better separation theorem could relate to the origin of adversarial examples where the adversarial and clean examples can be considered as two similar inputs. Our theory indicates that, due to the non-linearity of activation function, they are quite largely separated in the gradient feature space which may attribute to the observed distinct class predictions between adversarial and clean examples.\n\n\n**Q3**: “*Figure 2 provides numerical evidence that the separation phenomenon also holds for various non-linear activation functions. However, the behaviour differs across ReLU², GeLU, and Tanh. Can your theoretical framework be extended to accommodate other non-linear activations and account for the observed differences?*”\n\n**A**: The main message we had in Figure 2 is that, similar to the ReLU case, for small input angles (i.e., left end of each picture), non-linearly activated NNs are better separated; and when depth increases, the separation is larger. Therefore, the behavior is not significantly different from ReLU. \n\nAs for the extension beyond ReLU, the technical difficulty is to carry out a few integrations for other activations (similar to the ones in Line 652 and Line 661). In the paper, we numerically estimated them and showed the results in Figure 2. Therefore, if there exist close-form expressions for these integrations, our theory will extend to those non-ReLU activation functions."}]

Author rebuttal:

Meta-review: This paper discovers that non-linear activations, compared with linear networks, lead to larger feature separation in the gradient feature space and improved NTK conditioning in the infinite-width limit. The theoretical analysis focuses on ReLU while empirical observations are consistent across other activations.

The reviewers acknowledge that the discovery is simple (in the good sense), novel and insightful, supported by rigorous and non-trivial analysis, with interesting quantitative results. The core contributions are clearly presented with high-quality formulations, statements, and informative experiments. The paper is easy to read despite the theoretical depth involved.

The current analysis is restricted to ReLU and the NTK regime, which limits the generality of the formal results. This should be better reflected in the title and/or abstract. While the discovery is interesting, its implications for applications --- including how practitioners could benefit from the better separation --- remain unclear.

All five reviewers recommend acceptance with high confidence, and so does the AC.
Decision: Accept (poster)