Title: Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension
Venue: NeurIPS 2024
Accepted: yes
Keywords: neural tangent kernel

initialization

minimum eigenvalue

smallest eigenvalue

low-dimensional

hemisphere transform

spherical harmonics

separated
Abstract: Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension $d_0$ scales at least logarithmically in the number of samples $n$. In this work we remove both of these requirements and instead provide bounds in terms of a measure of distance between data points: notably these bounds hold with high probability even when $d_0$ is held constant versus $n$. We prove our results through a novel application of the hemisphere transform.

Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension

Abstract

Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension d 0 scales at least logarithmically in the number of samples n . In this work we remove both of these requirements and instead provide bounds in terms of a measure of distance between data points: notably these bounds hold with high probability even when d 0 is held constant versus n . We prove our results through a novel application of the hemisphere transform.

1 Introduction

A popular approach for studying the optimization dynamics of neural networks is analyzing the neural tangent kernel (NTK), which corresponds to the Gram matrix obtained from the Jacobian of the network parametrization map (Jacot et al., 2018). When the network parameters are adjusted by gradient descent, the network function follows a kernel gradient descent in function space with respect to the NTK. By bounding the smallest eigenvalue of the NTK away from zero it is possible to obtain global convergence guarantees for gradient descent parameter optimization (Du et al., 2019b; Oymak & Soltanolkotabi, 2020) as well as results on generalization (Arora et al., 2019a; Montanari &Zhong, 2022) and data memorization capacity (Montanari & Zhong, 2022; Nguyen et al., 2021; Bombari et al., 2022). These key advances highlight the importance of deriving tight, quantitative bounds for the smallest eigenvalue of the NTK at initialization.

While initial breakthroughs on the convergence of gradient optimization in neural networks (Li & Liang, 2018; Du et al., 2019a; Allen-Zhu et al., 2019) required unrealistic conditions on the width of the layers, subsequent and substantive efforts have reduced the level of overparametrization required to ensure that the NTK is well conditioned at initialization (Zou & Gu, 2019; Oymak & Soltanolkotabi, 2020). In particular, Nguyen (2021); Nguyen et al. (2021); Banerjee et al. (2023) showed that layer width scaling linearly in the number of training samples n suffices to bound the smallest eigenvalue and Montanari & Zhong (2022); Bombari et al. (2022) obtained results for networks with sub-linear layer width and the minimum possible number of parameters ˜ Ω( n ) up to logarithmic factors. However, and as discussed in Section 2, the bounds provided in prior works require that the data is drawn from a distribution satisfying a Lipschitz concentration property, and only hold with high probability if the input dimension d 0 scales as √ n (Bombari et al., 2022) or polylog ( n ) (Nguyen et al., 2021). These existing results therefore require that the dimension of the data grows unbounded as the number of training samples n increases and as such there is a gap in our understanding of cases where the data is sampled from a fixed, or lower-dimensional space.

In this work we present new lower and upper bounds on the smallest eigenvalue of a randomly initialized, fully connected ReLU network: compared with prior work, our results hold for arbitrary

data on a sphere of arbitrary dimension. Our techniques are novel and rely on the hemisphere transform as well as the addition formula for spherical harmonics.

We study neural networks denoted as functions f : R d 0 ×P → R , where P is an inner product space. To be clear, f ( x ; θ ) denotes the output of the network for a given input x ∈ R d 0 and parameter choice θ ∈ P . For brevity we occasionally write f ( x ) in place of f ( x ; θ ) if the context is clear. We use n to denote the size of the training sample, d 0 the dimension of the input features, L the network depth, d l the width of the l th layer and σ : R → R the ReLU activation function. Given n input data points x 1 , · · · , x n ∈ R d 0 we write X = [ x 1 , · · · , x n ] ∈ R d 0 × n and define F : P → R n to be the evaluation of the network on these n data points as a function of the parameter θ ,

We define the neural tangent kernel (NTK) of F as

where the gradient ∇ and adjoint ∗ are taken with respect to the inner product on P and the Euclidean inner product on R n . More explicitly [ K ( θ )] ik = ⟨∇ θ f ( x i ; θ ) , ∇ θ f ( x k ; θ ) ⟩ . For convenience we write K in place of K ( θ ) . We are concerned with the minimum eigenvalue λ min ( K ) , which depends both on the input data X and the parameter θ . We say the dataset x 1 , · · · , x n is δ -separated for δ ∈ (0 , √ 2] if min i = k min( ∥ x i -x k ∥ , ∥ x i + x k ∥ ) ≥ δ , which is a measure of distance in direction.

̸

Main contributions. Our results are for data that lies on a sphere and is δ -separated for some δ ∈ (0 , √ 2] . Unlike prior work we do not make any assumptions on the distribution from which the data is sampled, e.g., uniform on the sphere or Lipschitz concentrated, and we do not require the input dimension d 0 to scale with the number of samples n .

̸

In Theorem 1 we consider shallow ReLU networks with input dimension d 0 and hidden width d 1 and prove that if d 1 = ˜ Ω( ∥ X ∥ 2 d 3 0 δ -2 ) then with high probability λ min ( K ) = ˜ Ω( d -3 0 δ 2 ) . Furthermore, defining δ ′ = min i = k ∥ x i -x k ∥ , we have λ min ( K ) = O ( δ ′ ) .

In Theorem 8 we illustrate how our results for shallow networks can be extended to cover depthL networks. In particular, if the layer widths satisfy a pyramidal condition, meaning d l ≥ d l +1 for l ∈ { 1 , · · · , L -1 } , d L -1 ≳ 2 L log( nL/ϵ ) and d 1 = ˜ Ω( nd 3 0 δ -4 ) , then λ min ( K ) = ˜ Ω( d -3 0 δ 4 ) and λ min ( K ) = O ( L ) with high probability.

Our results allow us to analyze the smallest eigenvalue of the NTK for data drawn from any distribution for which one can establish δ -separation with high probability in terms of d 0 and n . For example, for shallow networks with data drawn uniformly from a sphere, in Corollary 2 we show that if d 0 d 1 = ˜ Ω( n 1+4 / ( d 0 -1) ) , then with high probability λ min ( K ) = ˜ O ( n -2 / ( d 0 -1) ) and λ min ( K ) = ˜ Ω ( n -4 / ( d 0 -1) ) . Moreover, this bound is tight up to logarithmic factors for d 0 = Ω(log( n )) matching prior findings for this regime.

The rest of this paper is structured as follows: in Section 2 we provide a summary of related works and compare and contrast our results with the existing state of the art; in Section 3 we present our results for shallow networks; finally in Section 4 we extend our shallow results to the deep case.

Notations. With regard to general points on notation we let [ n ] = { 1 , 2 , · · · , n } denote the set of the first n positive integers. If x ∈ R d then we let [ x ] i denote the i th entry of x . If f and g are real-valued functions, we write f ≲ g or f = O ( g ) when there exists an absolute constant C such that f ( x ) ≤ Cg ( x ) for all x . Similarly, we write f ≳ g or f = Ω( g ) when there exists a constant c such that f ( x ) ≥ cg ( x ) for all x . We write f ≍ g when f ≲ g and f ≳ g both hold. The notation ˜ Ω hides logarithmic factors. Logarithms are generally considered to be in base e , though in most settings the particular choice of base can be absorbed by a constant.

2 Related work

Prior work on the NTK. Jacot et al. (2018) highlight that the optimization dynamics of neural networks are controlled by the Gram matrix of the Jacobian of the network function, an object referred to as the NTK Gram matrix, or, as we refer to it here, simply the NTK. That work also shows that

in the infinite-width limit the NTK converges in probability to a deterministic kernel. Of particular interest is the observation that in the infinite-width setting the network behaves like a linear model (Lee et al., 2019). Further, if a network is polynomially wide in the number of samples then the smallest eigenvalue of the NTK can be lower bounded in terms of the smallest eigenvalue of its infinite-width analog. As a result, assuming the latter is positive, global convergence guarantees for gradient descent can be obtained (Du et al., 2019a,b; Allen-Zhu et al., 2019; Zou & Gu, 2019; Lee et al., 2019; Oymak & Soltanolkotabi, 2020; Zou et al., 2020; Nguyen & Mondelli, 2020; Nguyen, 2021; Banerjee et al., 2023). The positive definiteness of the NTK is equivalent to the Jacobian having full rank, which can also be used to study the loss landscape (Liu et al., 2020, 2022; Karhadkar et al., 2023). Beyond the smallest eigenvalue, there is interest in characterizing the full spectrum of the NTK (Basri et al., 2019; Geifman et al., 2020; Fan & Wang, 2020; Bietti & Bach, 2021; Murray et al., 2023), which has implications on the dynamics of the empirical risk (Arora et al., 2019b; Velikanov &Yarotsky, 2021) as well as the generalization error (Cao et al., 2021; Basri et al., 2020; Cui et al., 2021; Jin et al., 2022; Bowman & Montúfar, 2022). Finally, although a powerful and successful tool for analyzing neural networks it must be noted that the NTK has limitations, most notably perhaps that it struggles to explain the rich feature learning commonly observed in practice (Lee et al., 2020a; Chizat et al., 2019; Liu et al., 2020).

Prior work on the smallest eigenvalue of the NTK. Many of the prior works discussed so far assume or prove that λ min ( K ) is positive, but do not provide a quantitative lower bound. Here we discuss works seeking to address this issue and to which we view our work as complementary. For shallow ReLU networks and data drawn uniformly from the sphere, Xie et al. (2017, Theorem 3) and Montanari & Zhong (2022, Theorem 3.2) provide lower bounds on the smallest singular and eigenvalue value of the Jacobian and NTK respectively. In addition to requiring the data to be drawn uniform from the sphere both of these results are high dimensional in the sense that for Xie et al. (2017, Theorem 3) to be non-vacuous it is necessary that d 0 = Ω( d 1 n 2 ) , while Montanari & Zhong (2022, Theorem 3.2) requires, as per their Assumption 3.1, that d 0 = ˜ Ω( √ n ) .

Nguyen et al. (2021, Theorem 4.1) derives lower and upper bounds for the smallest eigenvalue of the NTK for deep ReLU networks under standard initialization conditions assuming the data is drawn from a distribution satisfying a Lipschitz concentration property. They show that the NTK is well conditioned if the network has a layer of width of order equal to the number of data points n up to logarithmic factors. Concretely, if at least one layer has width linear in n (ignoring logarithmic factors) and the others are at least poly-logarithmic in n , then λ min ( K ) = Ω( µ 2 r ( σ ) d 0 ) (or Ω( µ 2 r ( σ )) with normalized data), where µ r ( σ ) denotes the r th Hermite coefficient of σ with any even integer r ≥ 2 . However, in their result the bound holds with high probability only if d 0 scales as log( n ) .

Bombari et al. (2022, Theorem 1) derive lower and upper bounds for the smallest eigenvalue of the NTK under similar conditions as Nguyen et al. (2021, Theorem 4.1) aside from the following: they consider smooth rather than ReLU activation functions, the widths follow a loose pyramidal topology, meaning d l = O ( d l -1 ) for all l ∈ [ L -1] , d L -1 d L -2 scales linearly in n (ignoring logarithmic factors), and there exists a γ > 0 such that n γ = O ( d L -1 ) . Under these conditions they show that λ min ( K ) = Ω( d L -1 d L -2 ) with high probability as both d L -1 and n grow. This result illustrates that for the NTK to be well conditioned it suffices that the number of neurons grows as ˜ Ω( √ n ) . The loose pyramidal condition on the widths implies d L -1 d L -2 = O ( d 2 0 ) and as they also assume that n = o ( d L -1 d L -2 ) then n = o ( d 2 0 ) which in turn implies d 0 = Ω( √ n ) .

The rough strategy used by both Bombari et al. (2022) and Nguyen et al. (2021), as well as in our own results, can be described in terms of two main steps. In the first step, one bounds the smallest eigenvalue of a shallow network. The results for the shallow case can then be extended to the deep case, e.g., via a layerwise decomposition of the NTK matrix. This second step is architecture-dependent and its proof depends on the bounds derived in the first step. Our results focus on improving the first step which imply corresponding improvements for the second step.

3 Shallow networks

Here we study the smallest eigenvalue of the NTK of a shallow neural network. The parameter space P of this network is R d 1 × d 0 × R d 1 and it is equipped with the inner product

For convenience we sometimes write d = d 0 . The neural network f : R d 0 ×P → R is defined as

where W = [ w 1 , · · · , w d 1 ] T ∈ R d 1 × d 0 are the inner layer weights, v = [ v 1 , · · · , v d 1 ] T ∈ R d 1 the outer layer weights, and θ = ( W , v ) . We consider the ReLU activation function applied entrywise with σ ( z ) = max { 0 , z } . The derivative ˙ σ satisfies ˙ σ ( z ) = 1 for z > 0 and ˙ σ ( z ) = 0 for z < 0 . Although σ is not differentiable at 0, we take ˙ σ (0) = 0 by convention. Unless otherwise stated we assume that the entries of W and v are drawn mutually iid from a standard Gaussian distribution N (0 , 1) . Our main result for shallow networks is the following theorem.

̸

Theorem 1. Let d ≥ 3 , ϵ ∈ (0 , 1) , and δ, δ ′ ∈ (0 , √ 2) . Suppose that x 1 , · · · , x n ∈ S d -1 are δ -separated and min i = k ∥ x i -x k ∥ ≤ δ ′ . Define

If d 1 ≳ ∥ X ∥ 2 λ log n ϵ , then with probability at least 1 -ϵ ,

A proof of Theorem 1 is provided in Appendix C.7. Suppressing logarithmic factors, Theorem 1 implies that d 1 = ˜ Ω ( ∥ X ∥ 2 d 3 0 δ -2 ) suffices to ensure that λ min ( K ) = ˜ Ω( d -3 0 δ 2 ) and λ min ( K ) = O ( δ ′ ) with high probability (note the trivial bound ∥ X ∥ 2 ≤ ∥ X ∥ 2 F ≤ n ). We emphasize that unlike existing results i) we make no distributional assumptions on the data, instead only assuming a milder δ -separated condition, and ii) our bounds hold with high probability even if d 0 is held constant.

A few further remarks are in order. First, the condition d 0 ≥ 3 is necessary because our technique relies on the addition formula for spherical harmonics (Efthimiou & Frye, 2014, Theorem 4.11); the bound we derive based on this formula (Lemma 15 in Appendix A.2) becomes vacuous for d 0 < 3 . However, for d 0 = 2 analogous bounds could be derived using more elementary tools while the case d 0 = 1 is of little interest as only a trivial dataset is possible. Moreover, data in S 1 could be embedded in S 2 since we do not impose any distributional assumptions.

Second, one can use Theorem 1 to bound the smallest eigenvalue of the NTK for data drawn from the uniform distribution on the sphere by bounding δ with high probability in terms of n and d . We use that δ = Ω( n -2 /d 0 ) and δ ′ = O ( n -2 /d 0 ) with high probability. We direct the interested reader to Appendix C.8 for further details.

Corollary 2. Let d ≥ 3 , n ≥ 2 , ϵ ∈ (0 , 1) , x 1 , · · · , x n ∼ U ( S d -1 ) be mutually iid. Define

.

If d 1 ≳ 1 λ ( 1 + n +log(1 /ϵ ) d ) log n ϵ , then with probability at least 1 -ϵ over the data and network parameters,

.

The above corollary implies that if d 0 d 1 = ˜ Ω ( n 1+4 / ( d 0 -1) ) , then with high probability λ min ( K ) = ˜ Ω( n -4 / ( d 0 -1) ) and λ min ( K ) = ˜ O ( n -2 / ( d 0 -1) ) . In particular, for data sampled uniformly from a sphere, the scaling d 0 = Ω(log n ) is both necessary and sufficient for λ min ( K ) to be ˜ Θ(1) . In particular the bounds are sharp in this case.

3.1 Proof outline for Theorem 1

Recall the definitions of F ( θ ) and K in (1). For the choice of f given in (2), a straightforward decomposition of the NTK with respect to the inner and outer weights gives

where K 1 = ∇ W F ( θ ) ∗ ∇ W F ( θ ) and K 2 = ∇ v F ( θ ) ∗ ∇ v F ( θ ) = 1 d 1 σ ( WX ) T σ ( WX ) . As both K 1 and K 2 are positive semi-definite,

see, e.g., Horn & Johnson (2012, Theorem 4.3.1). Our proof now follows the highlighted steps below.

1) Bound the smallest eigenvalue in terms of the infinite-width limit. We proceed to bound both λ min ( K 1 ) and λ min ( K 2 ) in terms of the smallest eigenvalues of their infinite-width counterparts, see Lemmas 3 and 4 below, which act as good approximations for sufficiently wide networks.

Lemma 3. Suppose that x 1 , · · · , x n ∈ S d -1 . Let

If λ 1 > 0 and d 1 ≳ λ -1 1 ∥ X ∥ 2 log n ϵ , then with probability at least 1 -ϵ

Lemma 4. Suppose that x 1 , · · · , x n ∈ S d -1 . Let

If λ 2 > 0 and d 1 ≳ n λ 2 log ( n λ 2 ) log ( n ϵ ) , then with probability at least 1 -ϵ

We prove Lemmas 3 and 4 in Appendices C.1 and C.2 respectively. Observe that while the parameters of the model are initialized as Gaussian, the expectations above are taken with respect to the uniform measure on the sphere. The motivation for using the uniform measure on the sphere is that it enables us to work with spherical harmonics, for which there is the highly useful addition formula (see, e.g., Efthimiou & Frye, 2014, Theorem 4.11). The exchange of measures is possible in the case of Lemma 3 due to the scale invariance of ˙ σ , while for Lemma 4 it is possible because σ is homogeneous.

Interpret the infinite-width kernel in terms of a hemisphere transform. Next, for a given X and ψ ∈ { √ dσ, ˙ σ } we define the limiting NTK K ∞ ψ ∈ R n × n as

Consider a fixed vector z ∈ S n -1 and interpret the Euclidean inner product ⟨ ψ ( X T u ) , z ⟩ as a function of u ∈ S d -1 . It will prove useful to think of this map as an integral transform. To this end let M ( S d -1 ) denote the vector space of signed Radon measures on S d -1 and fix ψ ∈ { √ dσ, ˙ σ } . For a signed Radon measure µ ∈ M ( S d -1 ) we introduce the integral transform T ψ µ : S d -1 → R , defined as

Note for ψ ∈ { √ dσ, ˙ σ } this is a hemisphere transform (Rubin, 1999) as the integrand ψ ( ⟨ u , ·⟩ ) is supported on a hemisphere normal to u . We provide background material on the hemisphere transform in Appendix B. Let M X ⊂ M denote the space of signed Radon measures supported on the data set { x 1 , · · · , x n } . For each measure µ ∈ M X there exists a vector z ∈ R n such that µ = ∑ n i =1 z i δ x i , where δ x is the Dirac measure supported on x . We write µ = µ z to indicate this correspondence. The following lemma relates the smallest eigenvalue of K ∞ ψ to the norm of the hemisphere transform of a measure supported on the data; a proof is provided in Appendix C.3.

Lemma 5. Fix X ∈ R d × n and ψ ∈ { √ dσ, ˙ σ } . For all z ∈ R n , ⟨ K ∞ ψ z , z ⟩ = ∥ T ψ µ z ∥ 2 . Moreover,

3) Bound the hemisphere transform norm via spherical harmonics. We proceed to lower bound ∥ T ψ µ z ∥ 2 for all z ∈ R d . Let L 2 ( S d -1 ) denote the Hilbert space of real-valued, square-integrable functions with respect to the uniform probability measure on S d -1 , and let C ( S d -1 ) ⊂ L 2 ( S d -1 ) denote the subspace of continuous functions. For µ ∈ M ( S d -1 ) and g ∈ C ( S d -1 ) we define

If g 1 , · · · , g N ∈ L 2 ( S d -1 ) are orthonormal, in particular consider g r as spherical harmonics, then via a Bessel inequality

Importantly, T ψ is self-adjoint (see Lemma 17 in Appendix B for details) and the spherical harmonics are eigenfunctions of T ψ , i.e., T ψ g a = κ a g a . Asummary of the key properties of spherical harmonics needed for our results are provided in Appendix A.2. Therefore

where D ∈ R N × n is a matrix with entries [ D ] ai = g a ( x i ) . As a result

4) Bound the hemisphere transform and spherical harmonics on the data. The following result shows that if we let the functions ( g a ) a ∈ [ N ] be spherical harmonics and allow N to be sufficiently large, then we can bound the minimum singular value of D . In what follows let H d r denote the vector space of degreer harmonic homogeneous polynomials on d variables.

Lemma6. Suppose x 1 , · · · , x n ∈ S d -1 are δ -separated. Suppose that β ∈ { 0 , 1 } and that R ∈ Z ≥ 0 are such that N := ∑ R r =0 dim( H d 2 r + β ) satisfies N ≥ C ( δ 4 2 ) -( d -2) / 2 where C > 0 is a universal constant. Let g 1 , · · · , g N be spherical harmonics which form an orthonormal basis of ⊕ R r =0 H d 2 r + β . If D ∈ R N × n is defined as D ai = g a ( x i ) then σ min ( D ) ≥ √ N .

A proof of Lemma 6 can be found in Appendix C.4. By carefully choosing values for R and N in Lemma 6 and performing some asymptotics on the resulting expressions, we arrive at the following bound on the hemisphere transform of a measure.

Lemma 7. Let d ≥ 3 and suppose that x 1 , · · · , x n ∈ S d -1 are δ -separated. For all z ∈ R n with ∥ z ∥ ≤ 1 then

A proof of Lemma 7 is provided in Appendix C.5. The lower bound of Theorem 1 follows by bounding λ 1 , as defined in Lemma 3, using Lemma 7.

Before proceeding to the upper bound, we pause to remark on the generality of this argument for handling other activation functions. First, we use the positive homogeneity of the activation function in order to write λ min ( K ∞ ψ ) as the L 2 ( S d -1 ) norm of a function on the sphere. This is beneficial as it allows us to work with the spherical harmonics and use the associated addition formula. The ReLU activation and its derivative are also convenient with regard to computing the eigenvalues of the hemisphere transform (or more generally the eigenvalues of the integral operator). In particular, this requires evaluating integrals against Gegenbauer polynomials for which analytic expressions are available. For polynomial or piecewise polynomial activations similar results could be obtained. However, for other activations, e.g., tanh or sigmoid, such quantities appear challenging to compute.

5) Upper bound. The upper bound of Theorem 1 is simpler than the lower bound and hinges on the following calculation. Let x i , x k be two data points. Then

Therefore it suffices to upper bound the norm of ∇ θ f ( x i ) - ∇ θ f ( x k ) . We choose i, k ∈ [ n ] such that x i , x k are the two closest points in the dataset. We then translate this into a statement about the gradients. If ∥ x i -x k ∥ ≤ δ , then with high probability over the network parameters, ∥∇ θ f ( x i ) -∇ θ f ( x k ) ∥ 2 ≲ δ (see Lemma 29), and we arrive at the desired upper bound in Theorem 1.

4 From shallow to deep neural networks

Our goal here is to detail just one approach as how the results of Section 3 can be extended to deep networks. To be clear, here we consider a fully connected network with input dimension d 0 and L layers, where each layer has width d 1 , · · · , d L respectively and d L = 1 . The parameter space P is a product space of matrices ∏ L l =1 R d l × d l -1 , equipped with the inner product

The feature maps f l : R d 0 ×P → R d l of the neural network are given by

where W l ∈ R d l × d l -1 for all l ∈ [ L ] , θ = ( W 1 , · · · , W L ) and σ is the ReLU function x ↦→ max(0 , x ) applied elementwise. We define the network map f to be the final feature map multiplied by a normalizing constant:

Given n data points x 1 , · · · , x n , we bound the smallest eigenvalue of the NTK (1) associated with this particular choice of f . √

Theorem 8. Suppose ϵ ∈ (0 , 1 / 3) , δ ∈ (0 , 2] , d 0 ≥ 3 , the data x 1 , x 2 , · · · , x n ∈ S d 0 -1 is δ -separated and define

With regard to the network architecture, let L ≥ 3 , d l ≥ d l +1 for all l ∈ [ L -1] , d L -1 ≳ 2 L log ( nL ϵ ) and d 1 ≳ n λ log ( n λ ) log ( n ϵ ) . Then with probability at least 1 -ϵ over the network parameters

We emphasize that these bounds make no distributional assumptions on the data other than lying on the sphere and being δ -separated; in particular, they hold even for constant d 0 . Indeed, if we consider d 0 as some constant then Theorem 8 implies that if the first layer is sufficiently wide, d 1 = ˜ Ω( nδ -4 ) , then with high probability over the parameters λ min ( K ) = ˜ Ω( δ 4 ) and λ min ( K ) = O (1) .

A few remarks are in order. First, the pyramidal condition on the network widths could be relaxed by more directly borrowing techniques from Nguyen et al. (2021). We adopt this condition as it has the advantage of making the dependence of our bounds on the network depth L clearer. Second, compared with Theorem 1 and ignoring log factors, we observe the lower bound differs by a factor of δ 2 . This arises as a result of the smallest eigenvalue of the feature Gram matrix F T 1 F 1 being equivalent to the Jacobian of a shallow network with respect to the second layer weights, not the inner layer weights, which has a different lower bound as per Lemma 7. For reasons apparent in the proof outline below the lower bound on λ min ( K ) lacks a dependency on L , however we hypothesize it should also grow linearly with L thereby matching the dependency of the upper bound. Finally, the upper bound itself follows a similar approach as used by Nguyen et al. (2021) and is weak in the sense that we cannot take advantage of the dataset separation for gradients deeper into the network. We remark that this is also a common problem in the prior work of Nguyen et al. (2021) and Bombari et al. (2022), we refer the reader to the proof outline below for further details.

4.1 Proof outline for Theorem 8

The proof of the deep case is structured around the decomposition of the NTK provided in Lemma 9 below. To state this decomposition we introduce the following quantities. For l ∈ [ L -1] we define the feature matrices F l ∈ R d l × n by

For l ∈ [ L -1] and x ∈ R d we define the activation patterns Σ l ( x ) ∈ { 0 , 1 } d l × d l to be the diagonal matrices

Finally, we let 1 n denote the vector of all ones in R n .

Lemma 9. Let x 1 , · · · , x n ∈ R d be nonzero. There exists an open set U ⊂ P of full Lebesgue measure such that f ( x i ; · ) is continuously differentiable on U for all i ∈ [ n ] . Moreover, for all θ ∈ U the NTK Gram matrix K defined in (1) with network function (7) satisfies

where the i th row of B l ∈ R n × n l is defined as

For completeness we prove Lemma 9 in Appendix D.1. Observe each matrix summand in Lemma 9 is positive semi-definite (PSD) and recall for any two PSD matrices A and B one has λ min ( A + B ) ≥ λ min ( A ) + λ min ( B ) (see e.g. Horn & Johnson, 2012, Theorem 4.3.1) and λ min ( A ⊙ B ) ≥ λ min ( A ) min i ∈ [ n ] [ B ] ii (Schur, 1911). Therefore

In order to upper bound the smallest eigenvalue we follow Nguyen et al. (2021) and analyze the Raleigh quotient R ( u ) = u T Ku ∥ u ∥ 2 . In particular, for any nonzero u ∈ R n we have λ min ( K ) ≤ R ( u ) and therefore λ min ( K ) ≤ R ( e i ) = [ K ] ii for all i ∈ [ n ] . As a result

Combining the upper and lower bounds we have

where the right hand side holds for any i ∈ [ n ] . Based on (8), we proceed first by bounding the norm of the network features. We achieve this via an inductive argument, bounding the norm of the features at one layer with high probability, and then conditioning on this event to bound the norm of the features at the next layer with high probability.

Lemma 10. Let x ∈ S d 0 -1 , L ≥ 2 and l ∈ [ L -1] . If d k ≳ l 2 log( l/ϵ ) for all k ∈ [ l ] , then

holds with probability at least 1 -ϵ over the network parameters.

A proof of Lemma 10 is provided in Appendix D.2. Next we derive upper and lower bounds on the backpropagation terms [ B l ] i, : . Our strategy for this is as follows: for l ∈ [ L -2] , let S l ( x ) = Σ l ( x ) ( ∏ L -1 k = l +1 W T k Σ k ( x ) ) and observe

Since x i ∈ S d 0 -1 , it is sufficient to lower bound ∥ S l ( x ) W T L ∥ 2 2 for an arbitrary x ∈ S d 0 -1 . As the vector W T L ∈ R d L -1 is distributed as W T L ∼ N ( 0 d L -1 , I d L -1 ) , following Vershynin (2018, Theorem 6.3.2) we have that for any A ∈ R d l × d L -1 and t ≥ 0

for some constant C > 0 . As a result, with t = 1 2 ∥ A ∥ 2 F then

In order to lower bound ∥ S l ( x ) W T L ∥ 2 with high probability over the parameters it therefore suffices to condition on appropriate bounds for ∥ S l ( x ) ∥ 2 F and ∥ S l ( x ) ∥ 2 2 . These bounds are provided in Lemmas 34 and 35 in Appendices D.3 and D.4 respectively. With these two lemmas in place we can bound ∥ S l ( x i ) W T L ∥ 2 .

Lemma 11. Let x ∈ S d 0 -1 , suppose L ≥ 3 , d k ≥ d k +1 for all k ∈ [ L -1] and d L -1 ≳ 2 L log ( L ϵ ) . Then, for any l ∈ [ L -1] , with probability at least 1 -ϵ over the network parameters

By combining Lemma 11 with a union bound we arrive at the following corollary, relevant for the lower bound of (8).

Corollary 12. Let x i ∈ S d 0 -1 for all i ∈ [ n ] , L ≥ 3 , d l ≥ d l +1 for all l ∈ [ L -1] and d L -1 ≳ 2 L log ( nL ϵ ) . Then, for any l ∈ [ L -1] , with probability at least 1 -ϵ over the network parameters

The first-layer feature Gram matrix F T 1 F 1 in the deep case is identically distributed to K 2 in the two-layer case; see (3) and the related definitions. Therefore we can apply Lemma 4 to lower bound the smallest eigenvalue of F T 1 F 1 . This, in combination with Corollary 12, yields the lower bound of Theorem 8. The upper bound follows by combining the bound on the feature norms provided by Lemma 10 with the bound on the backpropagation terms given in Lemma 11. A detailed proof of Theorem 8 is provided in Appendix D.6.

5 Conclusion

Summary and implications. Quantitative bounds on the smallest eigenvalue of the NTK are a critical ingredient for many current analyses of network optimization. Prior works provide bounds which are only applicable for data drawn from particular distributions and for which the input dimension d 0 scales appropriately with the number of data samples n . This work plugs an important gap in the existing literature by providing bounds for arbitrary datasets on the sphere (including those drawn from any distribution on the sphere) in terms of a measure of distance between data points. Furthermore, these bounds are applicable for any d 0 , in particular even d 0 held constant with respect to n .

Limitations. Our bounds currently only hold for the ReLU activation function. Another limitation, also present in prior work, is that our upper bound on the smallest eigenvalue of the NTK for deep networks in Theorem 8 does not capture the data separation. Finally, a mild limitation of this work is that we require the data to be normalized so as to lie on the sphere.

Future work. The proof techniques developed here could be applied to analyze the NTK in the context of other homogeneous activation functions. One could potentially relax the homogeneity condition on the activation function, or the condition of unit norm data, by considering an integral transform on the space L 2 ( R d , µ ) rather than L 2 ( S d -1 ) , where µ denotes the standard Gaussian measure (since the weights are drawn from a Gaussian distribution). Beyond fully connected networks, conducting comparable analyses in the context of other architectures, e.g., CNNs, GNNs, or transformers, would be valuable future work.

Acknowledgment

This project has been supported by NSF CAREER 2145630, NSF 2212520, DFG 464109215 within SPP 2298 Theoretical Foundations of Deep Learning, and BMBF in DAAD project 57616814.

References

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 242-252. PMLR, 2019. URL https://proceedings.mlr.press/v97/allen-zhu19a.html .

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 322-332. PMLR, 09-15 Jun 2019a. URL https://proceedings.mlr. press/v97/arora19a.html .

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019b. URL https://proceedings.neurips. cc/paper/2019/file/dbc4d84bfcfe2284ba11beffb853a8c4-Paper.pdf .

Sheldon Axler, Paul Bourdon, and Ramey Wade. Harmonic function theory , volume 137. Springer Science & Business Media, 2013. URL https://doi.org/10.1007/978-1-4757-8137-3 .

Keith Ball. An elementary introduction to modern convex geometry. Flavors of geometry , 31:1-58, 1997.

Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, and Mikhail Belkin. Neural tangent kernel at initialization: Linear width suffices. In The 39th Conference on Uncertainty in Artificial Intelligence , 2023. URL https://openreview.net/forum?id=VJaoe7Rp9tZ .

Ronen Basri, David W. Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Advances in Neural Information Processing Systems 32 , pp. 4763-4772, 2019. URL https://proceedings.neurips.cc/ paper/2019/hash/5ac8bb8a7d745102a978c5f8ccdb61b8-Abstract.html .

Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 685-694. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/ v119/basri20a.html .

Alberto Bietti and Francis Bach. Deep equals shallow for ReLU networks in kernel regimes. In International Conference on Learning Representations , 2021. URL https://openreview.net/ forum?id=aDjoksTpXOP .

Simone Bombari, Mohammad Hossein Amani, and Marco Mondelli. Memorization and optimization in deep neural networks with minimum over-parameterization. In Advances in Neural Information Processing Systems , volume 35, pp. 7628-7640. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 323746f0ae2fbd8b6f500dc2d5c5f898-Paper-Conference.pdf .

Benjamin Bowman and Guido Montúfar. Spectral bias outside the training set for deep networks in the kernel regime. In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=a01PL2gb7W5 .

Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pp. 2205-2211, August 2021. URL https://doi.org/10. 24963/ijcai.2021/304 .

Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ ae614c557843b1df326cb29c57225459-Paper.pdf .

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. In Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=Da_EHrAcfwd .

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 1675-1685. PMLR, 2019a. URL https://proceedings.mlr.press/v97/du19c.html .

Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2019b. URL https://openreview.net/forum?id=S1eK3i09YQ .

Costas Efthimiou and Christopher Frye. Spherical harmonics in p dimensions . World Scientific, 2014. URL https://doi.org/10.1142/9134 .

Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linearwidth neural networks. In Advances in Neural Information Processing Systems , volume 33, pp. 7710-7721. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/ 2020/file/572201a4497b0b9f02d4f279b09ec30d-Paper.pdf .

Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the Laplace and neural tangent kernels. In Advances in Neural Information Processing Systems , volume 33, pp. 1451-1461. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1006ff12c465532f8c574aeaa4461b16-Paper.pdf .

Izrail Solomonovich Gradshteyn and Iosif Moiseevich Ryzhik. Table of integrals, series, and products . Academic press, 2014. URL https://doi.org/10.1016/C2010-0-64839-5 .

Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, 2 edition, 2012. URL https://doi.org/10.1017/CBO9780511810817 .

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_ files/paper/2018/file/5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper.pdf .

Hui Jin, Pradeep Kr. Banerjee, and Guido Montúfar. Learning curves for Gaussian process regression with power-law priors and targets. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=KeI9E-gsoB .

Kedar Karhadkar, Michael Murray, Hanna Tseran, and Guido Montúfar. Mildly overparameterized ReLU networks have a favorable loss landscape. arXiv:2305.19510 , 2023.

Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics , 28(5):1302-1338, 2000. URL https://doi.org/10.1214/aos/ 1015957395 .

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ 0d1a9651497a38d8b1c3871c84528bd4-Paper.pdf .

Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. In Advances in Neural Information Processing Systems , volume 33, pp. 15156-15172. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper/2020/file/ ad086f59924fffe0773f8d0ca22ea712-Paper.pdf .

Wonyeol Lee, Hangyeol Yu, Xavier Rival, and Hongseok Yang. On correctness of automatic differentiation for non-differentiable functions. In Advances in Neural Information Processing Systems , volume 33, pp. 6719-6730. Curran Associates, Inc., 2020b. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/4aaa76178f8567e05c8e8295c96171d8-Paper.pdf .

Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian Journal of Mathematics & Statistics , 4(1):66-70, 2010.

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/ 2018/file/54fe976ba170c19ebae453679b362263-Paper.pdf .

Chaoyue Liu, Libin Zhu, and Mikhail Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. In Advances in Neural Information Processing Systems , volume 33, pp. 15954-15964. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ b7ae8fecf15b8b6c3c69eceae636d203-Paper.pdf .

Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in overparameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis , 59:85-116, 2022. URL https://www.sciencedirect.com/science/article/ pii/S106352032100110X . Special Issue on Harmonic Analysis and Machine Learning.

Andrea Montanari and Yiqiao Zhong. The interpolation phase transition in neural networks: Memorization and generalization under lazy training. The Annals of Statistics , 50(5):2816-2847, 2022. URL https://doi.org/10.1214/22-AOS2211 .

Michael Murray, Hui Jin, Benjamin Bowman, and Guido Montúfar. Characterizing the spectrum of the NTK via a power series expansion. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Tvms8xrZHyR .

Paul Nevai, Tamás Erdélyi, and Alphonse P Magnus. Generalized jacobi weights, christoffel functions, and jacobi polynomials. SIAM Journal on Mathematical Analysis , 25(2):602-614, 1994.

Quynh Nguyen. On the proof of global convergence of gradient descent for deep ReLU networks with linear widths. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 8056-8062. PMLR, 2021. URL https://proceedings.mlr.press/v139/nguyen21a.html .

Quynh Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. In Advances in Neural Information Processing Systems , volume 33, pp. 11961-11972. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper/2020/file/8abfe8ac9ec214d68541fcb888c0b4c3-Paper.pdf .

Quynh Nguyen, Marco Mondelli, and Guido Montúfar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 8119-8129. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/nguyen21g.html .

Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory , 1(1):84-105, 2020. URL https://doi.org/10.1109/JSAIT.2020.2991332 .

Boris Rubin. Inversion and characterization of the hemispherical transform. Journal d'Analyse Mathématique , 77:105-128, 1999. URL https://doi.org/10.1007/BF02791259 .

Schur. Bemerkungen zur Theorie der beschränkten Bilinearformen mit unendlich vielen Veränderlichen. Journal für die reine und angewandte Mathematik , 140:1-28, 1911. URL http://eudml.org/doc/149352 .

Robert T Seeley. Spherical harmonics. The American Mathematical Monthly , 73(4P2):115-121, 1966. URL https://doi.org/10.1080/00029890.1966.11970927 .

Joel A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics , 12:389-434, 2012. URL https://doi.org/10.1007/s10208-011-9099-z .

Maksim Velikanov and Dmitry Yarotsky. Explicit loss asymptotics in the gradient descent training of neural networks. In Advances in Neural Information Processing Systems , volume 34, pp. 2570-2582. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_ files/paper/2021/file/14faf969228fc18fcd4fcf59437b0c97-Paper.pdf .

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018. URL https://doi.org/10.1017/ 9781108231596 .

Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , pp. 1216-1224. PMLR, 2017. URL https://proceedings.mlr.press/v54/xie17a.html .

Ziqing Xie, Li-Lian Wang, and Xiaodan Zhao. On exponential convergence of Gegenbauer interpolation and spectral differentiation. Mathematics of Computation , 82(282):1017-1036, 2013. URL https://doi.org/10.1090/S0025-5718-2012-02645-7 .

Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ 6a61d423d02a1c56250dc23ae7ff12f3-Paper.pdf .

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep ReLU networks. Machine learning , 109(3):467-492, 2020. URL https: //doi.org/10.1007/s10994-019-05839-6 .

A Background material

A.1 Concentration bounds

In order to bound the smallest eigenvalue of the finite-width NTK in terms of the expected, or infinite width NTK, we use the following matrix Chernoff bound variant.

Lemma 13. Let R > 0 , and let Z 1 , · · · , Z m ∈ R n × n be iid symmetric random matrices such that 0 ⪯ Z 1 ⪯ R I almost surely. Then

Here C > 0 is a universal constant.

Proof. By Theorem 1.1 of Tropp (2012), for all δ > 0

Let δ = 1 2 and let C = 1 2 log ( e 2 ) > 0 . Substituting into the above bound, we obtain

Some of our NTK bounds will depend on the operator norm of the input data matrix X , so it will be helpful to upper bound ∥ X ∥ with high probability.

Lemma 14. Let ϵ > 0 . Let X = [ x 1 , · · · , x n ] ∈ R d × n be a random matrix whose columns are independent and uniformly distributed on S d -1 . Then with probability at least 1 -ϵ ,

Proof. We use a covering argument. Fix u ∈ S d -1 and v ∈ S n -1 . By Lemma 2.2 of Ball (1997), for each i ∈ [ n ] and t ≥ 0 ,

In other words ∥⟨ u , x i ⟩∥ ψ 2 ≲ 1 √ d . Then by Hoeffding's inequality, for all t ≥ 0

where C 1 > 0 is a constant.

Let u 1 , · · · , u M be a ( 1 4 ) -covering of S d -1 . That is, u 1 , · · · , u M are a set of points in S d -1 such that for all u ∈ S d -1 , there exists j ∈ [ M ] such that ∥ u -u j ∥ ≤ 1 4 . Since the ( 1 4 ) -covering number of S d -1 is at most 12 d (see Vershynin, 2018, Corollary 4.2.13), we can take M ≤ 12 d . Similarly, let u 1 , · · · , u N be a ( 1 4 ) -covering of S n -1 with N ≤ 12 n . By applying a union bound to (9), we obtain

Hence if then

Rearranging yields

A.2 Spherical harmonics

Here we review some preliminaries on spherical harmonics necessary for our main results. For further details we refer the reader to Efthimiou & Frye (2014) and Axler et al. (2013, Chapter 5). Let L 2 ( S d -1 ) denote the Hilbert space of real-valued, square-integrable functions on the sphere S d -1 , equipped with the inner product

where dS is the uniform probability measure on S d -1 . We let C ( S d -1 ) ⊂ L 2 ( S d -1 ) denote the subset of functions which are continuous. We say that a function g : R d → R is harmonic if it is twice continuously differentiable and

for all x ∈ S d -1 . We say that a polynomial g : R d → R is homogeneous if there exists r ∈ Z ≥ 0 such that

Let us condition on this event for the rest of the proof. Now suppose that u ∈ S d -1 and v ∈ S n -1 . By construction there exist j ∈ [ M ] and k ∈ [ N ] such that ∥ u -u j ∥ ≤ 1 4 and ∥ v -v k ∥ ≤ 1 4 . Then

Since this holds for all u ∈ S d -1 and v ∈ S n -1 , we obtain

for all λ ∈ R and x ∈ R d . Let H d r denote the vector space of degree r harmonic homogeneous polynomials on d variables, viewed as functions S d -1 → R . Each space H d r is a finite-dimensional vector space, with

For ν ≥ 0 and r ∈ N , we define the Gegenbauer polynomials C ν r by

There exists an orthonormal basis of H d r consisting of functions Y d r,s , 1 ≤ s ≤ dim( H d r ) , known as spherical harmonics . The spherical harmonics in H d r satisfy the addition formula

for all x , x ′ ∈ S d -1 . In particular, from the identity C ν r (1) = Γ(2 ν + r ) Γ(2 ν )Γ( r +1) it follows that

We can orthogonally decompose L 2 ( S d -1 ) into a direct sum of the spaces of spherical harmonics:

That is, the spaces H d r are orthogonal and their linear span is dense in L 2 ( S d -1 ) .

Lemma 15. Let δ > 0 and suppose that x , x ′ ∈ S d -1 satisfy ∥ x -x ′ ∥ , ∥ x + x ′ ∥ ≥ δ . If R ∈ Z ≥ 0 , and β ∈ { 0 , 1 } , then

Proof. Let us define

By the addition formula (10),

In order to bound the right hand side of the above equation, we will need a bound for the Gegenbauer polynomials C ( d -2) / 2 2 r + β . By Theorem 1 of Nevai et al. (1994) (see also equation 2.8 of Xie et al. 2013), for all ν ≥ 1 2 , r ≥ 0 , and t ∈ [0 , 1) ,

Rearranging the above expression yields

We now substitute the above bound into (11):

The expression inside the sum is increasing as a function of r , so the above expression is bounded above by

By Stirling's approximation,

Substituting this into (12) yields

| P ( x , x ′ ) | ≤ 1 d 1 / 4 (1 -⟨ x , x ′ ⟩ 2 ) ( d -2) / 4 Γ(2 R + β + d -1) 1 / 2 Γ( d -1) 1 / 2 Γ(2 R + β +1) 1 / 2 ≍ d 1 / 4 ( R + d ) 1 / 2 (1 -⟨ x , x ′ ⟩ 2 ) ( d -2) / 4 Γ(2 R + β + d ) 1 / 2 Γ( d )Γ(2 R + β +1) 1 / 2 = d 1 / 4 ( R + d ) 1 / 2 (1 -⟨ x , x ′ ⟩ 2 ) ( d -2) / 4 ( 2 R + β + d -1 d -1 ) 1 / 2 ≲ 1 (1 -⟨ x , x ′ ⟩ 2 ) ( d -2) / 4 ( 2 R + β + d -1 d -1 ) 1 / 2 Since x , x ′ ∈ S d -1 , 1 -⟨ x , x ′ ⟩ 2 = (1 + ⟨ x , x ′ ⟩ )(1 -⟨ x , x ′ ⟩ ) = 1 4 ∥ x + x ′ ∥ 2 ∥ x -x ′ ∥ 2 ≳ 1 δ 4 .

To conclude, we rewrite

B Preliminaries on hemisphere transforms

Let M ( S d -1 ) denote the vector space of signed Radon measures on S d -1 . We denote the total variation of µ by | µ | . We have a natural inclusion L 2 ( S d -1 ) ⊂ M ( S d -1 ) by associating a function g to a signed measure µ defined by

If µ ∈ M ( S d -1 ) and g ∈ C ( S d -1 ) , we define the pairing ⟨ µ, g ⟩ by

This agrees with the usual definition of the inner product on L 2 ( S d -1 ) when µ ∈ L 2 ( S d -1 ) .

Fix ψ ∈ { √ dσ, ˙ σ } . If µ ∈ M ( S d -1 ) , we define its hemisphere transform (Rubin, 1999) T ψ µ : S d -1 → R by

As is the case with many integral transforms, a hemisphere transform increases the regularity of the functions it is applied to.

Lemma 16. If µ ∈ M ( S d -1 ) , then T ψ µ ∈ L 2 ( S d -1 ) . If g ∈ L 2 ( S d -1 ) , then T ψ g ∈ C ( S d -1 ) .

Proof. Suppose that µ ∈ M ( S d -1 ) . Then

so Tµ ∈ L 2 ( S d -1 ) .

Now suppose that g ∈ L 2 ( S d -1 ) and ψ = ˙ σ . Suppose that ξ , ξ ′ ∈ S d -1 , and observe that

Similarly, so

̸

We apply this calculation to bound the distance between T ψ g ( ξ ) and T ψ g ( ξ ′ ) :

̸

Here the third line follows from Cauchy-Schwarz. As ξ → ξ ′ , arccos( ⟨ ξ , ξ ′ ⟩ ) → 0 and so | T ψ g ( ξ ) -T ψ g ( ξ ′ ) |→ 0 . Therefore, T ψ g ∈ C ( S d -1 ) .

Finally suppose that g ∈ L 2 ( S d -1 ) and ψ = √ dσ . For all ξ ∈ S d -1 ,

So by the dominated convergence theorem, for all ξ ′ ∈ S d -1 ,

Therefore T ψ g ∈ C ( S d -1 ) .

By the above lemma, for any µ ∈ M ( S d -1 ) and g ∈ L 2 ( S d -1 ) , the expressions ⟨ T ψ µ, g ⟩ and ⟨ µ, T ψ g ⟩ are well-defined and finite. In fact, they are equal to each other.

Lemma 17. Suppose that µ ∈ M ( S d -1 ) and g ∈ L 2 ( S d -1 ) . Then

It remains to justify the change in order of integration in the third line. This follows from Fubini's theorem and the calculation

where the last line follows since g ∈ L 2 ( S d -1 ) ⊂ L 1 ( S d -1 ) .

Proof. We compute

In order to characterize how a hemisphere transform acts on L 2 ( S d -1 ) and in particular on the spherical harmonics, we will use the Funk-Hecke formula (see Seeley, 1966) which states that a certain class of integral operators on S d -1 has an eigendecomposition of spherical harmonics.

Lemma 18 (Funk-Hecke formula) . Let ψ : [ -1 , 1] → R be a measurable function such that

Then for all g ∈ H d r

where

Wewill now use the Funk-Hecke formula to compute the coefficients c r,d in the cases where ψ = √ dσ and ψ = ˙ σ . In the following calculations we will use the Legendre duplication formula

and Euler's reflection formula

Lemma 19. For all d ≥ 3 and r ≥ 0 ,

and

Proof. We apply the following identity (see Gradshteyn & Ryzhik, 2014, Equation 7.311.2):

By the Legendre duplication formula, we have

so we can rewrite the above equation as

Substituting ρ = -r/ 2 and ν = ( d -2) / 2 into (13) yields

which establishes the first identity of the claim.

Substituting ρ = (1 -r ) / 2 and ν = ( d -2) / 2 into (13) yields

which establishes the second identity of the claim.

Lemma 20. Suppose that g ∈ H d r and d ≥ 3 . Then for all r ≥ 0 , T ˙ σ g = c r,d g , where

Proof. Let g ∈ H d r . By Lemma 18, where

By Lemma 19, this is equal to

as claimed.

Now we proceed with the second statement. We claim that whenever 0 ≤ r ≤ R ,

We prove this by induction on R . For the base case R = r , the claim trivially holds. Now suppose that the claim holds for some R ≥ r . Then

Hence by induction | c 2 R +1 ,d | ≤ | c 2 r +1 ,d | for all 0 ≤ r ≤ R . Now suppose that 0 ≤ r ≤ R . By Euler's reflection formula,

Moreover, if 0 ≤ r ≤ R , then so

Lemma 21. Suppose that g ∈ H d r and d ≥ 3 . Then T √ dσ g = c r,d g , where

Moreover, if 0 ≤ r ≤ R , then

Proof. The proof is analogous to that of Lemma 20. Let g ∈ H d r . By Lemma 18,

where

By Lemma 19, this is equal to

as claimed.

We claim that whenever 0 ≤ r ≤ R , so

We prove this by induction on R . For the base case R = r , the claim trivially holds. Now suppose that the claim holds for some R ≥ r . Then

Hence by induction | c 2 R | ≤ | c 2 r | for all 0 ≤ r ≤ R . Now suppose that 0 ≤ r ≤ R . By Euler's reflection formula,

C Proofs for Section 3

First we observe the connection between the smallest eigenvalue of the expected NTK when the weights are drawn uniformly over the sphere versus as Gaussian.

Lemma 22. If X ∈ R d 0 × n , then

Proof. Since the distribution of w is rotationally invariant, we can decompose w = α u , where α = ∥ w ∥ , u is uniformly distributed on S d 0 -1 , and α and u are independent. Then

Lemma 22 is useful in that studying the expected NTK in the shallow setting for uniform weights here will prove more convenient than working directly with Gaussian weights.

C.1 Proof of Lemma 3

Lemma 3. Suppose that x 1 , · · · , x n ∈ S d -1 . Let

If λ 1 > 0 and d 1 ≳ λ -1 1 ∥ X ∥ 2 log n ϵ , then with probability at least 1 -ϵ

Proof. By the scale-invariance of ˙ σ ,

For each i ∈ [ n ] and j ∈ [ d 1 ] , and therefore

For each j ∈ [ d 1 ] , let ξ j ∈ { 0 , 1 } be a random variable taking value 1 if | v j | ≤ 1 and taking value 0 otherwise. Since v j is a standard Gaussian there exists a universal constant C 1 > 0 with E [ ξ j v j ] = C 1 for all j . We also define Z ′ j = ξ j Z j . Note that Z ′ j ⪰ 0 , and by the inequality λ max ( A ⊙ B ) ≤ max i [ A ] ii λ max ( B ) ,

where

Furthermore by the inequality λ min ( A ⊙ B ) ≥ min i [ A ] ii λ min ( B ) ,

So by Lemma 13, for all t ≥ 0

where C 2 > 0 is a constant. Since Z j ⪰ Z ′ j for all j ∈ [ d 1 ] , if d 1 ≥ 1 C 2 λ 1 ∥ X ∥ 2 log ( n ϵ ) , then

C.2 Proof of Lemma 4

Lemma 23. Suppose that x 1 , · · · , x n ∈ S d 0 -1 . Let

If λ 2 > 0 and d 1 ≳ n λ 2 log ( n λ 2 ) log ( n ϵ ) , then with probability at least 1 -ϵ , λ min ( K 2 ) ≥ λ 2 4 .

Proof. Note that by Lemma 22,

For each i ∈ [ n ] and j ∈ [ d 1 ] , and therefore

where

By Vershynin (2018, Theorem 6.3.2), for each j ∈ [ d 1 ]

So by Hoeffding's inequality, for all t ≥ 0

for some constant C 1 > 0 . Let s = n C 1 log 4 n λ 2 C 1 . For each j ∈ [ d 1 ] let ξ j ∈ { 0 , 1 } be a random variable taking value 1 if ∥ X T w j ∥ 2 ≤ s and taking value 0 otherwise. Let Z ′ j = ξ j Z j . For each j ∈ [ m ] , Z ′ j ⪰ 0 , and

Moreover,

Here we used (14) in line 6. By Weyl's inequality,

Since Z ′ j ⪯ Z j for all j , for d 1 ≥ 2 s C 2 λ 2 log n ϵ this implies

In other words,

C.3 Proof of Lemma 5

Lemma 5. Fix X ∈ R d × n and ψ ∈ { √ dσ, ˙ σ } . For all z ∈ R n , ⟨ K ∞ ψ z , z ⟩ = ∥ T ψ µ z ∥ 2 . Moreover,

Proof. We compute

which establishes the first part of the result. The second part of the result follows immediately by writing

C.4 Proof of Lemma 6

Lemma6. Suppose x 1 , · · · , x n ∈ S d -1 are δ -separated. Suppose that β ∈ { 0 , 1 } and that R ∈ Z ≥ 0 are such that N := ∑ R r =0 dim( H d 2 r + β ) satisfies N ≥ C ( δ 4 2 ) -( d -2) / 2 where C > 0 is a universal constant. Let g 1 , · · · , g N be spherical harmonics which form an orthonormal basis of ⊕ R r =0 H d 2 r + β . If D ∈ R N × n is defined as D ai = g a ( x i ) then σ min ( D ) ≥ √ N 2 .

Proof. Note that

̸

Let us write D = [ d 1 , · · · , d n ] . Fix i, k ∈ [ n ] with i = k . By the addition formula (10),

By Lemma 15 and δ -separation, there exists a constant C > 0 such that

Suppose that

.

Observe that σ min ( D ) is the square root of the minimum eigenvalue of D T D . By the Gershgorin circle theorem, the minimum eigenvalue of D T D is at least

̸

̸

The result follows.

C.5 Proof of Lemma 7

Lemma 24. Let ϵ ∈ (0 , 1) and let δ > 0 . Suppose that x 1 , · · · , x n ∈ S d -1 form a δ -separated dataset. Let R ∈ N be such that

where C > 0 is a universal constant. Then

for all z ∈ R n with ∥ z ∥ ≤ 1 .

Proof. Let C be the same constant as in Lemma 6 and suppose that

Let β ∈ { 0 , 1 } satisfy β = 1 when ψ = ˙ σ and β = 0 when ψ = dσ . Let N = ∑ R r =0 dim( H d 2 r + β ) . Note that

Let g 1 , · · · , g N be spherical harmonics forming an orthonormal basis of ⊕ R r =1 H d 2 r -1 , and let B ∈ R N × n be the matrix defined by B ai = g a ( x i ) . By Lemma 6, σ min ( B ) ≥ √ N 2 with probability at least 1 -ϵ . Since the functions g a are orthonormal,

By Lemma 17 the above expression is equal to

By Lemmas 20 and 21, T ψ Y 2 r + β,s = c 2 r + β,d Y 2 r + β,s , where c 2 r + β ∈ R and

Hence

So by (15),

We now separately analyze the cases where ψ = ˙ σ and ψ = dσ .

Case 1: ψ = ˙ σ . In this case

Then by Stirling's approximation,

Here the third inequality follows from the observations

and

Case 2: ψ = √ dσ . In this case

Then by Stirling's approximation,

Hence we have established the desired bound on ∥ T ψ µ z ∥

2 in all cases.

Lemma 7. Let d ≥ 3 and suppose that x 1 , · · · , x n ∈ S d -1 are δ -separated. For all z ∈ R n with ∥ z ∥ ≤ 1 then

Proof. We will consider multiple cases depending on the relative scaling of d and n . Let C > 0 be the same constant as in Lemma 24. First suppose that d ≥ C ( δ 4 2 ) -( d -2) / 2 . Let R = 1 . Then

By Lemma 24, ∥ T ψ µ z ∥ 2 ≳ 1 in this case.

Next suppose that d ≤ C ( δ 4 2 ) -( d -2) / 2 and √ d log d ≥ (8 log(1 + C ) + 16 d ) log 2 δ . Let

Note that since d ≤ ( δ 4 2 ) -( d -2) / 2 , we have

Then by Lemma 24, the following bounds hold. If ψ = ˙ σ , then

and therefore

By definition, so that

If ψ = √ dσ , then

Finally suppose that √ d log d ≤ (8 log(1 + C ) + 16 d ) log 2 δ and let R = ⌈ (1 + 2 C ) d ( 2 δ ) 2( d -2) / ( d -1) ⌉ . Then

So by Lemma 24 the following bounds hold. If ψ = ˙ σ , then

If ψ = √ dσ , then

Hence we have shown the desired bound on ∥ T ψ µ z ∥ 2 in all cases.

and

C.6 Upper bound on the minimum eigenvalue of the NTK

Our strategy to upper bound λ min ( K ) will be to prove that if two data points x , x ′ are close, then the Jacobian of the network does not separate points too much. We will need to find upper bounds for both ∥ σ ( Wx ) -σ ( Wx ′ ) ∥ and ∥ ˙ σ ( Wx ) -˙ σ ( Wx ′ ) ∥ .

Lemma 25. Let ϵ ∈ (0 , 1) . Suppose that x , x ′ ∈ S d -1 with ∥ x -x ′ ∥ = δ . If d 1 = Ω ( log 1 ϵ ) , then with probability at least 1 -ϵ ,

Proof. Note that ∥ σ ( Wx ) -σ ( Wx ′ ) ∥ 2 can be written a sum of iid subexponential random variables:

Since the entries of each w j are iid standard Gaussian random variables and σ is 1 -Lipschitz,

Moreover,

So by Bernstein's inequality, for all t ≥ 0

where C > 0 is a universal constant. Setting t = δ 2 d 1 with d 1 ≥ 1 C log 2 ϵ yields

This establishes the result.

Lemma 26. Suppose that x , x ′ ∈ S d -1 . If w ∼ N ( 0 , I d ) , then

̸

Proof. Recall that for x , x ′ ∈ S d -1 ,

̸

where θ is the angle formed by x and x ′ ; that is, θ ∈ [0 , π ] with

By Taylor's theorem, 1 -cos( θ ) = 1 2 θ 2 + O ( θ 3 ) , so 1 -cos( θ ) ≍ θ 2 for θ ∈ [0 , π ] . This implies that θ 2 ≍ ∥ x -x ′ ∥ 2 , so θ ≍ ∥ x -x ′ ∥ and therefore

̸

Lemma 27. Let ϵ ∈ (0 , 1) . Suppose that x , x ′ ∈ S d -1 with ∥ x -x ′ ∥ ≤ δ . If d 1 = Ω ( 1 δ log 1 ϵ ) , then with probability at least 1 -ϵ ,

where Z j ∈ { 0 , 1 } is equal to 1 if

Proof. Observe that

̸

and 0 otherwise, and S consists of the j ∈ [ d 1 ] such that Z j = 1 . The Z j are iid Bernoulli random variables with parameter p , where p ≍ δ by Lemma 26. By Chernoff's inequality (see Vershynin, 2018, Theorem 2.3.1), for all t ≥ d 1 p

Then setting t = ed 1 p with d 1 ≥ 1 p log 4 ϵ yields

By the lower bound of Chernoff's inequality, for all t ≤ d 1 p

Then setting t = d 1 p e with d 1 ≥ 2 e -2 1 p log 4 ϵ yields

Therefore, with probability at least 1 -ϵ 2 ,

Let us denote this event by ω . Observe that

and recall that v 2 j ∼ N (0 , 1) for all j ∈ [ d 1 ] . By Bernstein's inequality, for all t ≥ 0

where C 1 > 0 is a universal constant. Setting t = |S| yields

Then

where we used that ω is measurable with respect to S in the fourth line. So if d 1 ≥ e C 1 δ log 4 ϵ , then

Lemma 28. Suppose that x ∈ S d -1 . If d 1 = Ω ( log 1 ϵ ) , then with probability at least 1 -ϵ ,

Proof. Since ˙ σ ( ⟨ w j , x ⟩ ) ∈ { 0 , 1 } for all j ∈ [ d 1 ] ,

Since the entries v j are iid standard Gaussian random variables, Bernstein's inequality implies for all t ≥ 0

Setting t = d 1 with d 1 ≥ 1 C log 2 ϵ yields

Now we prove our main lemma which we will use to relate the separation between data points to the NTK.

Lemma 29. Let x , x ′ ∈ S d -1 with ∥ x -x ′ ∥ ≤ δ ≤ 2 . Let ϵ ∈ (0 , 1) . If d 1 = Ω ( 1 δ log 1 ϵ ) , then with probability at least 1 -ϵ ,

Proof. By Lemma 27, if d 1 ≳ 1 δ log 1 ϵ , then with probability at least 1 -ϵ 4 ,

Let us denote this event by ω 1 . By Lemma 28, if d 1 ≳ log 1 ϵ , then with probability at least 1 -ϵ 4 ,

Let us denote this event by ω 2 . If both ω 1 and ω 2 occur, then

By Lemma 25, if d l ≳ log 1 ϵ , then with probability at least 1 -ϵ 2 ,

Let us denote this event by ω 3 . If ω 1 , ω 2 , and ω 3 all occur (which happens with probability at least 1 -ϵ ) , then

C.7 Proof of Theorem 1

̸

Theorem 1. Let d ≥ 3 , ϵ ∈ (0 , 1) , and δ, δ ′ ∈ (0 , √ 2) . Suppose that x 1 , · · · , x n ∈ S d -1 are δ -separated and min i = k ∥ x i -x k ∥ ≤ δ ′ . Define

If d 1 ≳ ∥ X ∥ 2 λ log n ϵ , then with probability at least 1 -ϵ ,

Proof. First we prove the lower bound. Let λ 1 be as it is defined in Lemma 3. By Lemma 5,

Let

By Lemma 7, λ 1 ≥ C 1 λ for some constant C 1 > 0 . By Lemma 3, there exist constants C 2 , C 3 > 0 such that if d 1 ≥ C 2 λ 1 ∥ X ∥ 2 log n ϵ then

This establishes the lower bound.

Next we prove the upper bound. Let i, k ∈ [ n ] be two indices with i = k such that ∥ x i -x k ∥ ≤ δ ′ . If d 1 ≳ 1 λ log 1 ϵ ≳ 1 δ ′ log 1 ϵ , then by Lemma 29 there exists C 4 > 0 such that

̸

Let us denote this event by ω . If ω occurs, then

Hence, with probability at least 1 -ϵ 2 , λ min ( K ) ≲ δ ′ . This establishes the upper bound for the minimum eigenvalue. The two-sided bound then immediately follows from a union bound.

C.8 Uniform data on a sphere

Our main bounds for the smallest eigenvalue of the NTK are stated in terms of the amount of separation between data points. To interpret our results in terms of probability distributions on the sphere, we will use a couple of lemmas which quantify the amount of separation for data which is uniformly distributed.

For δ ∈ (0 , 1 / 2) and x ∈ S d -1 , we define the spherical cap

Then for such d 1 , and the double spherical cap

We can also obtain a corresponding upper bound on the volume of a spherical cap.

Lemma 30. For x ∈ S d -1 and δ ∈ (0 , 1 / 2) ,

Here C > 0 is a universal constant.

Proof. For ϕ ∈ [0 , π ] , let S ϕ denote the set of all x ′ ∈ S d -1 such that the angle between x and x ′ is at most ϕ (that is, ⟨ x , x ′ ⟩ ≥ cos( ϕ ) ). The measure of S ϕ is given by

(see, e.g. Li, 2010). Here the numerator refers to the incomplete beta function and the denominator refers to the beta function. We can bound

and

The above two bounds imply

Nowsuppose that x ′ ∈ Cap ( x , δ ) . Then ∥ x -x ′ ∥ ≤ δ , so 1 -⟨ x , x ′ ⟩ ≤ 2 δ 2 . Let ϕ = arccos( ⟨ x , x ′ ⟩ ) be the angle between x and x ′ . By Taylor's theorem, cos( ϕ ) = 1 -ϕ 2 2 + O ( ϕ 3 ) , so 1 -cos( ϕ ) ≍ ϕ 2 for ϕ ∈ [0 , π ] . Thus

So the angle between x and x ′ is at most Cδ for some universal constant C > 0 . It follows that Cap ( x , δ ) ⊆ S Cδ . Finally by (19),

Since δ ≤ 1 , the sets Cap ( x , δ ) and Cap ( -x , δ )

2 are disjoint by the triangle inequality. Hence dS ( DoubleCap ( x , δ )) = 2 Cap ( x , δ )

and in particular by Lemma 30

for a constant C > 0 .

Lemma 31. Suppose that n ≥ 2 and ϵ ∈ (0 , 1) . If x 1 , · · · , x n ∈ S d -1 are independent and uniformly distributed on S d -1 , then with probability at least 1 -ϵ , the dataset is δ -separated with

Proof. Let e = [1 , 0 , · · · , 0] T ∈ S d -1 . For each x ∈ S d -1 , there exists an orthogonal matrix O x such that O x x = e . Note that for all x ∈ S d -1 and i ∈ [ n ] , O x x i d = x i . Let i, k ∈ [ n ] with i = k . Then for all δ ∈ (0 , 1 / 2) ,

̸

The expression on the final line is the measure of DoubleCap ( e , δ ) , and by (20) is bounded above by

̸

where C > 0 is a constant. So

̸

̸

Therefore, for this value of δ , the dataset is δ -separated with probability at least 1 -ϵ . To conclude, note that since

Lemma 32. Suppose that n ≥ 2 and ϵ ∈ (0 , 1) . If x 1 , · · · , x n ∈ S d -1 are selected iid from U ( S d -1 ) , then with probability at least 1 -ϵ , there exist i, k ∈ [ n ] with i = k such that

.

Proof. Let e = [1 , 0 , · · · , 0] T ∈ S d -1 . For each x ∈ S d -1 , there exists an orthogonal matrix O x such that O x x = e . Note that for all x ∈ S d -1 and i ∈ [ n ] , O x x i d = x i . Let i, k ∈ [ n ] with i = k . Then for all δ ∈ (0 , 1 / 2) ,

̸

̸

The expression on the final line is the measure of Cap ( e , δ ) , and by Lemma 2.3 of Ball (1997) it is bounded below by 1 2 ( δ 2 ) d -1 . For each i ∈ [ n ] , let ω i denote the event that ∥ x j -x k ∥ > δ for all j, k ∈ [1 , i ] with j = k . Trivially P ( ω 1 ) = 1 . If ω i occurs for some i ∈ [1 , n -1] , then the sets Cap ( x j , δ/ 2) for j ∈ [ i ] are disjoint. Indeed, if x ∈ Cap ( x j , δ/ 2) ∩ Cap ( x k , δ/ 2) , then by the triangle inequality

which contradicts ω i . Now since these smaller spherical caps are disjoint, we can bound

Since x i +1 is chosen independently from x 1 , · · · , x i , this implies

̸

By repeatedly conditioning we obtain

Let us set δ = min ( 1 4 , 4 ( 2 n 2 log 1 ϵ ) 1 d -1 ) . The above bounds imply that

so with probability at least 1 -ϵ , there exist i, k ∈ [ n ] such that ∥ x i -x k ∥ ≤ δ with

which is what we needed to show.

Corollary 2. Let d ≥ 3 , n ≥ 2 , ϵ ∈ (0 , 1) , x 1 , · · · , x n ∼ U ( S d -1 ) be mutually iid. Define

(

log(

n/ϵ

log(

λ

=

1 +

d

)

ϵ

2

n

4

.

If d 1 ≳ 1 λ ( 1 + n +log(1 /ϵ ) d ) log n ϵ , then with probability at least 1 -ϵ over the data and network parameters,

Proof. By Lemma 14, with probability at least 1 -ϵ 4 ,

Let us denote this event by ω 1 . Let us define

̸

̸

In particular, the dataset x 1 , · · · , x n is δ -separated. By Lemma 31, with probability at least 1 -ϵ 4 ,

Let us denote this event by ω 2 . By Lemma 32, with probability at least 1 -ϵ 4 ,

)

)

-

3

(

)

1

/

(

d

-

1)

and

Let us denote this event by ω 3 . We condition on ω 1 , ω 2 , and ω 3 for the remainder of the proof. Define

and note that

By Theorem 1, if

then with probability at least 1 -ϵ 4 over the network weights,

This is exactly the bound that we needed to show. By taking a union bound over all of the favorable events, it follows that this event happens with probability at least 1 -ϵ .

D Proof of Theorem 8

D.1 Recap of the deep setting

Recall for the deep case we consider fully connected networks with L layers and denote the layer widths with positive integers, d 0 , · · · , d L where d 0 = d and d L = 1 . For l ∈ [ L -1] we define the feature matrices F l ∈ R d l × n as

For l ∈ [ L -1] and x ∈ R d we define the activation patterns Σ l ( x ) ∈ { 0 , 1 } d l × d l to be the diagonal matrices

Lemma 9 provides a useful decomposition of the NTK.

Lemma 9. Let x 1 , · · · , x n ∈ R d be nonzero. There exists an open set U ⊂ P of full Lebesgue measure such that f ( x i ; · ) is continuously differentiable on U for all i ∈ [ n ] . Moreover, for all θ ∈ U the NTK Gram matrix K defined in (1) with network function (7) satisfies

where the i th row of B l ∈ R n × n l is defined as

and

Proof. For any i ∈ [ n ] , observe that f ( x i , · ) is a PAP function (Lee et al., 2020b, Definition 5) and therefore f ( x i , · ) is differentiable almost everywhere (Lee et al., 2020b, Proposition 4). As the union of n null sets is also a null set, we conclude that there exists an open set U of full measure such that for all i ∈ [ n ] then f ( x i , θ ) is differentiable for any θ ∈ U .

Let ∂f ∂ θ denote the true derivative of f with respect to θ when it exists and be the minimum norm sub-gradient otherwise. Using (Lee et al., 2020b, Corollary 13) then

where ∂F L ( θ ) ∂ W l ∈ R d l d l -1 × n . By inspection, to prove the result claimed it therefore suffices to show for any l ∈ [ L ] , θ ∈ U and i, j ∈ [ n ] that

First observe

therefore establishing (21) for l = L . To establish (21) for l ∈ [ L -1] , recall for k ∈ [ L -1] that Σ k ( x ) = diag ( ˙ σ ( W k f k -1 ( x ))) and define Σ L ( x ) = 1 . Observe for 1 ≤ l < k , k ∈ [ L ] that

while for k = l

As a result,

where the first equality arises from iterating (22) and the second by applying (23). Proceeding,

D.2 Proof of Lemma 10

Lemma 33. Let z ∈ R d be a fixed vector and W ∈ R m × d a random matrix with mutually iid elements [ W ] ij ∼ N (0 , 1) for all i ∈ [ m ] and j ∈ [ d ] . Consider the random vector y ∈ R m defined as y = σ ( Wz ) where σ denotes the ReLU function applied elementwise. For δ ∈ (0 , 1) if m ≳ δ -2 log(1 /ϵ ) then

Proof. For i ∈ [ m ] define Z i = w T i z ∥ z ∥ , then Z i ∼ N (0 , 1) are mutually iid. Let B i = ✶ ( Z i > 0) , note by symmetry B i ∼ Ber (1 / 2) , furthermore these random variables for i ∈ [ n ] are also mutually iid with respect to one another. As y i = ∥ z ∥ B i Z i then

For convenience let y ′ = y / ∥ z ∥ and define S = { i ∈ [ n ] : B i = 1 } , then

From (Laurent & Massart, 2000, Lemma 1) we have for any t > 0

For δ 1 ∈ (0 , 1) let t = |S| δ 2 1 4 , then

Observe |S| = ∑ m i =1 B i ∼ Bin ( m, 1 / 2) . With δ 2 ∈ (0 , 1) then applying Hoeffding's inequality we have

Let ω denote the event that (1 -δ 2 ) m 2 ≤ |S| ≤ (1 + δ 2 ) m 2 . If m ≥ 16 δ 2 1 δ 2 2 (1 -δ 2 ) log(4 /ϵ )

For some δ ∈ (0 , 1) let δ 2 = δ 1 = δ/ 3 , then if m ≥ 1944 δ -2 log(4 /ϵ ) we have

from which the result claimed follows.

Lemma 10. Let x ∈ S d 0 -1 , L ≥ 2 and l ∈ [ L -1] . If d ≳ l 2 log( l/ϵ ) for all k ∈ [ l ] , then

holds with probability at least 1 -ϵ over the network parameters.

Proof. For k ∈ [ l ] let ω k denote the event that the inequality

holds. We proceed by induction to establish that P ( ω k ) ≥ (1 -ϵ l ) k for all k ∈ [ l ] . For the base case note that f 1 ( x ) = σ ( W 1 x ) and ∥ x ∥ 2 = 1 . Applying Lemma 33 with δ = 1 l , if d 1 ≳ l 2 log( l/ϵ ) then P ( ω 1 ) ≥ 1 -ϵ l . Now suppose for k ∈ [ l -1] that P ( ω k ) ≥ (1 -ϵ l ) k . Note

Recall f k +1 ( x ) = σ ( W 1 f k ( x )) . Conditioned on ω k , then again applying Lemma 33 with δ = 1 l and as d k +1 ≳ l 2 log( l/ϵ ) we have

which completes the proof of the induction hypothesis. As (1 -ϵ/l ) l ≥ 1 -ϵ and e -1 ≤ (1 -1 /l ) l ≤ (1 + 1 /l ) l ≤ e then

holds with probability at least 1 -ϵ .

D.3 Proof of Lemma 34

Lemma 34. Let x ∈ S d 0 -1 , L ≥ 2 and assume d k ≳ L 2 log ( L ϵ ) for all k ∈ [ L -1] . For any l ∈ [ L -1] with probability at least 1 -ϵ over the network parameters the following holds,

Proof. In what follows for convenience we define an empty product of scalars or matrices as the scalar one. Let K ∈ { L -1 } , l ∈ [ K ] , and for some arbitrary x ∈ S d 0 -1 define

Let ω l,K denote the event

̸

It suffices to lower bound the probability of the event ω l,L -1 . Let F K denote the σ -algebra generated by W 1 , · · · , W K and note that S l,K ∈ F K . Let γ l denote the event that f l ( x ) = 0 , then

Fixing ϵ ∈ (0 , 1) , our goal is to show each term in this product is at least (1 -ϵ L ) : indeed, if this is true then

and our task is complete. To this end, first observe that as d k ≳ L 2 log( L/ϵ ) for all k ∈ [ L -1] , then P ( γ l ) ≥ 1 -ϵ L by Lemma 10. Proceeding to the term P ( ω l,l | γ l ) , recall [ Σ l ( x )] jj = ✶ ([ W l f l -1 ( x )] j > 0) . By symmetry the diagonal entries of Σ l ( x ) are mutually iid Bernoulli random variables with parameter 1 2 . Therefore, using Hoeffding's inequality for all t ≥ 0

Let t = d l , if d l ≥ log 2 L ϵ then with K ≥ 1 , L ≥ 2 it follows that

̸

We now proceed to analyze P ( ω l,h +1 | ω l,h ) for h ∈ [ l, K -1] . Note if ω l,h is true then ∥ S l,h ∥ 2 F > 0 . By definition this implies ∥ Σ l ( x ) ∥ 2 F > 0 , however, if f h ( x ) = 0 then ∥ Σ l ( x ) ∥ 2 F = 0 . Therefore ω l,h being true implies f h ( x ) = 0 . For convenience in what follows we denote the j th column of W h +1 as w j . By definition

therefore,

̸

As highlighted already, if we condition on ω l,h then f h ( x ) = 0 and therefore the random variables ( ˙ σ ( ⟨ w j , f h ( x ) ⟩ )) j ∈ d h +1 are mutually iid Bernoulli random variables with parameter 1 2 . Again by symmetry ˙ σ ( ⟨ w j , f h ( x ) ⟩ ) is independent of ∥ S l,h w j ∥ 2 . Therefore conditioned on ω l,h

Moreover, under the same conditioning

where the last line follows from Theorem 6.3.2 of Vershynin (2018). As a result, conditioned on ω l,h then using Bernstein's inequality (Vershynin, 2018, Theorem 2.8.1) there exists an absolute constant c such that for all t ≥ 0

If d h +1 ≥ 4 L 2 c log 2 L ϵ and t = d h +1 ∥ S l,h ∥ 2 F 2 L then conditioning on ω l,h we obtain

As a result, for any h ∈ [ l, K -1] we have P ( ω l,h +1 | ω l,h ) ≥ 1 -ϵ L from which the result claimed follows.

D.4 Proof of Lemma 35

Lemma 35. Let x ∈ S d 0 -1 , L ≥ 3 and assume d k ≥ d k +1 and d k ≳ √ log 1 ϵ for all k ∈ [ L -1] . For any l ∈ [ L -1] with probability at least 1 -ϵ over the network parameters the following holds,

Proof. By Theorem 4.4.5 of Vershynin (2018), for any k ∈ [ L -1] and all t ≥ 0

Using a union bound it follows that

Note that ∥ Σ k ( x ) ∥ ≤ 1 for all k ∈ [ L -1] , therefore conditional on the above event we have

To conclude we square both sides.

D.5 Proof of Lemma 11

Lemma 11. Let x ∈ S d 0 -1 , suppose L ≥ 3 , d k ≥ d k +1 for all k ∈ [ L -1] and d L -1 ≳ 2 L log ( L ϵ ) . Then, for any l ∈ [ L -1] , with probability at least 1 -ϵ over the network parameters

Proof. Let x ∈ S d 0 -1 be arbitrary and recall S l ( x ) = Σ l ( x ) ( ∏ L -1 k = l +1 W T k Σ k ( x ) ) . Also recall that W T L ∈ R d L -1 is distributed as W T L ∼ N ( 0 d L -1 , I d L 1 ) . Therefore by Vershynin (2018, Theorem 6.3.2) for any A ∈ R d 2 × d L -1 and t ≥ 0

for some constant C > 0 . As a result, with t = 1 2 ∥ A ∥ 2 F then for some constant C > 0

Therefore, in order to lower bound ∥ S l ( x ) W T L ∥ 2 2 with high probability it suffices to condition on a suitable upper bound for ∥ S L -1 ( x ) ∥ 2 2 and a suitable lower bound for ∥ S L -1 ( x ) ∥ 2 F . Let ω denote the event that both

are true. Combining Lemmas 34 and 35 using a union bound, then as long as L ≥ 3 , d k ≥ d k +1 and d k ≳ L 2 log nL ϵ for all k ∈ [ L -1] then P ( ω ) ≥ 1 -ϵ 2 . As a result and also as d L -1 ≳ 2 L log(2 /ϵ )

and

then

as claimed.

D.6 Proof of Theorem 8

Theorem 8. Suppose ϵ ∈ (0 , 1 / 3) , δ ∈ (0 , √ 2] , d 0 ≥ 3 , the data x 1 , x 2 , · · · , x n ∈ S d 0 -1 is δ -separated and define

With regard to the network architecture, let L ≥ 3 , d l ≥ d l +1 for all l ∈ [ L -1] , d L -1 ≳ 2 L log ( nL ϵ ) and d 1 ≳ n λ log ( n λ ) log ( n ϵ ) . Then with probability at least 1 -ϵ over the network parameters

Proof. Recall (8),

where the upper bound holds for any i ∈ [ n ] . We start by analyzing the lower bound. Observe that F 1 F T 1 = σ ( W 1 X ) T σ ( W 1 X ) has the same distribution as d 1 K 2 in the shallow setting; see (3). Let λ 2 be defined as in Lemma 4:

As the dataset x 1 , x 2 , · · · , x n ∈ S d 0 -1 is δ -separated then by Lemma 7

Furthermore, if d 1 ≳ n λ 2 log ( n λ 2 ) log ( n ϵ ) then by Lemma 4

with probability at least least 1 -ϵ 4 and as a result

with probability at least 1 -ϵ 4 . Furthermore, as L ≥ 3 , d l ≥ d l +1 for all l ∈ [ L -1] and d L -1 ≳ 2 L log ( 4 nL ϵ ) then

with probability at least 1 -ϵ 4 . Via a union bound we conclude that the condition

holds with probability at least 1 -ϵ 2 . Fixing some i ∈ [ n ] , for the upper bound observe trivially by construction that

By assumption d k ≳ L 2 log(4 L 2 /ϵ ) for all k ∈ [ L -1] . With l ∈ [0 , L -1] then by Lemma 10

holds with probability at least 1 -ϵ 4 L . Likewise by Lemma 34 for l ∈ [2 , L ] ,

with probability at least 1 -ϵ 4 L . Combining these via a union bound then for any l ∈ [0 , L -1] ,

holds with probability at least 1 -ϵ 2 L . Again using a union bound now over the layers, it follows that

with probability at least 1 -ϵ 2 . As a result, using a final union bound we conclude both the upper and lower bounds hold with probability at least 1 -ϵ .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state our contributions in the introduction along with references to where we prove each of our results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss assumptions and include a Limitations paragraph in our conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We precisely state our theorems and prove them in rigor in respective appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code or data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: None of the potential harms mentioned apply directly to our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is primary theoretical and does not have direct societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper gives a lower bound of the smallest eigenvalue of the NTK matrix for shallow and deep connected ReLU networks through the application of hemisphere transform.

STRENGTHS:
The main significance of this paper is dropping the requirement of the input data dimension from [Nguyen2021] on the same topic, enabling more flexible application in other machine learning problems. The main result is presented with a clear step-by-step proof sketch.  


Reference:
- *Quynh Nguyen, MarcoMondelli, and GuidoMontúfar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8119–8129. PMLR, 18–24 Jul 2021.*

WEAKNESSES:
I do not see any major weakness of this paper. It could be nicer if this paper also offers experimental results to support their calm.

QUESTIONS:
How tight is the lower bound shown in Theorem 1 where the quantity $\lambda$ is defined in terms of the $\delta$-seperated-ness of the data, besides the case where data distributed uniformly on the sphere mentioned in line 152-153?

LIMITATIONS:
This paper is theoretical and limitations are stated clearly in the paragraphs (lines 317-320).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis theory paper fits within a general framework in which one tries to get information on training of deep learning models using the formalism of the so-called Neural Tangent Kernel.\n\nSpecifically, the topic is smallest eigenvalue control for the NTK kernel, and the authors study the minimum eigenvalue under the assumption that one has datasets of unit norm and assume that the data are \"well spread\" i.e. they have controlled separation constants (and sometimes controlled covering, meaning the data are \"uniformly spread\"). The obtained bounds depend on this separation constant and on the input and output space dimensions.\n\nThe technique uses the so-called hemisphere transform and basic harmonic analysis on the sphere. These methods have not been used before for this particular problem.\n\nSTRENGTHS:\nThe studied problem is arguably relevant for dynamical study of NN evolution.\n\nThe techniques used are innovative within this field.\n\nWEAKNESSES:\nThe main weakness is that requirement on the data distribution to be delta-separated is not as \"harmless\" or \"general\" as the authors claim (furthermore, I did not find a justification of this claim in the paper; the authors just state that $\\delta$-separation is \"milder\" than previous work requirements, without explaining why and without verifying that).\nIn practice, it is not trivial to ensure that a sample from a data distribution is well separated in the sense of theorem 8, or a Delone set with controlled constants, making it uniformly separated in the sense of Theorem 1. The assumption of iid data is in practice easier to justify, and checking for delta-separation may be itself a hard problem.\n\nQUESTIONS:\nMain question:\n\nA step for a good comparison to previous work is in having formulated Corollary 2, that is an \"iid data analogue\" of the main result of thm 1. However it is not clear how the bounds from previous works compare to this result. I suggest to put some effort to explicit this comparison in the most explicit way possible. \n\nOther minor observations and questions:\n1) the notion of \"$\\delta$-separatedness\" is a terminology used in the community for the case that points are at minimum distance larger or equal than $\\delta$. The notion is not the same as used in this paper, and defined in line 44. Also, at 3 instances in the paper the notion of \"collinearity\" is used, which is a bit misleading: any two points are collinear. So I suggest to replace \"collinearity\" with something more explicit such as \"being on the same line through the origin\" and that the notion of \"$\\delta$-separated\" is either called by a different name, or that it be emphasized the difference with the usual notion.\n\n 2) In the paragraph before line 39, there is an instance in which $\\mathbb R^{d\\times n}$ should be replaced by $\\mathbb R^{d_0\\times n}$. \n\n 3) Section 2 has a large overlap with the introduction. Could it be shortened or merged?\n\n4) lines 141-142, about data in $\\mathbb S^1$: this sentence is not clear to me, and it is not clear how passing from $\\mathbb S^1\\subset \\mathbb R^2$ to $\\mathbb S^1\\times\\{0\\}\\subset \\mathbb R^3$ affects the constant $\\delta'$ from Theorem 1.\n\n5) line 243-244 and lines 313-315: the fact that data are required to be $\\delta$-separated has to be mentioned, since it restricts generality.\n\nLIMITATIONS:\nThe main concerns were mentioned in the \"weaknesses\" part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This work provides bounds on the smallest eigenvalue of the Neural Tangent Kernel corresponding to fully connected ReLU networks trained on data supported on spheres. The novelty is that usual assumptions coupling the input data dimension to the sample size are able to be weakened. Similarly, assumptions on the data distribution are replaced by a condition on its realization, that the datapoints are $\delta$-separated.

STRENGTHS:
This paper is well written and provides a novel result that gives valuable insight into the behaviour of the NTK.

WEAKNESSES:
The major weaknesses of the paper are addressed by the authors and provide avenue for future research.

QUESTIONS:
None

LIMITATIONS:
The authors have addressed the limitations of the work. This is a theoretical paper so the broader societal impact is negligible.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper investigates the neural network optimization and memorization  in terms of the bounds on the smallest eigenvalue of NTK, without requiring distributional assumptions on the data. The theoretical results are technically sound and contribute to the understanding of neural network convergence behavior.

STRENGTHS:
1. The bounds hold without requiring the distributional assumptions on the data and being applicable to high-dimensional settings. 

2. The authors introduce a novel application of the hemisphere transform and the addition formula for spherical harmonics, which serves as an innovative approach to analyzing the NTK. 

3. The results are applicable to both shallow and deep networks.

WEAKNESSES:
1. The current results are constrained to scenarios where the activation function is exclusively ReLU, which potentially limits the applicability. What is the primary impediment to generalizing the current results ? 

2. The structure of the paper needs to be improved. The main conclusion are presented in Theorem 1 and 8, but it takes a lot of space to present the proof sketch in the corresponding section, more discussions on how the upper/lower bounds influence the performance of the DNNs should be included ?

3. Theorem 8 requires the layer width satisfy a pyramidal condition, I wonder whether modern DNNs architecture fullfills this requirement?

4. I acknowledge the theoretical contributions made by the authors, but  I would recommend the authors to add some empirical studies to support their theoretical claims.

minors:
$X \in \mathbb{R}^{d \times n}$ should be $X \in \mathbb{R}^{d_0 \times n}$

QUESTIONS:
Please refer to weaknesses

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The paper derives new bounds on the smallest eigenvalue in NTK kernel matrices crucially used in the analyses of neural network training and generalization. Hereby it uses new analytical techniques. One main point improving over most previous bounds is that they are widely distribution independent. The only (standard) assumptions are that data lies on the unit sphere, and they are not too collinear as introduced by Oymak & Soltanolkotabi, 2020. It seems that the new results improve over previous work Banerjee et al., 2023. But I am missing direct comparisons which could be addressed in the rebuttal.

STRENGTHS:
* widely distribution independent bounds
* only standard assumptions
* linear dependence on network width
* extension to multilayer NN

WEAKNESSES:
* limited to ReLU activation
* missing out some direct comparisons to previous bounds
* study is motivated by NN optimization, but only the initialization phase is actually considered

QUESTIONS:
* how do the new bounds affect the full (S)GD optimization e.g. for squared loss? (for instance while the width is linear for your bounds, many previous analyses require larger polynomial width in the course of optimization, even where low width suffices in the initialization)
* how do the new bounds relate to existing bounds for classification under cross entropy loss? For instance the delta dependence that is between linear and quadratic in Thm 1 reminds closely of the results in https://arxiv.org/abs/2206.12802 that are also between linear and quadratic in gamma, the separation margin in NTK for classification (which seems closely related to your delta).
* any idea on resolving the linear/quadratic gap?

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12