Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time
Authors: Arvind Venkat Mahankali
Jeff Z. HaoChen
Kefan Dong
Margalit Glasgow
Tengyu Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Nonconvex Optimization
Mean-Field Analysis
Beyond NTK
Deep Learning Theory
TL;DR: we provide the first mean-field analysis of neural networks that acheives polynomial width, polynomial runtime, and a strictly better sample complexity than NTK

Abstract: Despite recent theoretical progress on the non-convex optimization of two-layer neural networks, it is still an open question whether gradient descent on neural networks without unnatural modifications can achieve better sample complexity than kernel methods. This paper provides a clean mean-field analysis of projected gradient flow on polynomial-width two-layer neural networks. Different from prior works, our analysis does not require unnatural modifications of the optimization algorithm. We prove that with sample size $n = O(d^{3.1})$ where $d$ is the dimension of the inputs, the network trained with projected gradient flow converges in polynomial time to a non-trivial error that is not achievable by kernel methods using $n \ll  d^4$ samples, hence demonstrating a clear separation between unmodified gradient descent and NTK. As a corollary, we show that projected gradient descent with a positive learning rate and a polynomial number of iterations converges to low error with the same sample complexity.

Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time
Arvind Mahankali Stanford University amahanka@stanford edu
Jeff Z. HaoChen Stanford University
Kefan Dong
Stanford University
jhaochen@stanford ford edu
kef kefan wnndongdstanford...ooo
Margalit Glasgow
Tengyu Ma Stanford University
Stanford University
mglaagow@stant ord. edu
tengyuma@stanfore werettanfodddeeee edu
Abstract
Despite recent theoretical progress on the non-convex optimization of two-layer neural networks, it is still open question whether gradient descent on neural networks without unnatural modifications can achieve better sample complexity than kernel methods. This paper provides clean mean-field analysis of projected gradient flow on polynomial-wine width two-layer neural networks. Different from prior works, our analysis does not require unnatural modifications of the optimization algorithm. We prove that with sample size (dd..1) where d is the dimension of the inputs, the network trained with projected gradient flow converges in poly(d) time to non-trivial eтror that is not achievable by kernel methods using & d samples, hence demonstrating a clear separation between unmodified gradient descent and NTK. As corollary, we show that projected gradient descent with a positive learning rate and polynomial number of iterations converges to low error with the same sample complexity.
Introduction
Training neural networks requires optimizing non-convex losses, which is often practically feasible still not theoretically understood. The lack of understanding of non-convex optimization limits the design of new principled optimizers for training neural networks that use theoretical insights.
Early analysis optimizing neural networks with linear quadratic activations
29
36
51
35
65 38 48
39 60) relies on linear algebraic tools that do not extend to nonlinear and non non-quadratic activations. The neural tangent kernel (NTK) approach analyzes nonconvex optimization under certain hyperparameter settings, c.g., when the initialization scale is large and the learning rate is small (see, e.g., Du al. 30, Jacot al. Li and Liang 50 Arora et al. [10] Daniely al. (26).
However, subsequent research shows that neural networks trained with practical hyperparameter settings typically outperform their corresponding NTK kernels (9. Furthermore, the initialization and
learning rate under the NTK regime does not yield optimal generalization guarantees [76 22 34]
Many recent works study modified versions of stochastic gradient descent (SGD) and prove sample
complexity and runtime guarantees beyond NTK (33
2 58 52
5
79
78
25
6
70 20 [59].
These modified algorithms often contain multiple stages that optimize different blocks of parameters
"quual Contribution
Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and/or use different learning rates or regularization strengths. For example, the work of Li et al. [52 uses non-standard parameterization for two-layer neural networks and runs a two-stage algorithm with sharply changing gradient clipping strength; Damian et al. [33] use one step of gradient descent with a large learning rate and then optimize only the last layer of the neural net. Nichani et al. [59] construct non-standard regularizers based on the NTK feature covariance matrix, in order to prevent the movement of weights in certain directions which hinder generalization. Additionally, Abbe et al. [2) only train the hidden layer for O(1) time in the first stage of their algorithm, and then train the second layer to convergence in the second stage. They do not study how the population loss decreases during the first stage. However, oftentimes vanilla (stochastic) gradient descent with constant learning rate empirically converges to a minimum with good generalization error. Thus, the modifications are arguably artifacts tailored to the analysis, and to some extent, over-using the modification may obscure the true power of gradient descent.
Another technique to analyze optimization dynamics is the mean-field approach (21 56 57 61
28 2
44 62 76], which views the collection of weight vectors as (discrete) distribution over IR (where d is the input dimension) and approximates its evolution by an infinite-width neural network, where the weight vector distribution is continuous and evolves according to a partial differential equation. However, these works do not provide an end-to-end polynomial runtime bound on the convergence to a global minimum in the concrete setting of two-layer neural networks (without modifying gradient descent). For example, the works of Chizat and Bach (2]) and Mei et al. [56] do not provide a concrete bound on the number of iterations needed for convergence. Mei et al. [57] provide a coupling between the trajectories of finite and infinite-width networks with exponential growth of coupling error but do not apply it to a concrete setting to obtain a global convergence result with sample complexity guarantees. (See more discussion below and in Section 3 Wei et al. [76] achieve a polynomial number of iterations but require exponential width and an artificially added noise. In other words, these works, without modifying gradient descent, cannot prove convergence to a global minimizer with polynomial width and iterations. We also discuss additional related works in Appendix A
In this paper, we provide a mean- -ield analysis of projected gradient flow on two-layer neural networks with polynomial width and quartic activations. Under simple data distribution, demonstrate that the network converges to a non-trivial error in polynomial time with polynomial samples. Notably, our results show sample complexity that is superior to the NTK approach. As a corollary, we show that projected gradient descent with polynomially small step size and polynomially many iterations can converge a low error, with sample complexity that is superior to NTK. Our proof is similar to the standard bound on the error of Euler's method.
Concretely, the neural network is assumed to have unit-norm weight vectors and no bias, and the second-layer weights are all 1 where m the width of the neural network. The data distribution is uniform over the sphere. The target function is of the form y(z) = h(gg, x)) where h is an unknown quartic link function and q, is an unknown unit vector. Our main result (Theorem B.4) states that with n (dd.1)) samples, a polynomialwwidth neural network with random initialization converges in polynomial time to a non-trivial error, which is statistically not achievable by any kernel method with an inner product kernel using n < d' samples. To the best of our knowledge, our result is the first to demonstrate the advantage of unmodified gradient descent on neural networks over kernel methods.
The rank-one structure in the target function, also known as the single-index model, has been well- studied in the context of neural networks as simplified case demonstrate that neural networks can learn a latent feature direction 4+ better than kernel methods [58] 2 23 16] Many works on single- index models study (stochastic) gradient descent in the setting where only single vector (or "neuron") in Rd is trained. This includes earlier works where the link function is monotonic, or the convergence analyzed with quasi-convexity 46 40 (55) 64), along with more recent work [69] 72 [[] for more general link functions. Since these works only train a single neuron, they have limited expressivity in comparison to a neural network, and can only achieve zero loss when the link function equals the activation. We stress that in our setting, the link function h is unknown, not necessarily monotonic, and does not need to be equal to the activation function. We show that in this setting, the first layer weights will converge to a distribution of neurons that are correlated with but not exactly equal to q+. so that even without bias terms, their mixture can represent the link function h. Our analysis demonstrates that gradient flow, and gradient descent with a consistent poverse-oolyoomil learning rate, can simultaneously learn the feature q. and the link function h, which is a key challenge that is side-stepped in previous works on neural-networrs which use two-stage algorithms [58, 12 23 [4]

---

The main novelty of our population dynamics analysis is designing a potential function that shows that the iterate stays away from the saddle points.
Our sample complexity results leverage coupling between the dynamics on the empirical loss for finite-width neural network and the dynamics on the population loss for an infinite-widhh width neural network. The main challenge stems from the fact that some exponential coupling error growth is inevitable over certain period of time when the dynamies resemble a power method update. Heavily inspired by Li et al. [521, we address this challenge by using a direct and sharp comparison between the growth of the coupling error and the growth of the signal. In contrast, simple exponential growth bound on the coupling error similar to the bound of Mei and Montanari [54] would result in (2(1) sample complexity, which is not sufficient to outperform NTK.
2 Preliminaries and Notations
We use 0,),5, to hide only absolute constants. Formally, every occurrence of O(z) in this paper can be simultaneously replaced by a function f(x) where [f(x)] S C|xl, Vx R for some universal constant C > 0 (each occurrence can have a different universal constant C and f). We use a < b as a shorthand for < O(b). Similarly, (xx) is a placeholder for some g where 19(x) 2 | vr E E for some universal constant C' >> 0. We use a 2 b as a shorthand for M (bb and a = as shorthand indicate that a 2 b and a S simultancously hold.
Legendre Polynomials. We summarize the necessary facts about Legendre polynomials below, and present related background more comprehensively in Appendix C Let Pk.d [-1,1] - TR be the degree-k un-normalized Legendre polynomial [12 and Pk.a (L)
be the normalized Legendre polynomial, where Nx,d - (( - (Sth-3) is the normalizing factor.
The polynomials (t. form an orthonormal basis for the set of square- intterritteeeable functions over [-1,1] with respect to the measure pa(t)= & )) d=3 r(d/2) 1
1((d-1)/2) vi i.e. the density of 41 when u = (u,,-.. ua) is uniformly drawn from sphere gd-1, Hence, for every function [-1, - R such that [ < co, we can define ha.d & E [h(t) and consequently, we have h(t) -E hr. (t.
3 Main Results
We will formally define the data distribution, neural networks, projected gradient flow, and assump- tions on the problem-dependent quantities and then state our main theorems.
Target function. The ground-truth function y(x) Rdd + R that we aim to learn has the form ((x) hlg_ z). where h : IR - R is an unknown one- mineeimnnional even quartic polynomial (which is called a link function), and q- is an unknown unit vector in Rdd. Note that h(s)) has the Legendre expansion h(s) ho.d + ha a (2) + ha.aPa.a(s)
Two-layer nearal networks. We consider two-layer neural network where the first-layer weights all unit vectors and the second-layer weights are fixed and all the same. Let a(-) be the activation function, which can be different from h(.)) Using the mean-field formulation, we describe the neural network using the distribution of first-layer weight vectors. denoted by pt
fx( A Eupp la (uTTN) x))]
(3.1)
For example, when p = unif((u)... um 1m)), is discrete, uniform distribution supported on m vectors (ui...., um ). then fz(z) m E (u, 1). i.c. fo corresponds finite-width neural network whose first-layer weights are uj...., Um For continuous distribution p, the function f,(+) can be viewed as an infinite-width neural network where the weight vectors are distributed according to p (and can be viewed as taking the limit as m - oc of the finite-width neural network). We assume that the weight vectors have unit norms, i.e. the support of p is contained in $d-1 The activation σ R + R is assumed to be a fourth-degres polynomial with Legendre expansion (s)) Ei=o GR.aP (s).
3

---

The simplified neural network defined in Eq. B.I.. even with infinite width (corresponding to continuous distribution p), has limited expressivity due to the lack of biases and trainable second- layer weights. We characterize the expressivity by the following lemma:
and 7A1 = hr.a/ōn.. Suppose for some d and qu, there exists a network p such that fo(z) h(g 2) on gt-1, Then we have ōu,d = ha,d, and 0 S 72 I 74 S 722 Moreover, if this condition holds with strict inequalities, then for sufficiently large d, there exists network such that fp(x)) h(g, x) on gd-1. (A more explicit version is stated in Appendix G.7
Lemma 3.1 (Expressivity). Let 72
Informally, an almost sufficient and necessary condition to have fx( = h(g. 2)) for some p is that there exists a random variable w supported on [0, I] such that K[w²] x 72 and E[w"] s 74, which is equivalent to < 7/ I 74 S 72 S 1. In particular, assuming the existence of such random variable w, the perfectly- fit network that fits the target function has the form
9." Juu= - w, and g.9. u q. is uniformly distributed in the subspace orthogonal to g. (3.2)
Motivated by this lemma, we will assume that 72, 74 are universal constants that satisfy 0 S 1 < It S 72 S 1, and d is chosen to be sufficiently large (depending on the choice of 72 and %1) In addition, we also assume that 7/1 I O(72). that is, the equality 72 S 741 is somewhat tight this ensures that the distribution of w = q. u under the perfectly-fitted neural network is not too spread-out. We also assume that 72 is smaller than sufficiently small universal constant. This ensures that the distribution of q," under the perfectly-fitted network does not concentrate on 1, i.e. the distribution of u is not merely point mass around q.. In other words, this assumption restricts our setting to the most interesting case where the landscape has bad saddle points (and thus is fundamentally more challenging to analyze). We also assume for simplicity that ō = ho.d (because otherwise, the activation introduces a constant bias that prohibits perfect fitting), even though adding a trainable scalar to the neural network formulation can remove the assumption. ōo.d = ho.a. then we can assume without loss of generality that oo. λo,a = O, since oo, and ho.d will cancel with each other in the population and empirical mean squared losses (defined below). In summary, we make the following formal assumptions on the Legendre coefficients of the link function and the activation function.
Assumption 3.2. Let 722 = hzaa0ōzd and 7/4 = -ha.d/ōA.d- We first assume 7/4 2 1.1 72 For any universal constant 01 > 1, we assume that /22 S 66, S c1 02.0- and 7/1 S c17-- For sufficiently small universal constant 02 > O (which is chosen after e is determined), we assume O S 722 S C2- We also assume that d is larger than sufficiently large constant es (which is chosen after ei and C2.) We also assume ho,a = σo,a = 0, and hi hagd 0.
Our intention is to replicate the ReLU activation as well as possible with quartic polynomials; our assumption that 0² x 0²,n is indeed satisfied by the quartic expansion of ReLU because relug,d d x dd-1/' 1/2 and relug,d x d- 1/2 (see Proposition C.3). Following the convention defined in Section 2 we will simply write o x 2,d 74 2 1.1
72. and 7/ 212-
Our assumptions rule out the case that analysis could be extended to this case with some modifications. Our analysis also rules out the case where 1/2 = O and 7/4 * O, due to the restriction that 74 < c172- The case 7/ = 0 and 7 * O would have significantly different analysis, and potentially a different sample complexity, since our analysis in Section 4.2 and Section 5 makes use of the fact that the initial phase of the population and empirical dynamics behaves similarly to a power method, which follows from 72 being nonzero.
72. for the sake of simplicity. We believe that our
Data distribution, losses, and projected gradient flow. The population data distribution is assumed iid to be 8d-1, We draw n training examples 21...., Vo ~ gd-1, Thus, the population and empirical mean-squared losses are:
[(p)
- E
1
2 a--d-1 (((x)-v(zr)
and i(p)
1 2n
[(,(o)) -((l))))(a)))
(3.3)
To ensure that the weight vectors remain on 1d-1, we perform projected gradient flow on the empirical loss. We start by defining the gradient of the population loss L with respect particle at p and

---

the corresponding Riemannian gradient (which is simply the projection of the gradient to the tangent space of 1d
V.L(o)=
E N~ (Vo(x)) (0l)-oo()" x)+| and grad, uu V.L(p) (3.4)
Here we interpret L(p) as a function of a collection of particles (denoted by p) and VoL(p) as the partial derivative with respect to single particle evaluated at p. Similarly, the (Riemannian) gradient of the empirical loss L with respect the particle defined as
V v.i(p)- = -(s((s) - y(x;))o (u xt)a,, and grad, E(p) = -I-wul VV-uuT)V.I(p)) (3.5)
Population, Infinite-Wide Dynamies. Let the initial distribution po of the infinite-width neural network be the uniform distribution over Sd-1 We use to denote a particle sampled uniformly at random from the initial distribution Pu. A particle initialized at X follows a deterministic trajectory afterwards we use (x(x) to denote the location, at time t. of the particle that was initialized at X. Because (-) is a deterministic function, we can use X index the particles at any time based on their initialization. The projected gradient flow on an infinite- width neural network and using population loss L, can be described as
VX E S mo(x)) = x du, (x) grad [(pt),
(3.6)
(3.7)
dt
and P: distribution of (x(x) (where X Po).
(3.8)
Empirical, Finite-Widtt Dynamics. The training dynamics of a neural network with width m can be described in this language by setting the initial distribution to be a discrete distribution uniformly supported on m initial weight vectors. The update rule will maintain that at any time, the distribution of neurons is uniformly supported over items and thus still corresponds to width-m neural Lid
network. Let XI. XI...., Xm ~ gd-1 be the initial weight vectors of the width-m neural network, and let Ao unif( mii(((1>>> (X1,-.. )m)) be the uniform distribution over these initial neurons. We use E fxI...., X J to index neurons and denote single initial neuron as (x = Then, we can describe the projected gradient flow on the empirical loss L with initialization {xI...., Xm by:
dug (x) = grad (x)) [(Pc), dt
(3.9)
and Ar = distribution of (x(x) (where x ~ pn)
(3.10)
We first state our result the population, infinite-width dynamies.
Theorem 3.3 (Population, infinite-width dynamics), Suppose Assumption 3.2 holds, and let e E (0, 1)
be the target error. Let Pt be the result of projected gradient flow on the population loss, initialized with the uniform distribution on gd-1, as defined in Eq. (3.7). Let T., = inf > 0 I(px) s
(03 + - 2 be the earliest time t such that a loss of at most 1 [0 we have
2 2,4 + 0² oZ,a)² is reached. Then,
1 log d + (log log 2)
To. S
72 log
(3.11)
2,227
2
The proof is given in Appendix D The first term on the right-hand side of Eq. (3.11 corresponds to the burn-in time for the network reach a region around where the Polyak-Lojasiewiez condition holds. We divide our analysis of this burn-in phase into two phases, Phase I and Phase 2, and we obtain tight control on the factor by which the signal component, q, u. grows during Phase 1, while Phase 2 takes place for comparatively short period of time. (This tight control is critical for our sample complexity bounds where we must show that the coupling error does not blow up too much see more discussion below Lemma 5.4) Phases I and are mostly governed by the quadratic components the activation and target functions, and the dynamics behave similarly to power method update. After the burn-in phase, the dynamics operate for short period of time (Phase 3) in condition holds. We explicitly prove the dynamics stay away
a regime where the from saddle points during this phase, as further discussed in Section
5

---

We note that Theorem 3.3
provides a concrete polynomial runtime bound for projected gradient flow which not achievable by prior mean-fiele analyses [2T [6 using Wasserstein gradient flow techniques. For instance, while the population dynamies of Abbe et al. [2] only trains the hidden layer for ((1) time, our projected gradient flow updates the hidden layer for O d) time. The
main challenge in the proof is to deal with the saddle points that are not strict-saddle [32 in the loss
landscape which cannot be escaped simply by adding noise in the parameter space [45 49 24]
Our analysis develops fine-graines analysis of the dynamics that shows the iterates stay away from saddle points, which allows us to obtain the running time bound in Theorem 3.3 which can be
translated to a polynomial-we width guarantee in Theorem 3.4 In contrast, Wei et al. [76] escape the saddles by randomly replacing an exponentially small fraction of neurons, which makes the network require exponential width.
Next, we state the main theorem on projected gradient flow on empirical, knite-widtt dynamics.
Theorem 3.4 (Empirical, finite-width dynamics). Suppose Assumption 3.2 holds. Suppose e =
log logd the larget error and T., is the running time defined in Theorem 3.3 Suppose n 2
d' (log d)n())) for any constant μ >> 3, and the network width m satisfies m ž dz.0++/22 (log d)((()) and m S dd for some sufficiently large universal constant C. Let Pe be the projected grudient flow on the empirical loss, initialized with m uniformly sumpled weights, defined in Eq. (3.10). Then, with probubility at least Tetlogay over the randomness of the data and the initialization of the finite-width network, we have that L(Pr..) S (62,a + 02,,)c².
1
Plugging in μ = 3.1, Theorem 3.4 suggests that, when the network width is at least d4.05 (up to logarithmic factors), the empirical dynamics of gradient descent could achieve o (1/ log log d)² population loss with d3.1 samples (up to logarithmic factors).
In our analysis, we will establish coupling between neurons in the empirical dynamics and neurons in the population dynamies. The main challenge to bound the coupling error during Phase I where the population dynamics are similar to the power method. During this phase, we show that the coupling error (i.e., the distance between coupled neurons) remains small by showing that it can be upper bounded in terms of the growth of the signal of u in the population dynamics. Such a delicate relationship between the growth of the error and that of the signal is the main challenge to proving a sample complexity bound better than NTK. Even an additional constant factor in the growth rate of the coupling error would lead to an additional poly(d) factor in the sample complexity. Prior work (e.g. Mei et al. (57), Abbe et al. (2) also establishes coupling between the population and empirical dynamics. However, these works obtain bounds on the coupling error which are exponential in the running time- specifically, their bounds have a factor of e where K is universal constant and T is the time. In many settings, including ours, 2(log d) time is necessary to achieve a non-trivial error, (as we further discuss in Section 4.2) and thus this would lead to a (1)) bound on the sample complexity, where (1)) is unspecified and likely loose constant, which cannot outperform NTK. Our work addresses this challenge by comparing the growth of the coupling error with the growth of the signal, which enables us control the constant factor in the growth rate of the coupling error.
Projected Gradient Descent with 1 1/poly(d) Step Size. As a corollary of Theorem 3.4, we show that projected gradient descent with a small learning rate can also achieve low population loss in poly(d) iterations. We first define the dynamics of projected gradient descent as follows. As before,
we let XI...., Xm gd-1 be the initial weight vectors, and we let أ unif( .m }) denote the uniform distribution over these vectors. Thus, we can write a single neuron at initialization as ūo(x)= x for E ,x The dynamics of projected gradient descent, on finite-width neural network and using the empirical loss L, can be then described as
(x(x) - 17 grad T.(p))
(x+((x)
(3.12)
((ūx()) 1 grad grada,60 ((p)))l
and Pt distribution of (x) (where x ~ po).
(3.13)
where 17 >> 0 is the learning rate We show that projected gradient descent with 1 l/poly(d) learning rate can achieve a low population loss in poly(d) iterations:
Thhere are a long list of prior works on the loss landscape of neural networks 55
16 IS
7 3
X0 [8]
SSee Chapter page 20 of Boumal accessed on August 21, 2023.

---

Theorem 3.5 (Projected Gradient Descent). Suppose we are in the setting of Theorem 3.4 Let Pr be the discrete-ting projected gradient descent on the empirical loss, initialized with m weight vectors sampled uniformly from S defined in Eq. (.11). Finally, assume that S 16 t
for
sufficiently large universal constant B. Then, L(PT../n) S (62.2 og,)c²
Theorem 3.5 follows from Theorem 3.4 together with standard inductive argument to bound the discretization error. The full proof is in Appendix F
The following theorem states that in the setting of Theorem 3.4 kernel methods with any inner
product kernel require (2 (In 9) samples to achieve a non con-trivial population loss. (Note that the
zero function has loss Ex-ga [b(())²] 2 (h),a)².)
Theorem 3.6 (Sample complexity lower bound for kernel methods). Let K be an inner product kernel. Suppose is larger than universal constant and n S d (Ind) - Then, with probability at least 1/2 over the randomness of n i.idd data points {xx))=1 drawn from gd-1, any estimator of y of the form f(x) = Ei= B,K (x x) of {ai)i=1 must have a large error: Eg-ss 1 (y(x) - f(x))² 2 (hs,a)²
Theorem 3.4 and Theorem 3.6 together prove clear sample complexity separation between gradient flow and NTK. When ha.d x max (i.e., 72: 7/1 x 1), gradient flow with finite- neural networks can achieve (h4,aa² (log log d)-² population error with d3.1 samples, while kernel methods with any inner product kernel (including NTK) must have error at least 2//2 with d samples.
On a high level, Abbe et al. function is drawn randomly and the data points can be arbitrary (also see Kamath et al. 47, Hsu et al. Hsu (1).. ). In comparison, our lower bound works for fixed target function by exploiting the randomness the data points. In fact, we can strengthen Theorem 3.6 by proving a 2 k. loss
3 prove similar lower bounds in a different setting where the target
lower bound for any universal constant k 2 0 (Theorem H.2
We defer the proof of Theorem 3.6 to
Appendix
H
Analysis of Population Dynamics
4.1 Symmetry of the Population Dynamics
A key observation is that due to the symmetry in the data, the population dynamics Pr has symmetric distribution in the subspace orthogonal to the vector qx- As a result, the dynamics Pr can be precisely characterized by the dynamics in the direction of Q-.
Recall that w q._ denotes the projection of weight vector u in the direction of q. Let (I - 1--- )u be the remaining component. For notational convenience, without loss of generality, we can assume that q. eι and write = (w,,), where w [-1, 1] and z E v1 w gd-2, We will use this convention throughout the rest of the paper. We will use we (x) to refer to the first coordinate of the particle ux(x) and 2x(x) to refer the last (d - 1) coordinates.
Definition 4.1 (Rotational invariance and symmetry). We say neural network p is rotationally invariant for u (w, z) ~ p. the distribution of 2 w is uniform over vI wagd-2 almost surely. We also say p is symmetric w.E. a variable w if the density of w is an even functiinn
We note that any polynomial- widthh width neural network (e.g.. P) is very far from rotationally invariant, and therefore the definition is specifically used for population dynamics. If p is rotationally invariant and symmetric, then L(p) has simpler form that only depends on the marginal distribution of w.
Lemma 4.2. Let be a rotationally invariant neural network. Then, for any target function h and any activation o,
I(p) = Ei (unp hv.d)
(4.1)
In addition, suppose and a satisfy Assumption 3.2
and p is symmetric. Then
L(p) = 82.0 - " (aaup!P - 72) 2
(Bunpl (P.[()) - -7a)
(4.2)
7

---

The proof of Lemma 4.2 is deferred to Appendix D.I then L(p) only depends on the marginal distribution of w. Eq. (4.2) says that if the distribution of is additionally symmetric and a and h are quartic, then the terms corresponding to odd & and the higher order terms for k >> in Eq. (4.1) vanish. Note that P2.a(s) 8 and (s.a(s) RS 8 by Eq. (C44. Thus, L(p) essentially corresponds to matching the second and fourth moments of w to some desired values 7/ and 74- Inspired by this lemma, we define the following key quantities: for any time t 2 0, we define D2. = Eu-p, [P2,a(w) [[))] 72 and Da,t = Euwo, (w 74. where Pt. is defined according to the population, Shinite-width dynamics (Eq, (3.6). Eq. (3.7) and Eq. (3.8).
Eq. 4.11 says that if p is rotationally invariant,
We next show that the rotational invariance and symmetry properties of pe are indeed maintained:
Lemma 4.3. Suppose we are in the setting of Theorem 3.3 At any time t E 20), Pt is symmetric and rotationally invariant.
Rotational invariance follows from the rotational invariance of the data. To show symmetry, we use Eq. (4.Ip. and the facts that Pk.d an odd polynomial for odd & and Pt is symmetric at initialization. Using Lemma 4.2 and Lemma we obtain a simple formula for the dynamics of wy:
Lemma 4.4 (1-dimensional dynamics). Suppose we are in the setting of Theorem 3.3 Then, for any X gd-1, wriling wi == wx(x), we have
dw'
-(1 (l(w)) (Pr(we) + Qc(w)))
(4.3)
di
An(we)
where for any w E [[11, 1]. we have Pr(w) = D2,w + o Ds and Q+(w) = Na 1
X (x w², where Pag" () (8) - D2..De.l103.. More specifically, X8 28 J-I
DA.
6d+12 and 2² -
Da,t Da, 2-11
6d+9
Eq. (.33 is properly defined dynamics for w because the update rule for wt only depends we and the quantities D2,; and Da.- - additionally, D2, and D4. only depend on the distribution of w.
We henceforth refer to the (x) as particles this is well-defined by Lemma 4.4
4.2 Analysis of One-Dimensional Population Dynamics
We also use & E [-1, 1] to refer to the first coordinate of X. the initialization of a particle under the population dynamics. We note that {x, e1) and therefore, the distribution of is saa For any time t 2 0, we use (1) to refer to we (x) for any E g This notation is also well-defined by Lemma 4.4 We divide our proof of Theorem B.3 jinto three phases, which are defined as follows. For case of presentation, we will only consider particles w. (.) for 1 >> 0 in the following discussion. Our argument also applies for < 0 by the symmetry of Pt (Lemma 4.3
Definition 4.5 (Phase 1). Let Wmax
logd Togd and tu Vd
Let Ti > 0 be the minimum time such
that WT, (uu) = Wmax == dogd We refer to the time interval [0, Ti] as Phase 1.
Note that essentially all of the particles are less than (u log at initialization by tail bounds for pa- vd
During Phase 1, the term corresponding to D2, in the velocity dominates, and all particles grow by a
dlogdyz factor. During Phase 1, the loss does not decrease much but a large portion of the particles have grown by a large factor. During Phase 2, the particles and their velocity will become large.
Vd
Definition 4.6 (Phase 2). Let T2 > 0 be the minimum time such that either D2.,, 0 or D We refer to the time interval [Ti, T2] as Phase 2. Note that T2 > Ti by Lemma D.8
Definition 4.7 (Phase 3). Phase 3 is defined as the time interval [T2, oc).
We divide this phase into two cases, which are both more challenging to analyze than Phases 1 and 2: (i) D2,1, O and Da,T= < 0, and (ii) D2.7, < O and Da.2; = 0. Here we discuss Case 1 the analyses of Cases and are in Appendix D.3 and Appendix respectivvl..l Suppose Case 1 holds, i.e. D2,T, 0 and Da,, < 0. Then, for all > T2, we will have D2.- 2 0 and Da.- S 0 (Lemma 1.19 and Lemma D.20), meaning that for 2 T2 a poly(72) fraction of particles will be far from 0 and (Lemma D.17). However, this does not guarantee a large average velocity unlike
8

---

in Phases and 2, the velocity v(w) P((w) + Qi(w) defined in Lemma 4.4 may have positive root (0, 1), and the root r can give rise to bad stationary points because for certain values of if degenerates to the singleton distribution on the root r, the velocity at would be exactly 0. Indeed, we can construct such an r, intuitively because D2. and Da,t have different signs (r(w) and (w) = O(1d)) is a lower order term (Lemma (L.5m
Thus, we must leverage some information about the trajectory to show that the average velocity is large when D2.,, Da,t have different signs. To this end, we prove that Earp, [(w - r) Z
Em,,r'mpp (w - w')² is large by designing a novel potential function ((w)) := log( m
We show
that (b(w) (w()) ) is always increasing for any two particles w. w' (Lemma D.13p. Because + is Lipschitz on an interval bounded away from O and 1, a lower bound on [$(w) - o (ww) also leads to lower bound (w- w') and thus the particles away from 0 and will have large variance. Recall that when D2. > O and Da.t < O, a large portion of particles are away from 0 and 1, and hence, the average velocity is large. In Appendix B we include simulations to illustrate the effects of Phases 1, 2 and 3.
5 Analysis of Empirical, Finite-Width Dynamics
Coupling Between Empirical and Population Dynamics. To analyze the difference between the empirical and population dynamics, we define an intermediate process pt, where the initial particles are from fu but the dynamics of the particles then follow the population trajectories:
ýo Āo = unif((x),...,, .mm))..
and P:. = distribution of (x()) (where x x~ pp).
(5.1)
Xm)) H), let (x(x) ue(x). Let I be the joint distribution of (xx(x), âr (x)).. Then, I' forms a natural coupling between Pe and Đc. We will use ūn and ūn as shorthands for the random variables ūt (x), ûg (x) respectively the rest of this section. We define the average distance
For X ~ unif(1x:: ........ Xm
I == a.),, [1 Te [1.) Intuitively, Spc (1) and fp. (x)
are close when ún and ū, are close,
which is formalized by the following lemma:
Lemma 5.1. In the setting of Theorem 3.4 let T S 02, Then, with probability at least
1 exp(p over the initialization, we have for all I. E [0, T) that Bzage- 10 (x) - {x)))
(#). of (do'd(lo (loga)oc)) and Bg-ga-i ((f fp. (z)) $ (o 2,d + G7,
The proof of Lemma 5.1 is deferred to Appendix E As a simple corollary of Lemma 5.1 we can upper bound E Eg- (Fp, (x) - fa. [z) by the triangle inequality. Thus, so long as Ar, is small, we can show that the empirical and population dynamics achieve similar test error at time T....
Upper Bound for AT... The following lemma gives our upper bound on AT..:
Lemma 5.2 (Final Bound on Ar..). ). In the setting of Theorem 3.4 we have Ar., S d
We prove Lemma 5.2 by induction over the time S T. Let us define the maximal distance Amax, = max ū. u. (Emup((T)) [/ l2 where the max is over the finite support of the coupling I'y- We will control A, and Amax, simultaneously using induction. The inductive hypothesis is:
Assumption 5.3 (Inductive Hypothesis). For some universal constant > Φ 1/2 and ub E
(0, Ø 2)) we say that the inductive hypothesis holds time if, for all 0 s S T, we have A. < d and Amax,t S v
Showing that the inductive hypothesis holds for all requires studying the growth of the error
||ū - well. We analyze using the decomposition _ 00, 1.11 A. B. + C., where A, - (grad, L(pt) grad, (pt),, it me), B, = Tgrad ((p)) grad L(t)), 0e wz), and C. = (grad, [(p)) grad, û, ū.). Intuitively, A, captures the growth of the coupling error due to the population dynamics, B, the growth due to the discrepancy between the gradients from the
'Note that un ūt and Ar, B. and Ct implicitly depend on the initialization x. and could be written as Ax(x), (x)x), C. and (x). but we omit this dependence for ease of presentation.

---

finite-width and infinite-widd networks, and Ct the growth due to the discrepancy between finite samples and infinite samples. The following lemma gives upper bounds on these terms:
Lemma 5,4 (Bounds on Ar: Br, Ct). In the setting of Theorem 3.4 suppose the inductive hypothesis
(Assumption (5.) holds up to time t, for some T Te.e. Let de := un Ut- Let T1 be the runtime of Phase 1 (where Phase is defined in Definition 4.5). Then, we have
46 12,1 10&11 + 0
Li D2 [1011)) Tog
if Ti
A -
(5.2)
((1) (63.a + o. |
ST
Additionally, with probability at least 1 gillordy over the initialization and dataset, we have
B, S (6 + + o2,a) [[llgg dj O(1)
A,) |00||2, BB(B:) (02, ad(Lo d)o(1)
A (0 2,d + 02 A.
vm
vm
and C. - (6 + t (log
(v 116.112+ -==-"" A 16, 1 A 18 d= 2 16,113 n
m
The proof is in Appendix H We now give an explanation for each of these bounds. For A,, we establish two separate bounds, one which holds during Phase 1, and one which holds during Phases 2 and 3. Intuitively, the signal part in a neuron (i.e. w²) grows at a rate 403 ( during Phase 1 (which follows from Lemma 4.4 and in Eq. (5.2) we show that the growth rate of the coupling error due to the growth in the signal is also at most 46 2. (D2ll Intuitively, the growth rates match because the signal parts of the neurons follow similar dynamics to a power method during Phase For example, the worst case when o is mostly in the direction of 61, the factor by which o, grows is the same as that of w. More mathematically, this is due to the fact that the first-order term in Lemma is the dominant term in v(w) - v( ()) for any two particles w, w (where v(w) is the one-dimensional velocity defined in the previous section). To get sample complexity better than NTK, the precise constant factor 4 in the growth rate 46 | D2., w is important a larger constant would lead to 67, being larger by a poly(d) factor, thus increasing the sample complexity by poly(d).
The same bound for A, no longer applies during Phases 2 and 3. This is because the dynamics of w are no longer similar to power method, and the higher-order terms may make a larger contribution to v(w)- =(()) than to the growth of w, vice versa. Thus we use looser bound 0(1) (62,a + 72 [|5,]] in Eq. (5.2). However, since the remaining running time after Phase 1, very short (only poly log d)- this corresponds the second term of the running time in (.3)). the total growth of the coupling error is at most exp(poly(log log d)), which uub-polynomial and does not contribute to the sample complexity. Note that if the running time during Phases 2 and 3 is longer (e.g. Oloo d)) then the coupling eтror would grow by an additional d01)) factor during Phases 2 and 3, which would make our sample complexity worse than NTK.
Our upper bounds on B, and C. capture the growth of coupling error due the finite width and samples. We establish a stronger bound for TE Be] than for B,. In our proof, we use the bounds for B, and [[By] to prove the inductive hypothesis for Ag and Amax, I respectively. We prove the bound for C, by expanding the error due to finite samples into second-order and fourth-order polynomials and applying concentration inequalities for higher moments (Lemmal
All of these bounds together control the growth of || ū at time f. Using Lemma 5.4 we can show that for some properly chosen & and v, the inductive hypothesis in Assumption 5.3 holds until
T., (see Lemma E.3 for the statement), which naturally leads to the bound in Lemma 5.2 The full
proof can be found in Appendix
Conclusion
In this paper, we prove a clear sample complexity separation between vanilla gradient flow and kernel methods with any inner product kernel, including NTK. Our work leads to several directions for future research. The first question is to generalize our results to the ReLU activation. Another question is whether gradient descent can achieve less than d sample complexity in our setting. The work of Arous et al. [1I] shows that if the population loss has information exponent 2, then a sample complexity of d (up to logarithmic factors) can be achieved with single-neuron student network- it is an open question if similar sample complexity could be obtained in our setting. A final open question is whether two-layer neural networks can be shown to attain arbitrarily small generalization error one limitation of our analysis is that we require e 2 poly(1/ log logd).
10

---

Acknowledgments
The authors would like to thank Zhiyuan Li for helpful discussions. The authors would like thank the support of NSF IIS 2045685, NSF Grant CCF-1844628 and a Sloan Research Fellowship.
References
[1] Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems, 34:26999-77002, 2021.
[2] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-stairease property: a necessary and nearly sufficient condition for sgd learning sparse functions on two-layer neural networks. In Conference on Learning Theory, pages 4782-4887. PMLR, 2022.
[3] Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302. 2023.
[4] Radostaw Adamezak, Alexander E. Litvak, Alain Pajor, and Nicole Tomezak-Jaegermaal Quantitative estimates of the convergence the empirical covariance matrix in log-concave en- sembles. Journal of the American Mathematical Society, 23(2):335 oct 2009. doi: 10. 1090/ 40894-00347-09-006550- URL https: h/tp::://doi... org/ 10. /ttp://0000000000000000000000000
[5] Zeyuan Allen-Zhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? arXiv preprini arXiv: 1905. 10337, 2019.
[6] Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning, arXiv preprint arXiv:2001.044: 2020.
[7] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
[8] Luca Arnaooldi, Ludovic Stephan, Florent Krzakala, and Bruno Lourciro. From high- dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks, 2023.
[9] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ru- osong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv:1904.1199 32, 2019. URL
pdf
[10] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv pages 322-332, 2019.
preprint /9Xi
[11] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research, 22(1):4788 4444788888888,,,8888888888888888 2021.
[12] Kendall Atkinson and Weimin Han. Spherical harmonics and approximations the unit sphere: an introduction, volume 2044. Springer Science & Business Media, 2012.
[13] Drumi D Bainov and Pavel S Simeonov. Integral inequalities and applications, volume 57. Springer Science & Business Media, 1992.
[14] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764 2022.
[15] Gérard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Algorithmic thresholds for tensor pea. 2020.
11

---

[16] Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. arXiv preprint arXiv:2210.15651, 2022.
[17]
Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023. doi: 10. 111011991100916661111 64. URL
httplumaaa.eetttaaaaa net.boot
[18] Alon Brutzkus and Amir Globerson. Globally optimal gradient descent for convnct with gaussian inputs. arXiv preprint / 009966 2017.
[19] Fan Chen, Zhenjie Ren, and Songbo Wang. Uniform-in time propagation of chaos for mean field langevin dynamics, 2023.
[20] Minshuo Chen, Yu Bai, Jason D Lee, Tuo Zhao, Huan Wang, Caiming Xiong, and Richard Socher. Towards understanding hierarchical learning: Benefits of neural representations. Neural Information Processing Systems (NeurIPS), 2020.
[21] Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018.
[22] Lenaic Chizat and Francis Bach. A note on lazy training in supervised differentiable program- ming. arXiv preprint arXiv:1812.079551 2018.
[23] Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn represen- tations with gradient descent. In Conference on Learning Theory, pages 5413-5452. PMLR, 2022.
[24] Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi, and Thomas Hofmann. Escaping saddles with stochastic gradients. In International Conference on Machine Learning, pages 1155-1 166. PMLR, 2018
[25] Amit Daniely and Eran Malach. Learning parities with neural networks. Advances Neural Information Processing Systems, 33:20356- 20365, 2020.
[26] Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances Neural Information Processing Systems, pages 2253-2261 2016
[27] Kefan Dong and Tengyu Ma. Toward lox recovery of nonlinear functions: A polynomial sample complexity bound for gaussian random fields. arXiv preprint arXiv:2305.00332, 2023.
[28] Xialiang Dou and Tengyuan Liang. Training neural networks as learning data- adaptive ker- nels: Provable representation and approximation benefits. Journal of the American Statistical Association, 166555) 2021.
[29] Simon Du and Jason D Lee. On the power of over-parametin prerpparamttrization in neural networks with quadratic activation. International Conference on Machine Learning (ICML), 2018.
[30] Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv: I81 I. .03804, pages 1675-1685, November 2018
[31] Wolfgang Gabcke. New derivation and explicit remainder estimation of the Riemann-Siegel formula. PhD thesis, Saga--uuuuuuuussssteeeee Göttingen, 1979.
[32] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points pointe-online stochastic gradient for tensor decomposition. In Conference on Learning Theory, COLT 2015, pages 797-842, 2015.
[33] Rong Ge, Jason D Lee, and Tengyu Ma. Learning one-hidden-layer neural networks with landscape design. arXiv preprint 17X1v:1711.00001.
[34] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 14820-14830 2020.
12

---

[35] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias terms of optimization geometry. arXiv preprint 1rxiv:1802.08244 2018
[36] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pages 9461-9471, 2018.
[37] Karl Hajjar and Lenaic Chizat. On the symmetries in the dynamics of wide two-layer neural networks, 2023.
[38] Moritz Hardt and Tengyu Ma. a6 1.04231, 2016.
Identity matters in deep learning. arXiv preprint
[39] Moritz Hardt, Tengyu Ma, and Benjamin Recht. Gradient descent learns linear dynamical systems. Journal of Machine Learning Research, 19:1-44, 2018.
[40] Elad Hazan, Kfir Levy, and Shai Shalev-Shwartz Beyond convexity: Stochastic quasi-convex optimization. In Advances in Neural Information Processing Systems, pages 1594-1602, July 2015.
[41] Daniel Hsu. Dimension lower bounds for linear approaches to function approximation. Daneil Hsu's 's homepage, 2021.
[42] Daniel Hsu, Clayton H Sanford, Rocco Servedio, and Emmanouil Vasileios Vlatakis- Gkaragkounis. On the approximation power of two-layer networks of random relus. In Conference on Learning Theory, pages 2423-2461. PMIR,, 2021.
[43] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571-8580, 2018.
[44] Adel Javanmard, Marco Mondelli, and Andrea Montanari. Analysis of a two-layer neural network via displacement convexity. 2020.
[45] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM (JACM), 68(2):1-29 2021.
[46] Sham M Kakade, Varun Kanade, Ohad Shamir, and Adam Kalai. Efficient learning of general- ized linear and single index models with isotonic regression. Advances in Neural Information Processing Systems, pages 927-935, 2011.
[47] Pritish Kamath, Omar Montasser, and Nathan Srebro. Approximate is good enough: Probabilis- tic variants of dimensional and margin complexity. In Conference on Learning Theory, pages 2236-2262. PMLR, 2020.
[48] Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016.
[49] Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent converges to minimizers. arXiv preprint arXiv: arXiv:/602.0400 1602.04915, 1050:16, 2016.
[50] Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances Neural Information Processing Systems, pages 8157-8166, 2018. 8. URL https: hopps://wwww //www//wwww/w/ aaai org/ hts//cc//iiie weaics/iinx....e wwwaddx.pppppeeeeee hhpp/AAAT/AAAIIS paper/vies/160s
[51] Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in parr-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pages 2-47. PMLR, 2018
[52] Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Learning over-parametrized two-layer neural networks beyond ntk. In Conference on Learning Theory, pages 261 2613-2682. PMLR, 2020.
[53] Tengyu Ma. Lecture notes for machine learning theory (cs229m/stats214), June 2022.
13

---

[54] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptoties and double descent curve. arXiv preprint arXiv:1908.05335. 2019.
[55] Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics, 46(6A)::747-27744 2018.
[56] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layers neural networks. Proceedings of the National Academy of Sciences, pages 177655-E6771 2018.
[57] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-fres bounds and kernel limit. Conference on Learning Theory, pages 2388-2464, PMLR, 2019.
[58] Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn low-dimensional representations with sgd. arXiv preprint arXiv:2209. 148663,39. 2022.
[59] Eshaan Nichani, Yu Bai, and Jason D. Lee. Identifying good directions to es- cape the NTK regime and efficiently learn low-degree plus sparse polynomials. In NeurIPS, 2022. URL http::/papers.. mips cc/ wwwppppppr.a.e http:/paprrfffe//eeeeeeeeeeeeeee filee//paper/2222/haaa adles3a444as7777677754400bbb4442
[60] Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparameterization:: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 1(1):84-005 2020.
[61] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks. arXiv preprint arXiv: 100.vv/805.0100 1805.01053, 2018
[62] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1820-1822 2020.
[63] Maciej Skorski. Bernstein-type bounds for beta distribution. Modern Stochastics: Theory and Applications, 10(2):211-22 1-28., 2023.
[64] Mahdi Soltanolkotabi. Learning ReLUS via gradient descent. In Advances in Neural Information Processing Systems, pages 2007-2017, 2017.
[65] Mahdi Soltanolkotabi, Adel Javanmard, and Jason Lee. Theoretical insights into the op- timization landscape of wee-parameteriieed shallow neural networks. IEEE Transactions on Information Theory, 65(2):742- 7692, 2018.
[66] Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint 16055 2016.
[67] Taiji Suzuki and Shunta Akiyama. Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods. In International Conference on Learning Representations.
[68] Taiji Suzuki, Denny Wu, and Atsushi Nitanda. Convergence of mean-field langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction, 2023.
(69] Yan Shuo Tan and Roman Vershynin. Online stochastic gradient descent with arbitrary ini- tialization solves non-smooth, non-convex phase retrieval. arXiv preprint 1rXiv:1910.12837. 2019.
[70] Matus Telgarsky. Feature selection with gradient descent on two-layer networks in low-rotation regimes. arXiv preprint wxxi:22088002889 2022.
[71] Yuandong Tian. An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis. arXiv preprint arXiv: 1003.000000000 2017.
14

---

[72] Gal Vardi, Gilad Yehudai, and Ohad Shamir. Learning single neuron with bias using gradient descent. Advances Neural Information Processing Systems, 34:28690-28700, 2021.
[73] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv: 3011.007, 2010.
[74] Roman Vershynin. High-dimennsioaal probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[75] Colin Wei, Jason D. Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pages 9709 9721, 2019. URL / neurips courips.co wttppps.c.pppppeeeeepp hwcpppppe//01//ease// 8744:9928833300440666533000000 http://boosooo-abbbttt.....t
[76] Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances Neural Information Processing Systems, pages 9709-9721, 2019.
[77] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. arXiv preprint 200.:000000092777 2020.
[78] Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Learning high-dimensional single-neuron relu networks with finite samples. arXiv preprint arXiv:2303.02255 02255,, 2023.
[79] Weihang Xu and Simon S Du. Over--aaametteriaatoon exponentially slows down gradient descent for learning single neuron. arXiv preprint 200042 2023.
[80] Xiao Zhang. Yaodong Yu, Lingxiao Wang, and Quanquan Gu. Learning one-hidden- layer relu networks via gradient descent. In The 22nd international conference on artificial intelligence and statistics, pages 1524-1534. PMLR, 2019.
15

---

List of Appendices
Additional Related Works
17
B Simulations for Population Dynamics
17
Background on Spherical Harmonics
19
D Proofs for Population Dynamics
23
DJ Proof of Lemma 4.2
23
D.2 Proof of Lemma 4. and Lemma 4.4
24
D.3 Analysis of Phase
27
D.3.1 Summary of Phase
29
D.4 Analysis of Phase
30
D4.11 Summary of Phase 2
32
D.5 Phase 3, Case 1
33
D.6 Phase 3, Case 2
46
D.7 Summary of Phase
62
D.8 Proof of Main Theorem for Population Case
62
D.9 Additional Lemmas
62
0.10 Helper Lemmas
63
Proofs for Finite-Samples and Finite- Dynamics
64
Upper bound on A
69
Upper Bound on B,
73
E.3
Upper Bound C
77
Additional Lemmas
80
Proof of Theorem 3.5
84
Missing Proofs in Section
94
Proof of Lemma 3.I
94
H Proofs for Sample Complexity Lower Bounds for Kernel Method
97
Proof of Theorem 3.6
H.2 Proof of Lemma
98
Toolbox
100
Rademacher Complexity and Generalization Bounds
100
Concentration Lemmas
102
1.3 Helper Lemmas
112
16

---

A Additional Related Works
In addition to the related works on the NTK approach, mean-field analyses, and other works analyzing the training dynamics of neural networks that we discussed in Section we also discuss the following additional related works.
There are several works which study mean-field Langevin dynamics 19 68 These works show "uniform-in -n-timee propagation of chaos" estimates, i.c. their results imply bounds on the coupling error between finite-width and infinite-width trajectories which do not grow exponentially in the time However, it likely non-trivial to apply these analyses to directly obtain good test error and sample complexity in the setting of two-layer neural networks, since mean-field Langevin dynamics may require many iterations to obtain good population loss. More concretely, Theorem of Mei et al. [56] suggests that the inverse temperature À for mean-field Langevin dynamics has to be at least proportional to the dimension d in order for Langevin dynamics to achieve low population loss. This would cause the log-Sobolev constant in Suzuki et al. [68 to be on the order of e- leading to running time of -d In comparison, we show in Theorem 3.5 that projected gradient descent with poly(d) iterations can achieve low population loss. We also note that Chen al. [1 do not study the impact of finite samples on the coupling error.
We also note that there are several works which reduce the population dynamics to lower- dimensional dynamics 2 37 g as we do Section [4.] The focus of our work is on the analysis of the population dynamics (Section 4.2) and coupling error between the empirical and population dynamics (Section 5). rather than the dimension-free dynamics.
Finally, we note that our setting goes beyond that of Abbe et al. (2: our target function does not satisfy the merged-staircase property required by Abbe al. (2). since the lowest-degree term in our target function has degree 2. The work of Abbe et al. [3] studies functions which have higher "leap complexity," meaning that terms of the target function can introduce more than one new coordinate at once, or introduce new coordinate using Hermite polynomial with degree greater than 1 (in the case where the inputs are Gaussian). However, Abbe et al. 3 use non- non-standard algorithm which separates the coordinates based on whether they are large or small, and applying different projections to each subset of coordinates.
B Simulations for Population Dynamics
Here we include some simulations in a setting which is a simplified version of our I-dimensional dynamics, to illustrate the different phases. Here, the goal is to find a weight vector w E R& where 1000, that minimizes the following loss:
L(w)= = 2 u 72)² + + w - 74) 2 i=1
(B.1)
i=1
where "/2 = 0.8 and 7/2 0.7. Intuitively, the w; play the role of the particles in the 1-dimensional
dynamics of Section 4.2] We initialize w as follows - each coordinate is initialized according to standard Gaussian distribution with mean 0 and standard deviation 0.0001. We then train w using gradient descent on L(w) with learning rate 0.1, for 100, 000 iterations. We define D2 =
Z EE of 72 and D4- 1 Ed w/- 74.
The dynamics are illustrated by Figure [ Figure 2 Figure and Figure At initialization, since the standard deviation is 0.0001, all of the wi are very small, as shown in Figure Now, as shown in Figure while both D2 and D4 grow rapidly after certain point, D4 reaches first. At the point where D3 reaches O, the distribution of the coordinates of w is shown in Figure 2 coordinates w; have grown substantially, with a substantial fraction of coordinates around 0.5 or larger. This illustrates the intuition that all of the coordinates grow by large factor during Phases 1 and 2. The weight histogram after many iterations have passed during Phase is shown in Figure 3 In our infinite- width 1 I-dimensional population dynamics, when D4 > O and D2 < O, the particles w which are larger than the root r of the velocity will move towards 0, while the particles wr which are smaller than the root of the velocity will move away from O. This intuition is reflected in Figure 3 where not as many particles are concentrated around compared to at the end of Phase 2, but the particles have not grown larger than 1 or -1 in absolute value.
At this point, the
17

---

Weight Histogram
200
150
100
50
-0,0003
-0.0002
-0.0001
0.0000
0.0001
0.0002
0,0003
Figure 1: Distribution of coordinates of initialization
End of Phases and 0 Time 24349
250
200
100
50
-1.5
-1.0
-0.5
0.0
0.5
1.0
3.5
Figure 2: Distribution of coordinates of w after Phases 1 and 2
Final Weight Histogram
350
250
200
100
Figure 3: Distribution of coordinates of ur after time passes Phase
18

---

D2 and D4
20000
40000
60000
80000
100000
Figure 4: Evolution of D2 and D4
C Background on Spherical Harmonics
In the following, we summarize the background needed for this paper based on Atkinson and Han [12 Section 2].
Spherical Harmonics. Spherical harmonics are the eigenfunctions of any inner product kernels on the unit sphere gd-1 The eigenfunctions corresponding the k-th eigenvalue are degree-k polynomials, and form Hilbert space Y with inner product (f.g) = Ez- (z)) The dimension of Y is NN.d ( (d+k-3) "). For any universal constant k, NN.d x d Spherical harmonics with different degrees are orthogonal to each other, and their linear combinations can represent all square-integrable functions over $d-1
The following theorem states that Wy. is the eigenspace corresponding to the k-th eigenvalue for any inner product kernel on the sphere.
Theorem C.1 (Funk-Hecke formula S2, [h(t))aa(tdd oo, and A
Let [-1,1] IR be any one-dimensional function with Eewnn (h(t)Px,a())) Then for any function Ys Y
kgd
Vz gd-1
E x))(z AgHx(r).. zogd
(C.1)
Legendre Polynomials. The un-normalized Legendre polynomial defined recursively by Po,a(t) = 1, Pt,a(t) =t, 2k + - (t.a(t)= 1e 3-3
(C.2)
k k-1 k d Pk-2d((), 2
tP (PP--.a(t))
(C.3)
In particular, we have
(t. = d- 12 d-1' (t. (d + () + 4)
6d + 12
3
d2 - $2
(C.4)
dc --1
1-1
For any E gd-1, Phoa((u, (1, x)) is a spherical harmonic of degree k: and, for every k, k 0 and U2 E gd-1, E.nss 1 (PR,(((u ((11,)))) P (((u,, = 1 [k k'] Pa.a((u1, u2)) (Lemma C.5
Projection Operator: Let Tlg be the projection operator vg For any function gd-1 RR the projection operator IT, is given by
(I) /Nk.d
E $-88
(C.5)
In addition, if f is function of the form f(x) = li(u 2r) for some fixed vector u E Sd-1 and one-dimensional function h [-1, 1] + E. the Funk-Hecke formula (Theorem C.I) implies
C.I.
(x) = h a (k,a((u,
Vx Sd-1
(C.6)
19

---

where hagd = (e-pa((((t)Pk.a()) is the k-th coefficient in the Legendre decomposition of h.
Note that by the orthogonality of Legendre polynomials, have
Ve Sd-1
E
PP, x))l
E [Px,(()]= E (Pa.a(()Po,((())] = =0] (C.7) Emppe
tmpad
The following lemma computes the expectation of Pk.c((u,2)) x)) with respect to a distribution on which is rotationally invariant on the last (d 1) coordinates.
Lemma C.2. Let p be distribution on Sd that is rotationally invariant as Definition 4.1 Then, for any x E Sd-1 we have Pa x))] =
Proof. We prove this lemma by invoking Eq. (2.167) of Atkinson and Han every s, [-1, 1] and 0, 3,
12 which states that for
[ Pa + - - = P
(C.8)
In the following, for x E gd-1, we write x = (fe), a) 5), and for we write u (w, 2). Note
that when is symmetric and ~ P. conditioned on i1 we have (z, () (1- ( p for any fixed z. As result, plugging in w. (e1, 2) and (1- z 1/2 (2, we get
E (l x)) un] Pr Finally, taking expectation over we we prove the desired result.
(C.9)
Additional Useful Lemmas. In the following, we present some useful lemmas about spherical harmonics and Legendre polynomials.
The following proposition computes the coefficient in the Legendre polynomial decomposition of ReLU activation.
Proposition C.3 (Lemma C.2 of Dong and Ma (77). Let o(t) max(t, 0} be the ReLU activation. For every d 2 3 and even k, we have
E tapa
(o(t)P x /2/4// 5/4 5/2 - -k)--k---0 3344
(C.10)
As a corollary, when ls is absolute constant have
IE (t))P,,((t) x 1/2 trupa
(C.11)
Theorem C.4 (Addition Theorem 122.. Let (Yk.j 1 11 NR.d} be orthonormal basis of k.
that is,
an (e,,) = وا = j]]-
(C.12)
Then for all x, z E gd-1
No.
Ne.aPh. 2)). Σ (z)
(C.13)
Lemma C.5. For every k W 2 0, 3 and u, E gd-1, we have
E (p, E)) = 1 [k k7 Pk,a((u,)) E-gd-1
(C.14)
Proof. Let (Ya, S Ne.d be orthonormal basis of Yadd with
E
(V = j'].
(C.15)
8-88
20

---

Similarly define (Yrss :isje & Netdd / for W.d The addition theorem (Theorem C.4 implies that
Naa
Pr,a((u,E))
Σ (u)
(C.16)
Nu.
j=1
Nusa
Pk Pkeal(u,
(C.17)
Nw.
Recall that for o W, kdd and Ye are orthononal subspaces. Consequently,
NE. Na
$~ugd-1 -,, [Rnal(u, E)) E()
E
E (M Enga-1
v NN ==1
(C.18)
Nk.d
1 ks = Kl
Σ
(w)(())
(C.19)
Nk.d
Euga-1
1 [k k') (k))((u,,))
(C.20)
Lemma C.6. For any 2 0, 2 3, there exists a feature mapping o S - IR Ne.a such that for every E gd-1
(d(u), o(v)) No. PE,u
(C.21)
[ 12 NR.d:
(C.22)
((())(()) I. wow
(C.23)
Proof. Let "I,... YNNa 80-1 + R be an orthonormal basis for the degree-k spherical harmon- ics Yu.d We let d(x) = (Yi(z)) be the feature mapping, and in the following we verify By the addition theorem (Theorem C.4). for every u, 5d-1 we have
Eqs. CC.1
C.23
Ne.
(Ku),
Σ Yy(u))(()) = NeaPk.a((u, ==1
(C.24)
which proves Eq. CC21 Note that for every S PR.a((u, u/) (I. 1. Hence Eq. (C.22) follows directly. To prove Eq. (C.23), note that Y1, ... YNE. are orthonormal. Therefore
for every j,j E [Nk.a]
E [M()) = 1 =1
(C.25)
Hence, Eq. C.23 follows directly.
Lemma C.7. Let pd be the distribution of (u, v) when v is drawn uniformly at random from the unit sphere Sd-1 and Sd-1 is a fixed vector. For fixed 2 1, let T1,""" Fh $d-1 be (not necessarily independent) random variables drawn from the distribution pa. Then there exists universal constant e such that for any integer k 2
(
11/h
Vt 2 4 (kldd),
Pr
max d PR.a(xt) 21) 65
2n exp exexp
(3
32
(C.26)
In addition, we also have
max d Pha((;;) S (ndd)))
(C.27)
is
21

---

Proof. To prove the first part of this lemma, we first invoke Proposition C.8
which states that
[
[ klnd
Va
1
[Pk,a(x))
2
(C.28)
When x ~ Ad, Proposition I.5 states that
(- (dd
11 O, 2)( 2exp
(C.29)
2
Therefore for every t 2 4 (k: In d)) we have
Pr P 2t) $ Pr (1x) 2 #1/2kd 10111-22--1111
(C.30)
1 1/1
$ eexp
(C.31)
2
22xxp(-111k
(C.32)
By union bound we get
(-1//
11/k
Vt 2 (klddd, Pr max d Pr.a(x;))
2n exp
expep
(-32
(C.33)
isn
32
which proves the first part of this lemma.
As corollary, for any fixed c 41+h (A: d) we have
E
max dt Ph.d( inn
Set
Pr
max d Pr,a(,i)² 21) dt
(C.34)
isn
Se +
Pr
(dd px,)(t)² 2 t) dt
(C.35)
f.
12/k
c 2n
cxp
dt.
(C.36)
32
Now we upper bound the integral by changing variables. Letting tl/k /32, we get
//
exp
dt
32
exp (-u) du
(C.37)
32
1/1
(1///)
k-1
k.2 exp
32
(C.38)
32
32
1/k
$ 3 exp
E
(C.39)
32
(C.40)
where the last incquality comes from an upper bound for the incomplete gamma functions Theorem 4.4.3]. Therefore get
31
6//h
E max d Ph.d(1;)3 s c 64nck2 exp
(C.41)
32
Finally taking e (32k )A, , we prove the desired result.
Proposition C.8. For any k 2 0, d 2 2, we have
k
Vt E
[Pa.((t) +hgh
(C.42)
d
22

---

Proof. Let pa(l) A (1 12) r(d/2) 17/2 va be the density of uj when (ui, ; is drawn
unffrrmyy from gd-1, By Atkinson and Han [1 Theorem 2.24] we have
Pk.a(1) =
[1+ (1)1((1))113))
(C.43)
amp
As result,
[P.a(t))
E 8rppa-1
[ + i(1 -t)11/8 2
(te + (1 -1)-88/////
(C.44)
By Proposition 1.5 we have,
Pr(vd 1|s| S 22/kldd s 2exp(-2k Ind) S dt
(C.45)
Consequently,
E srupa-1
+1- (1- sd
1 [
S 22kklnd/Vd 1 (t2 ( s2)6/2 (C.46)
grppa-1
2k In d
h/2
S d k +
2
(C.47)
d d-1
Hence, when t 2
klnd
we get
( 2nd
6/2
(R.a(t))sd +
S 21
(C.48)
d d-1
D Proofs for Population Dynamics
D.1 Proof of Lemma 4.2
In the following, we prove Lemma 4.2 which provides way to simplify L(p) when p is rotationally invariant and symmetric.
Proof of Lemma 4.2
Recall that the population loss
L(p) = E
,,)))?
(D.I)
x~
and the activation function R - lR and target function h IR + R has the following decomposi-
tion:
a(l)=
Σ oR Pk.al).)(t),
(D.2)
h(t)
Σ
ha.
(D.3)
k=0
Therefore we have
2L(p) = - -
(D.4)
z
E
(o((u, x))] - 11((111, -M((,,)))
(D.5)
z-ga=1
urp
E
E
Gr.aPhaa
Σ k=
hr (haaea(((,ee))))
(D.6)
2-ga-1
k=
(okd IE P (P(a((u,a))) - MM.aPa,(eoeee
(D.7)
2n-ut1
umup
km
23

---

When the neural network is rotationally invariant, we can simplify the above equation by invoking which states that
Lemma C.2
( = umpp Pead(e))
(D.8)
unup
Continuing Eq. D.4p above we have, 21(p) =
Ōkd E ump
(R,a((e1,
- a (k.a((ei.
(D.9)
k=0
(okk ( k.a
(D.10)
2--8d-1
ump
k=0
Now we expand the square by invoking Lemma C.5 In particular, Lemma C.5
states that
=)) 2r-
[k: (x)(((e))))) 1 =1.
(D.11)
Consequently,
20
2L(p) Σ
Ond E (haalee, u}}] hg.d)) ump
(D.12)
as desired.
To prove the second part of this lemma, in the following, assume that (1) hv.d 0, Ak & {0, 4}.
(2) ho.a oo,d, and (3) that a is degree-4 polynomial. Then we get
L(p) = 2.d
E 12.((w) - 72)²
0g 2
E PPa(([)) - 74 + Σ ump
kid
E PP p.,[(w]]]]
2 ump
2
umpp
kE(1,3)
(D.13)
Since is symmetric and (w.)(w) is an odd function when ie is odd, we get [Prta[w)] 0.
Vk: E (1,3),
E uwup
(D.14)
It follows directly that
02.d I(p)
)²
[a.(ww] 77a²²
E (P)d((W] 72) umup
E ump
(D.15)
2
D.2 Proof of Lemma 4.3 and Lemma 4.4
The goal of this subsection is show Lemma 4.3 which states that Pt remains symmetric and rotationally invariant under the infinite-width population dynamics, and to show Lemma 4.4 which describes the dynamics of the first coordinates of the particles in pt.
The following lemma is a first step. Recall that when is rotationally invariant, then Lemma 4.2
allows to rewrite L(p) Ey-ge-1 (Jz(x) - v) as 1/2 E (ōr.a Eurp (Prpp[)) hu.d
The following lemma states that the gradient from the two formulas for L(p) are the same when is rotationally invariant.
Lemma D.1. Let be distribution on @d-1 and let be distribution [-1, Define
1 Σ (ōka E [Pr.a(w)] -hv.d) F(v)
(D.16)
2 k=0
wwwy
If p is rotationally invariant as Definition 4.1 and v is the marginal distribution of w for u
(w, p. then for any particle (w,2), have
(1 (1- VV m F(v) = grad..
(D.17)
where grad,,((p) denotes the first coordinate of L(p).
24

---

Proof of Lemma D.I For any particle u (w, 2). we write the first coordinate of (.L(p) as V [p), and the vector consisting the last (d 1) coordinates as VpL(p). Observe that
grad L(p)
(e1, gra.grad_.10
(D.18)
(e1, (I uu V
(D.19)
v..((p) - w (u, V V.. (.))
(D.20)
= (1 - V - w (z, LpD(
(D.21)
On the other hand, suppose satisfies the rotational invariance property, and that v is the marginal
distribution of the first coordinate under p. Then,
write p in terms of for any w E [-
the distribution of conditioned on w is v1 w2gd-1, Thus, since L(p) F(v) by Lemma 4.2
we have by the chain rule that
du Ow
V VwF(U)
E
(D.22)
0= Ow
= vwL(p) +
V (p) p))
(D.23)
8-
3
= v..((p) +
(V
w
(D.24)
8-8d-2
v1
vuL(p) +
E
(VeL(o)
w
(D.25)
1 w
= (uT(p)
w
E awou
(V =)}
(D.26)
2 1-
Moreover, for any fixed w
[-,,1], the quantity T 2) does not depend on 2, since by
Lemma D.2 for any rotation matrix and o' Az, we have (V= = (A. L. A2)
LV)) z). Thus, for any w E [-1, and = v1 - 2dd-2 we can write
(1- (v) = (1 - V L(p) - w (z. V L(p))) = grad L(p)
(D.27)
as desired.
In the proof of the above lemma, we used the following fact which we now prove: when p is rotationally invariant, V (p) is rotationally equivariant as function of u, i.c. if u is rotated by
rotation matrix A which only modifies the last (d - 1) coordinates, then V (p)( is rotated by A as well.
Lemma D.2 (Gradient is Rotationally Equivariant). Let p be rotationally invariant distribution as in Definition 4.1 Let A Rdxd be rotation matrix with Ae₁ El, i.e. only modifies the last (d- 1) coordinates. Let E gd-1 and u' = Au. Then, L. = -A V V.I(p), and L(p).
gr
Proof. The proof is by a straightffrwwrdd calculation:
V L(p) = E. [()(())))
(D.28)
E. [[/((>) (((()))-(((e1))))))) (()1,))))) x))=a)
(D.29)
= E.
(UA(())) - ((e,,, )o ((d', (Ad) Ax]
a-ge
(By rotational invariance of uniform distribution on gd-1)
A
E
[(6(Az)) h((e1,(eeeee Az) ((u', (x))x)
(D.30)
2-
A
E
[(f((Ax) - (e((1,, Ax)))o (o ((Au, Aa]
(By definition of u')
A
E
[(f((A)) n((11, Ax)) () x))x
2~
(B.c. A is rotation matrix so A A = Iaxa)
25

---

By the rotational invariance of we have fp(Ar) = fp(x), meaning
Ve T(p) A
E [Up()))) h((e,, Ax)))d'((u,
(B.c. fp(Ax) = fp(2) by rotational invariance of p)
= A E - p-
(((1, x)))) (B.c. Aei e1 and A = ldxd)
=A (p)(p)
(D.31)
This proves the first statement of the lemma. The second statement also follows from similar calculation:
grad L(p) (I (I--u'(e w'((() L()
(D.32)
= (I Auu T AA) A V.L(p)
(D.33)
A V.L - Auu T A A- V.L(p)
(D.34)
A (. AuuT. V.L(p))
(B.c. A A laxd)
A (1 uu T) (.L(p)
(D.35)
= A grad, L(p)
(D.36)
desired.
As corollary, we obtain our first formula for the dynamics of the 1-dimensional particles:
Lemma D.3 COne-Dimensione Velocity). Suppose we are the setting of Theorem 3.3 and that ps rotationally invariant. Then, for any particle ut (where we omit the initialization x for convenience). we the first coordinate of uv, then
dwi dt
-1
w
Σ km
(ōnad E [Pe,a[w)] - hra.)) ox.aPed' wmpps
(D.37)
Proof of Lemma D33 This follows directly from Lemma D.I and because the activation a and the target only have nonzero coefficients ox,d and higd for ls = 0, ,,......
We now complete the proof of Lemma 4.3 i.c that Pt remains rotationally invariant and symmetric if it is defined according to the population projected gradient flow as in Eq. (3.6) and Eq. (3.7). p. To prove that O: remains symmetric, we make use of the above formula for the I-dimensional dynamics.
First, we show the rotational invariance property. Assume inductively that at time t. Pt satisfies the desired rotational invariance property holds at initialization since Po the uniform distribution on gd-1, Consider particle and let A E IRd be rotation matrix which only modifies the last (d - 1) coordinates. Let ut Au; another particle. Then, it follows from Lemma that that
Proof of Lemma 4.3
duf
du
A
(D.38)
dt
dt
Thus, for fixed [-1, if
v1 u 2 then the distribution of da is uniform on cgd-2 for a fixed scalar that depends on w. Therefore, will remain rotationally invariant.
Next, we prove that is symmetric. First observe that P: is symmetric at initialization, since it is the uniform distribution on $d-1, Now, suppose that Pt is symmetric time t. The polynomial PR.a
odd if & is odd and even if l: is even - this can be seen from induction on Eq. (C.2) and Eq. (C.) Thus, Euy-ps (w)
for odd by our assumption that Pt is symmetric. Additionally, recall
that we defined h so that ão.d ho,d and hedd = 0 for odd k. Thus, by our definition of D2, and
DA in Section
dw: -1- (2, (w)) + Pa.d
(D.39)
dt
by Lemma D.3 Since Pz.d and Pa.d are even polynomials, we know Pz.á' and Pa.d are odd polynomials, and thus di
duc is odd polynomial in ws In other words, if w, and wa are the first
26

---

coordinates of two particles ot and un respectively, such that wa -w then del the distribution of dwz is symmetric, meaning that pr will remain symmetric. di
dut de
Thus,
dr
In the rest of the proofs for the infinite-width population dynamics, we make use of the symmetry property. In particular, many of the lemmas are stated for w > or > - however, the generalizations to in this section.
< 0 also hold. We use this symmetry implicitly throughout the proofs
In Lemma we are simply rewriting our formula for the 1-dimensional dynamics in more convenient form. The following proof shows the detailed calculation.
Proof of Lemma 4.4
By Lemma D.3 and the fact that only the second and fourth order terms are nonzero (which follows from Lemma 4.3, the fact that Pk.d is an odd polynomial for odd l (Eq. (C.2)
and Eq. (C.3) and because hi,d = ha.a 0), the velocity of wo is equal to the following:
-1 grad u L(p) (1 (1--w) -²) (02, P2 (w) + (a(1) Pa.a (a.')))
(D.40)
where grad [(p) denote the first coordinate of L(p). For convenience, during the 6d4
rest the proof of this lemma, let () 2 na L 4dd-8 and na(3) d=
Then, by
2-
Eq. C.4).
grad v L(p) = -(1 - w (62, D2(1P + 0 ( (.(())(a())))
(D.41)
=-(1 w² (62,) 2nd 0 Dz(t) (4 2nd (1w)
(D.42)
-1 w²) (P((we)) D2 (na (1) 1)w
(D.43)
D4(t) (na 1) - 20 20 D4 (t)nn
(D.44)
-- w²) (Pi(w) Q+(wo)
(D.45)
Here we have chosen Aa ()) = (7d - 1)D2(t) - D (3) and X (3)
4 ( 1). Observe that () (aa 0 o3nalDaal (eee) desired.
Dael)
D.3 Analysis of Phase
Intuitively, during Phase 1, the particles are all growing a similar rate because the particles are all less than Wma log in magnitude, the cubic term involving 3 in the velocity of w does not have a significant effect on the growth of w. We formalize this in the following lemma.
Recall the formal definition of Phase 1 in Definition 4.5
in the setting of Theorem 3.3 Consider time interval [ta, t1] E [0, Tb] in Phase or 2. We are interested in quantifying the (relative) growth rate of the neurons indexed (or initialized) at in and l2 (where 0 < l 12). defined
Lemma D.4 (Uniform Growth Lemma). Suppose
Wt, (41)
F
Wro (11)
Wes (12)
Fg
Wto (12)
Let 8 10,1) such that Va S 8 5 log F5
here, we implicitly assume that F2 S ed We also assume
that 8 < 12 16
Further assume that Wt, (12) 8. Finally assume that IP([c] 2 |(2)) <
z Then,
16
R
(D.46)
and additionally,
Fi x exp (f/ 20, 12,
(D.47)
27

---

Finally.
t1 to 2
log F2
(D.48)
0 2,
We note that Fy and F22 are both at least 1.
Note that this lemma also holds for l2 < 0, and with we, (12) -6, by the symmetry of pt-
Proof of Lemma D.4 In this proof, we define = //2 - then, by Assumption 3.2 we can write 74 = 8 78 where 1.1 I and 33 is less than some universal constant 61. Suppose at time f1,
Wt, (12) 8. One useful observation, which we use in the rest of this proof, is that for all t S &1,
w/ (12) S since for w V have dur 2 by Lemma 4.4 (here used the assumption that di
6 2 (a Thus,
E [P2,a(w]] = IP(1| (121) wr<p
E (2) |6]
[22]]
P(4| < (12))
E (1) |d
|eall (D.49)
wmpp
wmp
12
+
E
(2) 14| < [t2]]
16
wmp
(B.c.
[22a(1) S 1 by Eq. (2.116) of Atkinson and Han 12 and P([( > ((2)) S 8)
<
12
+ 8²
+ +0 ((2)
16
(By Eq. (C.4 and b.c. (ux(()) S 5 if |4| S [12| by Proposition D.37
74
(B.c. 8 < 22 and 7/2 = r2)
$
8
T6
Thus, writing 72 = 72, we have
74 + +0(2)
,2
D2.1
E [P))( 72 - wrup
- -22
(D.50)
where the last inequality is by Assumption 3.2 b.c. d 2 cg and 71 2 72. By a similar argument,
D =
E [P.c(w)] - -B-1
(D.51)
wmp
1
+ 8t + (1)
S 8T
16
(Using [Pa((W) s 1, P(4| > ((2l) I
12 16
8 I +
and
Eq. C.4
16
< (3 + 1/8)7
(B.c.
S
Br'
16
$ D2,1l
(D.52)
Thus, for any time / E [l,, 41,, for any e E (2). by Lemma 4.4 (and because wx(1) S for any 6 [41, 12] by Proposition D.37). we have
ve(w((()) = (1 w D2, (1 + (())) + Qc(w((())
(B.c. 02,4 S 02,0 by Assumption
3.2 Da.el - D2,rl. and wx(() S 6)
22 D2.rl
20 D2. w (1) (1 + 0())) - o
[w)(1)
(B.c. (wx(() < 8 and by bound on coefficients of Q. from Lemma 4.4
26 D2.1 w (1) (0))((144))) (1 0()))
(B.c. 8 )
In summary,
v((w((1)) = 262 D2, (w)()) =1=)
(D.53)
Thus.
v(Wr(12))
1 1 0(6) v(wr(41) < (1 0(0) w. w.(11)
v(w((L1)
(D.54)
w/ (12)
I 0(8) wr(11)
28

---

and wi (12) grows by factor
F/ exp (S/²) o(wrl(s)) d1) s exp (1
[" o(wz((1) di) F1 r21+(00)
(10) 0()))
(D.55)
wi((2)
wr(11)
Rearranging gives
P1 2 P2 (6))
F2
(D.56)
F2 (006)
Finally, observe that F0( < 0(1,, since by the assumption that ô S log F4
F S F2 10
(D.57)
and for any >> O, शमक meaning that F2 O(1). This implies that Fj 2 F2. From the above calculations, we can also obtain
exp (f.
2 exp (f/²
1 ve(w((12)) dt)
20 ad
(By Eq. (D53
1+ O(8)
we((2)
t ve(we((2)) wr((2)
2 exp
(1 (1-)((8)
dt)
(D.58)
2 F22 (66)
(D.59)
F2
(By Eq. (D.57
as desired. Similarly, we can obtain
exp (f,² hh)
$ exp (f/ ((+)00()))
"u(ur(uz)) wr(12)
26 12,
d1)
(D.60)
F2 1++0
(D.61)
S F2
(B.c. 5 < log
Finally,
Fi S exp (f^(1 (f((1+0(6)) (((w((12))
dl
(D.62)
we(12)
$ F2 (+)
(D.63)
$ F2
(B.c. 6 < log Fz
desired. Finally, show the running time bound. Since 5 72 and ?2 is less than a sufficiently small universal constant by Assumption [.2]] for any time t during this interval, ((w) Z (2))w(())))(
o (12) (where the second inequality is by Eq. (D.50 Thus, by Gronwall's 's inequality (Fact I.13 the time elapsed when wy (12) grows by factor of Fe is
at most 02 log Fz.
D.3.1 Summary of Phase
As a direct consequence of the above lemma, we can show Lemma D.5 the particles w, grow during Phase 1.
which characterizes how fast
Lemma D.5 (Phase 1). Suppose we are in the setting of Theorem
3.3 At the end of phase 1, i.e. at
va time Tú, for all E (0, have WT, (1) (Tngdzzl, Furthermore, T S 07,02 log Additionally, we have exp
Va X (log dy2"
Proof of Lemma
D.3
To show the lower bound on Wr, (t), we apply Lemma D.4 with to = 0 and logd and with = and i2 uu- Let verify that the assumptions of
41 Tí. with
Lemma Lemma h.4ldod:
29

---

Observe that why (av)
1/ logd
[log dlogd)s: meaning that log F2 S log and log P2 2 6.
v7
cu
log d/va
By Assumption
3.2
since d 2 c3.
log d
16
The assumption that ([( 2 uT)
holds by standard bound on the tail of paa.
Using the notation of Lemma
D.4
Fe WT, (tv)
Wmax
(D.64)
(log d)/vd
(log d)²
where W/mxx is as defined in Definition
4.5
By Eq. (D446). we therefore know that
vd
WT, (c)
vd
2
(D.65)
(log d)²
(log d)2
For the bound on the running time, observe that by Eq.
(D48) Ti S
Va 1 log
The final
22,772
(logd)² dya-
statement holds because F2 the proof.
va and from the second statement of Lemma D.4 This completes
[logdy
D.4 Analysis of Phase 2
Recall the formal definition of Phase from Definition 4.6 Our analysis in this section has two main goals: (1) to show that the running time of Phase 2 is small, and (2) to show that majority of the particles become at least log log magnitude, up to constant factor. We will achieve goal (2) using
Lemma D.4 then, we show that goal (1) holds using goal (2). Additionally, goal (2) will be useful
for the subsequent phases as well.
First, the following proposition defines range particles which will be relevant during Phase and Phase 3. This range of particles is a strict subset of the interval (0, (u) which we considered during Phase
we will show that this new range of particles grows uniformly during the beginning of Phase 2, until the largest of them becomes about loglog in magnitude. Specifically, we define
xyα where k = logloga The following proposition states that at initialization, a very large fraction of the i's have magnitude between 41 and (R.
IL
A < and IR A
Va
Proposition D.6. Let K E (0, 1). and suppose i is drawn uniformly at random from Pd. Define and lg ava Then, P([e] & (k)) S K.
4L Va
Proof of Proposition D.6 First, we upper bound the probability that [c] 2 4g. By Markov's 's inequal- ity,
S E[(3]
1/d
P(M| 2 R
x²
(D.66)
I/(x2d)
Now, we upper bound the probability that |t| S L. By Eqs. (1.16) and (1.18) of Atkinson and Han
[1]. the probability density function of i is p(() ( (1 - - Thus, we can simply
r(d/2)
upper bound the density by its maximum value:
P(M| S 4
r(d/2)
(1 (1-8) -2 2 dt
(D.67)
(TI((d - 1)/2)
-u
T(d/2)
211
(D.68)
VT - 1)/2)
$ Vd.u a
(By Stirling's 's Formula)
Vd. k
(D.69)
Vd
(D.70)
as desired.
30

---

Next, as a corollary of Lemma D.4, we show that for portion of Phase 2, the particles initialized L E 1 4Rl grow by similar factor. Specifically, this true until the largest of these particles is Ç log in magnitude.
Lemma D.7 (Corollary of Lemma D.4. Suppose we are in the setting of Theorem 3.3 Let TR > Ti
be the first time such that WTy (UR) E 2log log Then, for all with [LL:RR) AR]. we have
Iu (1) 2 En², and additionally,
wry (1) = En(log d)?. Furthermore, TR - T1 S 22,0 log log d. wr, (1)
Proof of Lemma D.7 By Lemma D.5 wr; (1R) = (log (log = <[log dy2 Let 4 = x and lR xVa va
where K = log log d Now, we apply Lemma D.4 with lo T, and 1 = TR. and with 4 1 for some Note that we assume without loss of generality that t 0 - the statement for 0 follows by the symmetry of pr. Let us verify that the assumptions hold:
&. [LL, (R] and (2 'g. and with 8 = log log
First, 8 = 2log logd 2 1. Addditionally, observe that by Lemma D.5
wrry (1n)
v
meaning that WT, (IR) x df Thus, using the notation of Lemma D.4
we have
F2 - 1/ logd I/[x[log d)J
asllog d)
log log d' and log F2 S 2 log log 2 log log log
C
< J. as
desired. (Here C' is a universal constant, and the last inequality holds by Assumption 3.2 since d 2 cz.)
By Assumption 3.2
log log S 16 since d 2 cs.
Additionally, the probability that |o| 2 |CR| is at most log log up to constant factors by Proposition D.6 and this is clearly at most
16 by Assumption 3.2
Using Lemma D.4, we obtain
Wra (1) WTx (AR)
x
x En(log d)2
(D.71)
WT, (1) Wr, (4x) wr, (4R)
Kellg dj=
Thus, using the conclusion of Lemma D.5 that WT,
va (log d)aa + we obtain
wr.(1 (1) 2 (1) Wr; (4)
2 Ex(log d)²
Vd Va~ Ek²
K
(D.72)
wr, (1)
(log d)2
as desired. Thus, we obtain the first statement of this lemma. In addition, by Eq. (D.48) and since WTx (IR) = E and WT, (TR) Z allogay the running time bound we obtain for this part of Phase 2 is
TR - Ti E T
log d/(K[ S " (log log d), as desired.
We now prove some auxiliary lemmas. First, we show an upper bound on L(py), which is needed
in order to later show an upper bound on the time required have [.p.) S (ā3, a + o..,).?. In the process, we also show a reasonably tight bound on the magnitude of D2, and D4,: during the first part of Phase 2.
Lemma D.8. Suppose are in the setting of Theorem 3.3
and Lemma D.7 and suppose 0 < t TR. S 2 As a corollary, for all t > 0, L(p:) S
Then,
3T2 - < D2.t 22 and - 3n4 < DA.t
5 (62.a + o2,))
Proof. Since S TR. we S = 21og log d (because for w 2 Va we have de
> b Lemma 4.4
By Proposition D.37 for all with [e| I LR. (w)( S &. Thus, by Proposition D.6
E wrupe
2 < P. pm (l1) E (R) H[w |4| < 4R] + P Pappo (le) 2 1R) E[w
|4|
4R]
(D.73)
(By Proposition D.6
Thus, D2, = 0 + +x) 72. meaning that 3rd < D2 < 32 and similarly, - 3%4 2
< D4 <
The corollary follows from the fact that 74 S 72 by Assumption 3.2
0
31

---

Next, we show that the particles are in fact increasing in magnitude during Phases 1 and 2. This makes use of our bound on the magnitudes of D2,t and Da. up to time TR.
Lemma D.9. Suppose we are in the setting of Theorem 3.3 and suppose [to, tı] is a time interval with 0 S to < 41 S T2. Then, for all with [0| E [4, 1R) Wh () is increasing on [lo, 41].
Proof. Recall that by Lemma
v((w) -11 w (P((w + Qc(w))
(D.74)
and for w 2 0, it suffices to show that Po(w)) + Qc(w)) & 0. For t - TR.
Po(w) + Qe(w)= $ 20 D2 + D Dew +0 0 02.alDe.t + o Daad) |w|
(D.75)
d
02.0 D2.1 + oā Da.el |ul 72w + - 22
d
(By Lemma D.8 b.c. D1, S 0)
< (B.c. न., S 02 and 72 2
Daal and 74 Z
|t.l (Assumption 3.2
d
Fort 2 TR. suppose w 2 E (which holds at time TR for E [4,, 4Rl by Lemma D.7. Then,
D2
02,a [4.
P((w) + Q+(w)= s 2 + 4 + 0
|w|
d
(By Lemma 4.4
S 2 D2. 2w - Cw - + 22 Da.t (o..e((1000 (4w²
-
Cw
(D.76)
for some universal constant C' > 0. The final expression on the right-hand side is negative to see this, note that 2w 0 if w 2 Éx², and 4w w if 2 Ex² since d 2 o3 by Assumption 3.2
This completes the proof of the lemma when 2 O. For < O, the lemma holds by the symmetry of Pe.
D.4.1 Summary of Phase 2
Finally, we show Lemma D.10 which concludes the analysis of Phase 2. For times > TR within
Phase 2, the particles with [ E [4, 4R] satisfy (t)) 2 E Using this observation, and the fact that D2,, and D4, have the same sign (meaning that the two terms of P. (,(w)
203 -2 D4 w do cancel) we show that the velocity v(w) is large on average. By Lemma D.39 this allows to show that the loss decreases quickly at any time t > TR in Phase 2, meaning that either the loss goes below (62.a + 0 ))² quickly, or we exit Phase quickly.
Let T2 be the minimum time such that either D2.T. = 0 or D4,T, 0, Le. T2 is the end of phase 2. Then, either
Lemma D.10 (Phase 2 Summary). Suppose we are in the setting of Theorem 3.3
T2 Ti S
(log log d) log (?) ( or T.,. T1 S (log log d) log
?2
t0
Proof of Lemma D.10 First, let TR be defined in Lemma D.7 By Lemma D.7
Tr
Ti S
02,272 log log d. Furthermore, at time TR. for all such that [| E [4, AR]. [w,(1) 2 ER²
Our proof strategy now to show that vww) is large on average, and then apply Lemma D.39
At any Thus,
time / (TR,Te] since D2, S O, Proposition D.38 implies that lE [w²] 7/ + 0
by Markov's 's inequality, and because d 2 (where C3 is defined in Assumption 3.2)
P Pwrp, (wl) 2 1/2) < 4. (72 + 2 5T
(D.77)
For all with [g] [LL, <g]. (wr(() increasing as a function of on [0, Tzl, by Lemma D.9 Thus.
by Proposition D.6 and union bound, P Pwwp. (Ex:² < (x)) < 1/2) 1 - -(() - 572 2 2 for all
t > Tk and all with |t] [eL, (Note that 1 - O(k) 5 _/ holds because 72 is less than
32

---

sufficiently small constant by Assumption 3.2)
Finally, for all w satisfying En² S w S 2
using
Lemma 4.4 we can compute that
[w)(w] 2 (1-w²) ((P((w)) - -P((w))-\(((w]]]
(By Lemma
( ( D2,1 1.1 02,d Pa
2 (1- - w²) |2,,alD2, D2,11lu0 46 Da /|000/ Da.e||w01
(By Lemma4.4
P4.el .5"m0) - 0 1 (
- D2,1l
+ 0 Ds.1
Da.e
2 (262,a) D2,1En2 + 46
100
d
(B.c. En² S |wo - 2
2 (26 D2. 40 -
(B.c. d is sufficiently large (Assumption 3.2
Thus, by Lemma D.39
(pr
S - - D2, Ex2 + (4, xx)²
(D.78)
dt
2 - D2,1 0 1)
(D.79)
I 152
D2,1 + ,², D1,a)) ((D2,l + (Daal) (
(D.80)
I
D2, + 0 Da.)²)
(D.81)
I (62 +
(By Lemma
4.2
Thus, since L(PT)) S (d]
+ a 12 (by Lemma D.8,, by Gronwall's inequality (Fact I.13
know that for t [Tk T2].
L(pt) S L(PT)) exp (fx -5Ox + o2
(D.82)
= [(PTr)e (n=TW)"""(03,
(D.83)
Thus, if L(p)) 2 (62,a 02 ,, this implies that
6²
(t- (t-Tw) 12 t0
(D.84)
72
and rearranging gives 1 TR <
( ( log( 3).
Thus, either T2 Tk <
(0 1z log( (),, there exists time 1 [TR. T3. such that I(pt) (o
and such that t Tk I (o). to log(2 (2). This completes the proof of the lemma.
D.5 Phase 3, Case
Phase 3, Case 1 is the case where D2,1, 0 and Da.;, < 0. ie. the where D2,4 reaches first. As we later show in Lemma DD119].1dd Lemma [D200] this occurs, then for all t 2 T2, it will be the case that D2,, 2 0 and D4, < O. We now outline our analysis of this case the goal of our analysis to show that either D2, < or DDael= holds within t0
poly log log d time after the
start of Phase 3, Case 1.
Proof Strategy for Phase 3, Case 1 Our overall proof strategy in this section is to show that [u(w)] is large for large portion of particles w, and then to apply Lemma D.39 to show that the loss decreases quickly. Recall that
vt (w) - (1 w2 (P:(w) + Q+(w))
(D.85)
and intuitively, we can ignore Qe(w) due to the factor of 1 in its coefficients, Thus, we can write
(w) xe -(1 w2 P((w) ~ -(1 w²) w
(D.86)
+
(D.87)
33

---

Since D1. < 0 and D2 > O, we can further rewrite the above as
ve(w) = -11 w²) w 46 Ds. r +r)
(D.88)
a. Da.
where 2.
is the positive root of the last factor (26 D2t + 1 (a. Thus,
to show that the velocity (r(w) is large, we want to show that there is a large fraction of particles
E [0, 1] simultaneously satisfying the following: (1) is far from (2)
far from O, and (3)
w is far from r. Indeed, in Lemma D.17 we show that out of the particles w E [0, 1, at least
fraction of these particles is in [72 from and 1.
72 1/2, enabling us to bound the distance of these particles
Thus, to show that (w) is large for these particles, suffices to show that |! -w-1i r| is large on average for the particles E [2// :1 - 72/2). i.e. the conditional expectation Beo-pp [( - r)²
w E 3/4 1 722
1/2l is large. In turn, this is at least the conditional variance of w conditioned on w
being in l?2
3/4 1 722 1/21 it suffices show that Var(w E h/2 3/4
1 7/2 1) is large, and by
Proposition D.40 it suffices to show that [[[ w²)² w. w' E (12 3/1
7/2 2/2 is large.
To show that the quantity [[[w w²)² w, w' E 3/4 7/ 1 is large, intuitively we would like
to show that during Phase 3, Case 1, w and w' are getting farther apart. However, this not true. 20 (2.1)), then the last factor is function of w E [0, 1], and thus "pushes" the different particles w, w' further apart. This does not work as a formal argument, because as two particles w, become closer to 0 or 1, their distance |w- w/| will decrease again due to the factors w and (1 w²) in the velocity.
we write the velocity ve(w)= (1 w²) w (40 D indeed increasing
However, we are only concerned with particles w E In2 3/4 1 72 1/21 we must make precise the intuition that the behavior of the particles very close to
does not matter.
To make this proof strategy precise, we define the potential function ((w) = log VI-w" Definition D.II and show that [Ø(w) - (ww')l is increasing in Phase 3, Case 1. As shown in Lemma D.12 we have &
T in
di ( = 22 D2, 40,, D4, - this is simply because dd((w dw
w(1-wPj In other words, dt (ww) is exactly v(w), but without the factors w and (1 w²)
we have selected this potential function ((w) specifically to eliminate these factors. Using this observation, easy show that [2(w)= (ww)) is increasing and thus obtain a lower bound on ((w)- ((w')l for a large fraction of pairs w'. Finally, to convert this to a lower bound on
3/4 1- 1/3 (Lemma D.14 which |w w/), we use the fact that Lipschitz on the interval [72 gives |w w/l p (w) (ww').
We now repeat the definition of the potential function:
Definition D.11 (Potential Function 4). We define & [0, 1] + (-o0, oc) ((w) log
(i-w
We note that d du w21 Next, we note the following useful fact about the potential function (we note that this does not require any assumption that we are in Phase 3, Case 1, and just follows from algebraic manipulations):
Lemma D.12. In the setting of Theorem 3.3
(w:))) -222, - a 21 (3) ())
(D.89)
di
where Xa (2) and Xa (3) are in the statement of Lemma 4.4
Proof of Lemma D.12 For any Wt.
d ((we) = dΦ dw, 1 dw
(D.90)
di
dw di w(1 w²) di
34

---

make this Thus, writing dwz obtain dt -1 (1-w²) (P((w) + Qe(w)) where Pt and Qi are defined in Lemma 4.4 we
dw,
dt (ww))
(D.91)
w(1 w²) dt
1
(Pi(w) + Q+(w))
(D.92)
Pr(w) Qe(w)
(D.93)
w w
Qr(w) w
2, --20 2, D2,t
2 w
(D.94)
4 D4, Aa (1) Ad w
(D.95)
where X () and Xa (3) are as in the statement of Lemma 4.4
Intuitively, when D2 > O and D1 < O, the main part of the velocity is Pc(w)) = (00u))==23...00 2 D2 46,..D,,tw which an upward sloping cubic polynomial the positive root of this polynomial, and the distance between any pair w, w' of particles increasing provided that and w' are not too close to 0 or 1. In the following few lemmas, intuition formal using the potential function Q. We first show that for two particles w, , the distance between ((w) and (ww") is increasing as long as D4.t S O. Note that this condition holds not only during Phase 3, Case 1, but also during Phases 1 and 2.
thus, the particles w move away from
Lemma D.13 ($(w) and F(w') Moving Apart). the setting of Theorem 3.3 suppose D S 0 ffr all t in some time interval [Lo, ti], and let (1, 12 > 0. Then,
[v(e((1)) ((v((12))
(D.96)
increasing on the interval [Co, 11].
Proof of Lemma D.13
Suppose 42 E [0, 1] such that 4 < 12 - by Proposition D.37 we will always have wr((1) S (t((2). For convenience, in the rest of this proof, we will write W1,t we (r((1) and W2,t (12))) Since is an increasing function of w, it will also always be the case that
o ((wz,) 2 ((w1,)) Thus, to show that ((wz,)) ((w1,1) is increasing, it suffices show that
(W22t) ((w11.) is increasing. By Lemma D.12
d (w2,,) - <(wLe)) = ( 262 2,d D2, - 46 D2 Aa - Aa 102,
(D.97)
dt
- 40 D5 Du, Aa - Aa (3)m,
(D.98)
--40 44, w Xa (3) (w², w
(D.99)
=40 D1, w Ao (3) (w3, w
(D.100)
where Xa () and (3) are in Lemma 02,0l Da,el a. Therefore,
4.4 Recall from the statement of Lemma that that MA (0) S
1
d ((2,,) - ((W1,))) 46 D4. (w.. wी Aa (3) wF.e)
(D.101)
di
2 36 D4., [w33 - w
(D.102)
and the right hand side is nonnegative since w2, 2 11,1 2 O. Thus, ((w2,) 4 (w.,)) is increasing, as desired.
Next, we prove a helpful lemma showing that the potential function is bi-Lipschitz. This useful both when we show that ( +(w') is initially large (as it allows us to leverage an initial lower bound on Iee w/l) and when we show that w/ - w/ is large later during Phase 3, Case (as allows us to leverage the lower bound we obtain on [(w) +b
35

---

Lemma D.14. For all w, w' E [0. 1]. ((wx) ((w')1 2 ww w/). Additionally, for < and w, w' [n, n/. ((w) (ww') # w w'l
Proof of Lemma D.14 First, for all w E [0, 1]. lemma follows. For the second statement of the lemma, observe that for w E [n, 1 7].
d (m)
1, and the first statement of the
du
w(i-w²j
(
1 n11 - 1))3) 111 - 2n + +)))
dw
w(1 - w
17(21) 17²)
1
(D.103)
where the last equality is because y) > 1f2 (so 2nj 1p > 1)) Thus, + is + -Lpiscccitz on the interval
[n, 1 -- n/.
We next need to show that the potential initially has large value, at the beginning Phase 3, Case 1. We show this in the next two lemmas, by showing that for any two particles w, w', the distance between them grows by a large factor during Phase 2 (specifically, by time Tg) and then using this to obtain a lower bound on (ww) 4(w") by the bi- bi-sipschitzness of tb
Lemma D.15. In the setting of Theorem 3.3
let TR be as defined in Lemma D.7 Then, for u d' e
[LL s)).
WTh (c)) WT(())
Z EnVd
(D.104)
Proof of Lemma D.I5 Let i, E [uL, with c'. For convenience, we will write w, (i(1) and w/} wz(('). Our goal is to obtain a lower bound on the factor by which wi - we grows by time TR. First, define Pt and Qt as Lemma 4.4 and observe that
d - (w/ [wr)-w))=-- wi) dt
-1 - (w:)) + Qr(w()) (1- (PI + Q((w:)
(D.105)
--(P((w))) -P((()))PP(())) + ((wf)PP((ud)) wfP((u:)) (Q((w())(((uz))
(D.106)
+ Q(w)))Q((wt)) (((()(((
(D.107)
We first obtain a lower bound on the first term - -(P.(w))) - Pr(wz)). and then show that the other terms are lower-order terms as long as t I TR. For t E [0,
-(P((w)) - P.(w))) - (203 D2, 467, De. Da,((()))) + (203 + 402, Da. Da.
(Lemma 4.4
-2203, D2, w/) (o. w
(D.108)
-20%, 12,1 - wa) -402 (2.I(((u))) - -w²)
(B.c. 5 TR. D2,1, D4.t by Lemma D.8
-262,1 D2, we) -402 DoelllD www, + wi wx)
(D.109)
2 202 (2, w.)
(B.c. omitted term nonnegative - w/ > we by Proposition D.37
Next, let us show that the remaining terms in Eq. (D107) are lower-order terms which will not decrease the growth rate of w/ wt too much. First let us deal with the absolute value of the second
36

---

term in Eq. D.107). For convenience, let WR, - wx(4)) Then,
(w))PP((w{) - wf w/PP(wu)) x (263 (2.2 + a (2,((())))
(D.110)
-28 D2,1 + a (wx)()))))
(D.111)
S 26 D2 (2x))( w:²1 + 487.al Da,=l((u{{ -w
(By Lemma
2 202 D2,1 (uue) + w/us - willur - wel
(D.112)
40 Da.1 (ull)(w()² (w() + (wf)² + w/w/
w/ || we
(D.113)
2 22. wel
(B.c. |we|, luyf wR, by Proposition D.37
where the last inequality is also because 07,0 E 62,0 and 74 S 7 (Assumption 3.2 and D2 > 7 and [D4] & 374 for S TR (Lemma D.8), meaning that the first term in Eq. (D.113) dominates up to 2 universal constant factors. Next, let deal with the absolute value of the third term in Eq. (D.107). By Lemma [.16
D2.alDea + +02,al D4.el
(Q((w))) Qr(w))) 2
|ac wel
(By definition of A (1) Ad (8), Lemma 4.4
0 D2.l
Wu wel
d
(B.c. 0 S 2,d and D4
S
S 72 S D22 for t < Tk (Lemma D.8
Using the same argument with Lemma I.16 we can bound the fourth term in Eq. D.107
((w))Q(((()) - wf(r(w))) 22.cl
uu wel
(D.114)
Combining the bounds obtained for all the terms of Eq. D.107). obtain
d di (w/. wx) 2 26 22, we) 0 (2 wh. + 1/d) (ury wi)
(D.115)
a D2,el - 0 D2 + +1/11) (wk wx)
(D.116)
and therefore,
d (wr wt) de
D2, > 20.. - Da. + 1/d)
(D.117)
w/ Wt
First, let consider how much w/ wt grows during Phase 1. At this point, apply Lemma D.4
with 12 = tu and < 4 < lu with Toy Let us first verify that the assumptions of Lemma D.4
are satisfied. By Lemma D.5 F2 x
flogdd d using the notation of Lemma D.4
Thus, log F3 < log d.
meaning 8 < log FT By Assumption
3.2 8
S
25
I and by Proposition 1.5 P(U > u) S log < 22
TO
clearly holds. Thus, the assumptions of Lemma D.4 are satisfied. As consequence of Lemma D.4 (specifically Eq. (D.47) we obtain
exp (f("
Vd
26 D2, Z F2 (log d)²
(D.118)
37

---

For I 7i, we have / I 8 1 0 (ogg)) by Definition 4.5
(and because wx((R) S we((v) by
Proposition D.37). Thus,
//, wri 2 exp
(So Vd
20 P2,1l (1- 0
log )
1
(By Eq. (D117))
w/b wo
1- 1-0(1/ l/gdd
(By Eq. D.118)
(log d)2
Vd
(B.c. d log v 0
(log d)2
Additionally, we apply Lemma D.4 with i2 = iR (and an arbitrary 41 E [1.. UR)) with 8 = & =
2og log and to Ti and t being the time TR such that ur, (1R) ç. We verify that the
conditions of Lemma D.4 hold. Using the notation of Lemma D.4 F2 (1R))R)
(un) Ex(log d)²
by Lemma D.7 and thus log F2 S 3. Again by Assumption 3.2 8
<
2 and by Proposition D.6
To
P(I4 > 1R) s O(k)= x
2á 16"
Thus, we apply Lemma D.4 to obtain
exp J/
26 26 22,
2 F2 2 Ex(log
(D.119)
For S TR. w7, S 5² S 8², meaning that
20 202.a1Ded (1- (1-00)))) w14² WIx 2 exp (f/²
(By Eq. (D.117
w/y WT,
2 (En(log ()2)2-0(62)
(By Eq. D.119
2 En(log d)²
(B.c. (log d) d) (log d)
Ol lon log
< 00()
Vd In summary, since w/ grows by (log dya (up to constant factor) from time O to time Ti, and {x(logd) d)² (up to a constant factor) from time Ti to time TR. this means
by
TT wik
2 ExVd
(D.120)
wi
from time 0 to time TR, desired.
We now use the previous lemma to obtain a lower bound on (+(w) - ((w')l by using the bi- Lipschitzness of C
Lemma D.16 (Initial Large Distance Between w and w'), Suppose we are in the setting of Theo-
rem 3.3 Let TR be as the statement of Lemma D.7 if are sampled independently from Pa, then
with probability at least 1 O(c),
| (t)) (ww,, (c)) EN3
(D.121)
Proof of Lemma D.16 By Eqs. (1.16) and (1.18) of Atkinson and Han 12 the probability density 1(d/2) (i 1)/25 (1 2 _ Thus, for i. E [R, the conditional density
function of i. is p(()
function plu (R), can be upper bounded as
p(()
p(c 1 E [R), S p(c E [LL, 4R])
(D.122)
r(d/2)
(1 (2) 23
(D.123)
T - -1)/2)
p(c E [x))
T(d/2)
(1 (1-2) 23
(By Proposition D.6
VTI((d- - (
- (k))
$ Vd
(By Stirling's 's Formula)
38

---

x²2 In particular, for any interval / of length at most
V
(R) P(LETILE (R)
vd. z n²²
(D.124)
and from this it follows that
16² Pip'opo (1c (1-6l U
1, E [4,, S x²
Sx²
(D.125)
vd
Next, suppose d' E [LL, 4x] with d' > 4, and
2
x² Va
Then, by Lemma D.15 if is the time
T' mentioned in the statement of Lemma D.7 then
WT, (c') WT, (1) 2 En
(D.126)
Suppose we sample i, 6' independently from pu- By Proposition D.6 with probability at least 1 (x). , E [LL, uR]. Thus, by Eq. D.125), the probability that e, E [&L, 4R] and |c | 1 - (x).. In summary, we sample i, independently from Po. then with probability ((s),
d'l 2 / is at least
((wn, (4) (wT, (7)(((()) 2 Ex
(D.127)
by Lemma D.14 as desired.
Next, we show that during Phase 3, Case 1, at any time, there significant fraction of particles whose distance from 0 or can be bounded below.
Lemma D.17. In the setting of Theorem 3.3 suppose DA,t S 0 and D2, 2 0. Then,
Pampi (lol
h//4
- 72 N 72
1/2
(D.128)
Proof Lemma D.17 For convenience, we define -12 and 1.1 such that 744 -8 (Here, 3 and are well-defined by Assumption 3.2 First, since D2,t 2 during Phase 3 Case 1, we know that
E [P) aPe--P
2 ?
(By Definition of D2,1)
and since Dr, S 0 during Phase 3 Case 1, we know that
E wa
[PLa s Br
(By Definition of Da.))
Rearranging using (2,a()) = d-1
12
d-1 we obtain
d
W
2
7
(D.129)
d
or
d 1
1
E
2
(D.130)
Werppt
Similarly, rearranging using Pt,a(t)
d' 2'46448,4
6d412,2 +
J- obtain
3
d=11
d²
+ 6d
6d + 12
3
E 100 +
S BT
(D.131)
d²
waripe
d2
We ~ip
d2
and rearranging gives
E
[wt] BT
5
0
(D.132)
39

---

because w I 1 and we can assume 3T < 1 due to Assumption 3.2
Suppose that with probability
more than 1 r' under Pt. w & [73/2 1 r]. Then,
E [w"] Pu pr (w 2 1 T) whopt
E Iw² w 2 11 wr
(D.133)
+ P wrpps (w [73/2 1 r])
E Iw²
[T73/2,
-]
(D.134)
wmppt
+ P wrpp (w S .
E
mwe w
73/2]
(D.135)
wmp
72
x P Piiple (w 2 -7 + +
2
(By assumption that P. Pwrpp (w E [73/2,1 11-//,1-111
- -]) 1 T
37²
$ P Pumpi (w 12-11111
4
(B.c. S 1/4 (by Assumption 3.2, T is sufficiently small))
which by Eq. (D130) implies that
Puopp (w 2 1 - r) 2
[w=)
3r"
2
-
2
+
37
22
7
+
2
72
(D.136)
wr-p
4
d
d
4
4
d
d
4
where the last inequality is because T S 1. On the other hand,
E [w*] = Iº P'uppe (w 2 1 - T) E Iw²
w 2 1 T]
T]
(D.137)
wrupe
wrppe
+ P pi (w E [73/2 11,111 rT)
E Iw w E E -]
(D.138)
+ P wrpp (w S 23/2).
E
mw 4
w S 73/21
(D.139)
wmppe
2 Pumpe (w 2 - T)
E -
w M 1 - r|
(D.140)
wmppt
2 (1 r),. PPwpp (w 2 1 --7) 7)
(D.141)
72
2 (1 - 7)
(By Eq.
(D.166)
4
72
2 64
(B.c. T S 1/2 (see Assumption 3.2
By Eq. (D.132). this a contradiction, since implies that
22 S BT + 0 ( S 2
(D.142)
64
where the second inequality is because 3 > 1.1 and d is sufficiently large (see Assumption 3.2 This implies that 128,72 2 1, which contradicts Assumption 3.2 (since 72 is chosen to be sufficiently
=² small). Thus, original assumption that P Puwp. (w 3/2 r]) S 2 is incorrect.
The purpose of the next three lemmas is show that if Phase 3, Case occurs, i.e. if D2,7, and DA.T, < O, then for all &. > T2, we have D2, 2 0 and Da.t S 0. First, the following lemma states that if Da.t is much smaller than D2,: in absolute value, and Da,; < 0 and D2, 0, then the velocity of the particles w 2 0 will be significantly negative.
3.3 and suppose DA.T, = 0 and t t2 T2 such that Da,t < O and D2, 2 0. D4,el S D2,1 where { log lug then for all w E [0. 1).
Lemma D.18. Suppose we are in the setting of Theorem
vu(w)= = -(1 w²) (1 + (())) 22,1u1
(D.143)
In particular, for w E 1/2 ----711
3/4 - 7/2 2.2
3/4 3/4_2 v((w) S -72 2,d D2.c
(D.144)
40

---

d Proof. Suppose D1,1l - 22,, where &= log Let Pe and Qe be as in Lemma 4.4 Then, for any [0,1],
((()) (202 22 + a a (,
(D.145)
(20 D2, 40 Da,e (u+))
(D.146)
D2,1|u P2,
O ((,
(D.147)
(1)) 20 D2
(By Assumption 3.2
0 S (2
Thus, for w E [0, 1].
v((w) = (1 - (P(( + Qc(w))
(By Lemma 4.4
D2,el
-- - w2) (P. 0
2,a
w
d
(B.c. [D2 2 D4,rl and 02,0 S 62 by Assumption 3.2
02. D2 w)
-- w²) ( (1
+ O
(By Eq. D.145)
-1 - w (1 O
(B.c. E 1
particular, for l72 7/2 1/2
3/4
va(w) 2 -(1 - (1 22 22))))) (1 O(E)) 202.a 20, D2.112 3/4
(D.148)
S - (1 (1 722 11 -020 D2.122 3/4
(B.c. { = log log
< - (2%/2 1/2 -22 22. 3/4
(D.149)
S 12
5/4 02, D2.1l
1/2 (B.c. 122
2 72 since T2 S 1)
as desired.
The next lemma essentially implies that at some point D4,: 0 and D2, > 0, then will never be the case that D4., > 0 and D2,, > O.
Lemma D.19. Suppose we are in the setting of Theorem 3.3 and suppose Da.t 0 and D2,1 2 0 for some t 2 0. Then, de D4.8 S 0.
Proof. We calculate that
d Da. = dt wrips
d [Paa(w)]]
[Pad'' (w) v((w]]
(D.150)
dt
Recall from Eq. C.4 that
d + 6d +8
6d +
Pta(t)
42 +
(D.151)
d -1
d2
d3
meaning that
Ptd'(t) = 4t3 =0
=0(2) =
+1)) 4t
(D.152)
41

---

Observe that for t 1 particular,
v0. (t) > 0 and for t Va
1. it may be the case that Pt.'(t) < 0. In
(Pdd vw(w)] = P([w) S 11/Va)
E
[aad v((w) | | E 1/vd
(D.153)
woept
wmpt
+ P([w] Z 1/vd)
E wmmpt
vx(w) |w| ž 1/vd
(D.154)
02 D2,el 2 50(
+ 1P[w/o] 2 1/Vo) E (Pa,'(w) vx(w) |w| 2 1/ 1/vd
d2
(B.c. |w|| 2 t and by Lemma D.18
02
D2
<0( s
+ P([[co] E h22 3/4
7/2 1//21)
(D.155)
d²
Pa.d (w)
ve(w) |w| E 1:2
- 7/2 1/21
(D.156)
wrppe
0/ 12
<
<
d2
P([w] E h2
3/4 - 7/2 12
9/4 5/4
12
2 D2.1
(B.c. Pa.d (w) Z 12 and by Lemma D.18
D2,1
<
0
2,d
d2
22
2
12
14/4
2,d
2
D2,1
(By Lemma D.17
<0
(By Assumption 3.2 since is sufficiently large)
meaning that 0x
Da.t < O, as desired.
The next lemma essentially implies that if at some point D4,6 & 0 and D2, >> O, will never be the case that DA. < 0 and D2.; 0.
Lemma D.20. Suppose we are in the setting of Theorem 3.3 and suppose Da. S 0 and D2, = 0,
for some t 2 0. Then, d D2.3 2
de
Proof. Recall from Eq.
C.4 that P2.((t)
"d1 dd1 meaning that P2.d(tt
2d d-1 t. Thus,
d
d E [w]]]
D2,t
E wm
[2w uy (w))
(D.157)
dt
di
wmp
If Da 0 and D2 0, then for any particle E [0, 1]. we write
vy(w)= (11 w) (P:(w) +Q:
(By Lemma
02. Da |101
-(1 w² (4 D +
(By Lemma 4.4
(1 w²) 226 Du + o 22 L ( [20]
(D.158)
42

---

where the last equality is because D1. S 0. Observe that vx(w) 2 0 for w 2 va the case that v (w) & 0 for w
1
while it may be
S Va Thus,
12w ur vr(w]] P([w x 1/Vd)
E 12w ve(w)
|w| S 1/Vd
(D.159)
wripe
+ P([wc] 2 1/vd)
E
12w0 vr(w) |w| S 1/vd
(D.160)
wmpp
02 D4icl + 1([1 2 1/Va) 2 -00
12t0 m(ww) [w| 2 1/vd
d2
wrpps
(By bounding Eq. D.158 when w
2 v7
02 Da,l)
2 -0
+ P(ww E h/41
7/2 111
(D.161)
d2
E wmpps
12w ui (w) |uo| E [12
7/2 /
(D.162)
-0 52 Da.rl
+ 1[[w]] E 3/4 ,1 - 7/2 1/211
72 0 Daal
2/4
dc
(By Eq.
(D158)
0 4,d
Da,cl
2V
72
-22
9/4
03,a Da.1
(By Lemma D.17
d2
2
0
(By Assumption 3.2
since d is sufficiently large)
as desired.
Putting all of the previous lemmas together, we obtain the following invariant: lower bound for (D)w) - ((w'),, for a large fraction of w, w', which holds throughout Phase 3, Case 1. Note that in the proof of this invariant, we need to make
of the fact that once enter the case where D2,, > and Da.t < O, we cannot leave this case. If this is not true, then we cannot use the fact that function of f (Lemma (.13) since if Da.: at any point, then (ww)) 2 ( could decrease, and we would not have control over this decrease.
[f(w)) o0 (w() is increasing
Lemma D.21 (Phase 3, Case Invariant). Suppose we are in the setting of Thhorrm 3.3, Let TR be as defined in the statement of Lemma D.7 and assume that D2,T2 = =0 and Da,7, < 0. Then, for all t 2 TR (in particular, for all times t during Phase 3 Case I), if L, are sampled independently from Po.
(())(()) ((wx(())) E
(D.163)
with probability at least 1 - O(K).
Proof of Lemma D.27 By Lemma D.19 and Lemma D.20 if D2.T. 0 and D4,7, 0, then for all L. 2 T2, we will have D2,, 2 0 and Da., S 0. Thus, the lemma follows from Lemma DD13 and Lemma 1.16 as well as the fact that D < for E [TR. To] (meaning that the potential difference is increasing in the time interval [TR, T2)).
Finally, we show Lemma D.22 which gives a running time bound for Phase 3, Case 1.
Lemma D.22 (Phase 3, Case 1 Summary). In the setting of Theorem 3.3 suppose that D2.T2 0 and D4 < (i.e. Phase Case holds). Then, the total amount of time 1. > To such that
(ppc) 2 2 (, + 6 is at most O 2 log( 22 ))-
Proof of Lemma D.22 Suppose 2 T2 and assume that D2,12 = 0 and Da., < 0 (i.e. Phase 3,
Case occurs). First, suppose |Dael I 22,cl where & Tog 3/4
1
Then, by Lemma D.18
|w| E [12 1 7/2 1/2 we have
[v([w]] 2 5/4 =2
02 D2el
(D.164)
43

---

(note that the lemma is stated for w E [72 3/4 2e (w(w) is an odd function). Therefore,
7/2 "2 but holds for |w| E 3/4 - 72 1/4 since
[wr(2=) 2 Pwopp (lw) E [72
wmpp. [lve(w)) 2 E [1)(1 |wl [72 3/1 1 /22²
3/4 ,1 7/2 1//
|w| E [72 , 7/ 2 (D.165)
3/4
wmups
(By Lemma D.17
2 wmpp
2 12
72
3/2
2 D2,1
(By Eq. D.164)
2 12
7/2
22 D2,1| 2
(D.166)
2 12 2,d
7/2 .2
(62
(4,1) (B.c. of. S 02 (Assumption 3.2) and [Da,1 S D2,,))
2 12
7/2 (pp)
(D.167)
Thus, by Lemma D.39 for any time t during Phase 3, Case 1 such that Daal S E2)l,, we have
dT(pp))
S 7/2 (2,
(D.168)
dt
Let T..c be in Theorem 3.3 and define
I (t s T., and in Phase Case
|D,,1 S D2,1)
(D.169)
Then, by Eq. (.168).
dL(pp)) < -
(D.170)
dt
Thus, by Gronwall's inequality (Fact 1.13).
IPr...) S exp (-C [(po)
1 Toic
7/2 dlt E [1o
(D.171)
T2
=exp
7/2 2,d C
1dl
(D.172)
Since L(po) $ (62.a + 222))2 by Lemma D.8 and L(PT..)) > (02 + + og,))² rearranging gives
J
1
122
1dl S
log
(D.173)
0
7/2
2.d
Now, suppose is such that D4,el D2.l Let us lower bound the average velocity for h/ 3/4
72 1/4 first (then, we can conclude the average velocity large by Lemma D.17) By
Lemma
4.4
[vr(w)] = - + Q.(w)
(D.174)
and write
P((w) 20 + (4, 40 D4. (w
(D.175)
where
83 Dait 24 Da.
is the positive root of P,. For |u||
[/2 1 - 1/2
3/1
we have
1 1 2 > 1 (1 1/3/2 = 27/22
1/2
72 2 7/
1/2
(D.176)
44

---

3/4 and w + 2 7/ Additionally, |uc| 2 7/2
3/4 Thus, for w E l22 3/4 ,1 - 2/22, we have
-1-22
|er(w))?
2 (1 - ) (((w)) + Qr(w))
(D.177)
2 72 (P((w)) 1c(1r))² 2
(B.c. 1 - w² 7/2 //3
2 72 (P((w))-2c(w)3))) (Q)(w)))
(B.c. (a+bb2 s 2a² 262, so a² 26 - 2(a 6)3)
222 (Pi(W)² - 0( 02 D2 + 1.
D2
2
(By Lemma 4.4
d²
2 12 (Pi(w)² - 0 (02, + 07,0) (pp))
w=2
(By Lemma
d²
(02 + (
2 72 (i 2² (w + r) w - -0
w²
dc
(By factorization of P:(w) above)
(w 0 (02 ( + 07,0)
212
w
(D.178)
d2
where the last inequality is by the discussion above Eq. (D177). Now, in order to take the expectation over P: (conditioned on E 3/4
7/2 D. we first compute Eun - r)²
1/2
h2 7/2 []:
3/4
3/4 E [(w r)2 w h2 wript
1 7/2
11
Var (w
3/4 , - 722 w h22
(D.179)
wmppe
E w.w
[[ w') w. w E 3/34
7/2 - (D.180)
pp
where the last inequality is by Proposition D.40 By Lemma D.21
Pu. (10(w) ( Z Ex3))
((s)
(D.181)
By Lemma D.14 if w. w' (172 1 - 7/2
3/4
1/2 then
( (w)) D(w')l S
w w'1
(D.182)
3/2 72
Thus, combining the previous two equations gives
Pwr.irop (ww- - w'l
3/2 7/2
Ex3 Exx
w, w' E 1:22
3/4 1 7/2
1/2)
(D.183)
S P Pup (10(w) - ( )1 < Ex²
w. w E 1h2 ,1 7/ 1/21)
3/4
(D.184)
PP -pp (10(w) D(w')) < ENS and w, w' E 172
3/4 . 7/ 1)
V2
IP wn (w, w/ 3/4 , 7/2 1/21)
(D.185)
2A
16
(By Lemma D.21
IP up (w, w/ E 172
3/4
-
1/2
24
(By Lemma D.17
and therefore, by Assumption large. Thus,
3.2 since n1/12 S t2 log log di this is at most 2, since d is sufficiently
E ~pt
[[w w²)2 lw. w' E =12
3/4 , 1-71/ 1/2
(D.186)
2 Puw (ww w/1 2 72
3/2 Ex3 w, w E E////4
3/4
722 1/21)
2222 R
(D.187)
2 2222
(D.188)
45

---

where the last inequality is because IP wuw - w/1 By Eq. (D.77), we have
3/2 En. w. w,,' w/ E E
3/4 , -1-22 -1-22/1) S
E [[v(w))| [12" 3/4 11-72"
-1 22 (
(
E
[w - r)2 E 3/4
7/2 [2/21
wmpp
(DD.199)
(02 (.pe))
-0
(D.190)
d2
72522 no 0 (02,0
02 02,0)
Z 72 ( D2, D2,
d
(By Eq. (D.80) and Eq.
D. D.188)
Since PP Pm-pp (w
3/4
, 722 --/ 7 by Lemma D.17 and the symmetry of pr, this implies that
(,1, 125 222 -0 (4 + 0 o(,e))
(r(w))?
2 7/2
(D.191)
wmppt
d
6 L(p (p -0 (o2,d 62 (
2 12
d 3.2 and |Da,1l 2 E)D2,a implies 0² 1, D2, 2 (2)())
(B.c. 09 2 2. (Assumption
o 2g ( + 0
2 12 (62,n
((p))
(D.192)
d
2 -/5 k L(pi)
(Because 2 cs by Assumption 3.2
Thus, by Lemma D.39 where Da.e s { Dz, since (ppo) S 02 22 2 (by Lemma D.8,, if we define
dp(p))
< - 1.a 72 2 L(p.) By a similar argument as for the subcase
8
dt
1p.122/121 = (1 < To.c and in Phase 3, Case |D4,al 2 D2a1)
(D.193)
then
72
1dl S
log
(D.194)
4.d 54
since MPT....
) Z o. Combining Eq.
D.173) and Eq. D.194.. we find that the total time spent 4.a a log 13 )). as desired.
Phase 3, Case is at most O
D.6 Phase 3, Case
Next, we analyze Phase 3, Case 2, where D4.T, = 0 and D2.T, < 0. In this case, we will show, for all / 2 T2, we will have D4 2 and D2.t S 0 (Lemma D.28 We will first give an outline of our analysis in this case our goal is again to show that either D2.ll S or Da,al S within t poly( loglo time after the start of Phase 3, Case 2.
Proof Strategy for Phase 3, Case We again show that [e(w) is large on average and then use Lemma D.39 to show that the loss decreases quickly. However, argument show that (o(w) is large on average significantly different from Phase 3, Case 1. We can approximately write
01 (w) - w (P,(w) + Q((w))
(D.195)
2 - (1 w2) (w)
(D.196)
22) 46
(D.197)
2 (1 2 u D2.1 46,.ul D4,1 2
(B.c. D2, S and D4,- 2 0)
~ - - w w Dr.e/ w22 - 28. D2.el)
(D.198)
83 Da 20 [
Letting
be the positive root of P. Pr(w), we obtain
vr(w) ~ -40 D4.e ( (w + r) w²)
(D.199)
46

---

This time, (w(w has negative slope at r, meaning that the particles are attracted towards r, and we cannot use the same argument as in Phase 3, Case 1.
Instead, the key point of our argument in this case is the following. Recall that Da.t 2 0 and D2. I 0 for all Z T2 - these roughly imply that Ew-p, [wr]] 2 744 and E' Euwo, [w I 72- By Assumption 3.2 we have 7/ 2 1.172 - in other words, Eu [w^] 2 1.1(E (w [w2 From this
observation, we know that a significant fraction of the particles must be far from r otherwise, if all of the particles are concentrated at r, then we would have Em--, [w0"] Rs (Bropp [1ee [w2)), which would give contradiction.
One complication in showing that the velocity large is that, a priori, nearly all of the particles could be close to 0 or 1 - their velocities could be small even if they are far from r. We make use of the potential function & defined in the analysis of Phase 3, Case to avoid this issue. In this case, turns out that for any two particles w, w', their potential difference (f(w) Φ [w'l is decreasing (Lemma D.23 We can use this to show that a 1- O(74) fraction of particles w with initializations in (4., (M] (where (M is defined later) cannot become too close to 0 or in Lemma D.24Jand Lemma D.25 Intuitively, this is because if (ww) ((w')) is bounded by log for two particles w. w' > 0, and C is a sufficiently large constant, then w S ( Jog l roughly implies that w' X B log lug a ) simply by the definition (ww) log( - ) together with some algebraic manipulations. Moreover, if
all of the particles w' in this group are at most B (ggg) then this implies that D2 has very large negative value, and this implies that all particles have a positive velocity (as argued in Lemma D.25), meaning that the particles w in this group cannot become smaller than ( log log ) We can show by similar argument that the particles having initializations in [41 'M] cannot become too close to
As long as the positive root r of P.(w) is at least 1. this group of particles will have a large velocity as argued in Lemma D.29 since a significant fraction of these particles must be at most 3- We encounter potential issue when the positive root is at most 1: it is possible that the particles initialized in [LL, 4M] get stuck at r. We mentioned above that by the constraints D2, < and D1. > O, there must always be some mass away from However, this mass may only be a 2 5/4 fraction of the total mass, while the particles initialized at [4. 'M] only have a O(7a) fraction of the total mass. Thus, once I 3. it is possible for the mass located far away from be disjoint from the set of particles initialized at [4., /M]- To resolve this, we argue in Lemma D.31 that the particles larger than we (I(( (which may be close 1) will move away from 1 at an exponentially fast rate while r S 2 Once these particles, which are larger than we (MM)) are sufficiently far away from 1, they will also contribute significant amount to the velocity. Thus, we can argue as discussed above that there must be a significant fraction of particles away from r due to the constraints D2.; < 0 and Da. 2 0 (Lemma D.32).
We now begin the formal proof. We first show that as long as Da. 2 O. the potential difference between any two particles w, w/ is increasing. In particular, this holds during Phase 3, Case (but note that this lemma is no longer applicable during Phase 1 or Phase 2).
Lemma D.23 (I(w) and (ww') Move Closer in Phase 3 Case 2). Suppose we are in the setting of Theorem 3.3 and suppose Da,t 2 0 for all t in some time interval (to, t.). Then, for all t, L',
v(wx())) - o (ww((())
(D.200)
is decreasing as a function of on the interval (Lo, 41]-
Proof of Lemma D.23
We mostly follow the proof of Lemma D.13 Suppose i, E [0, 1] such that
1. S '. Then, (t') 2 wx(1) for all times (by Proposition D.37). , and since + is an increasing function, ((wr(1)) 2 (w)( for all times f. Thus, it suffices to show that (wr(())) - (t))
is decreasing, For convenience, we write w/ w. (4') and w, (t). We do computation similar
47

---

to that in the proof of Lemma D.13
d (Q(w)) ((w))) = -2 - -0 - (u,(1)) - 2a Ad 2x (w/)²
(D.201)
di
- (-
2 (-2263,
- D iu, - >aa - da (a !
(By Lemma D.12
= = -1,0i,DD, - uff - >a Ad (ww))² wi
(D.202)
According to the statement of Lemma
4.4 Ad (3) 40 Ds.
6d49 2=-1 meaning that
āt (8(w)) ((w+)) - D2,e (111)2 dt
d
mnt) 46 D4,1 d² - 1 ((w))2 w
6d + 9
(D.203)
6d 9
= - Dr, (1- d² -1) ((w)) - w
(D.204)
<0 (B.c. Da,1= 2 0 and w/ > w/ > 0. and d 2 C3 (Assumption 3.2
Thus, on any interval [to: f1] where Da. 2 0, (ww() - (wx) is decreasing, as desired.
In the next lemma, show an upper bound (w)) ((w')) ), the beginning of Phase 3, Case 2, for particles w, w' which are respectively initialized at c, E [LL, tm]. Here (M is defined in the statement/proof of the following lemma.
Lemma D.24. Suppose are in the setting of Theorem 3.3 Assume that Da,ry = 0 and D2.T, < 0. Le, we are in Phase 3, Case 2. Then, there exists IM E [1, (R) such that Pimpo (lo) IM) S 74 and
for all u,d' E [LL,
D(wr,(1)) F(urs; s 2oo
+ log +
(D.205)
for some universal constant C.
Proof of Lemma D.24
Since D4,T2 = O, we have Ew-pry [a,a(w)] = 74. Thus, by Eq. (C.4
d² 6d
+8
6d + 12
3
[wi]
[w
+
7/
(D.206)
d2
woupT2
d2
wrupr,
d2
Since (1) < 1 for all Z and all e E |-1, 1], the left-hand side of the above equation is
Ewwn, [w* + Ot)), and rearranging gives
|w S 74 70 wmpp
E
(D.207)
By Markov's inequality.
P Pwwppry (wwl 2 1/2) S 74 =02) S
(D.208)
where the second inequality is because d 2 C3 by Assumption 3.2 Let to be such that WT, (4n)
2'
Let (m = min(1o, (R). Then, AM E (R). and IP, Pe--po( (11) > (M) 74, because Pe (l > (0) < 74
and IP PP (1x) > (R) S (R)) by Proposition] D.6 meaning that Pampoo (lrl > (M) S max k) S 74
(where the last inequality is because d 2 c3 by Assumption 3.2
Now, let & E [LL 1M]. Observe that on the interval T2). for all 4, w.(() is strictly increasing as
a function of f, for all (see Lemma D.9.. Thus, if IR is as defined in Lemma D.7 then for all
c [4, RR]. |WTT (4) 2 Ex² In particular, for all E [st, (M], we can write WTy (t) C for some
universal constant C. Additionally, for S (M. we have Wrz (c) < by Proposition D.37 Thus,
since (w) is increasing in w, for E [LL MM] we have
CEK²
@ (w),())) 2 ((CER?) = log
2 (CEx)) = log C + log + log
II- O2
(D.209)
48

---

and
1/2
1
D(wrs (t)) - (11/2) log
- 1og(1/v3) -- log (D.210)
/3/4
Thus, for any 1, E [12, uM]
((wT((())) - logg (log C' log { + +lllgg)) 2 log
1
log 1
C' (D.211)
where we have defined the constant C² --1 log - log C.
In the next lemma, we show that for E [L,, (m], during Phase 3, Case 2, w)(c) can be bounded away from O and provided that the conclusion of Lemma D.24 holds at time
Lemma D.25 (I Loower and Upper Bounds on Particles Between 4. and (m). Suppose that we are in the setting of Theorem 3.3 Let " T2, and suppose that for all & (T2, s). Da.t 2 0 and D22 Further suppose that for all [T2, 8), the conclusion of Lemmal D.24 holds, i.e.
1 + log k
1 +C E
((w(()) (ws((())) 2log
(D.212)
for all E [4L.MM] where im is such that Pawpo (14) um) Ya. and for some universal constant C. Then, for any 1 [T2, for all 2 u, wi(1) g²x². Furthermore, for all l E [72.8], we have we (IM) S k4g3.
Proof of Lemma D.25 RR Also, suppose e'
Consider some time t E [T2, s). Suppose that for some uM), (1) S [Lz, mM] such that > L. Then,
((w,(())) s 4 (w,(()) 2 2og
log +C
(By Eq.
(D.212) and b.c. + is increasing, /' > i and Proposition D.37
22
1
< log vI cdp
+ 2oo
+ log +C
(D.213)
1
1
log(2 (2222 + 2 log
+ log C
(B.c. d2 c3, Assumption 3.2
K
logg C'
(For some universal constant C*)
Thus, because < v
wr((') (wi (1)))²
log w((( S log
S log O'
(D.214)
Thus, wi((') S & and for all MM]. we have shown that wx(() S &. In addition, P Pi-po (14) (M) S 74. We can use this to calculate D2 and Da.t
d
D2,1
2
72
(By definition of P2.s (Eq. (C.4))
wmpt
< E. [ 7/2 wmpt
(B.c. $ 1)
< P Pirpol (11 5 (m) 2(ux(()2 || 1 ml + IP Pi-poo (14 > (M) (() ||4] 2 (M] - 72 (D.215)
< O(E2)
+ 74 72
(B.c. wx(() S E for E [M]
AM] and P([|) uM) S 74)
2 -72
(B.c. < c172 72 < C2 (c2 chosen after c₁) and d 2 C3. Assumption
3.2
49

---

and
d + 6d +8
6d + 12
3
Da.
E
w
w²
+
74
d2 -1
d2 d
d2 - 1.
(By definition of Pa.d. Eq. C.44
E [w-]] 74 + ( wap.
(D.216)
Parpo (14) vn) (L) 114 ml 74 + 0
=Pi um) Em)) (1) |d| S mm]
(D.217)
(0(55)+0(1))) + - 70 (
(B.c. wi(l) S & for E [i,, und and P(e > 1M) S 74)
$ 50(5 +7a) Sn
(D.218)
(B.c. d 2 c3. by Assumption 3.2
To complete the proof. we will show that if these bounds on D2, and Da hold, then for any w, ue (we)) 2 By Lemma 4.4 it suffices show that Pr(w) (v))) I 0 for 0, where P. and Q+ are as defined in Lemma 4.4 If C1 and C2 are two appropriately chosen positive universal constants, then by the above upper bounds on D2, and Da,t.
Pr(w) + Qw(w) D2 + +4 + ( Ad w
(D.219)
2, C2d now + +0 2 D2,e + 57 (idDDall)
(D.220)
d
(3 + +0 (
3
d
(By definition of Xa (1) Ad (3) in Lemma 4.4 and above bounds on D2,1, Da.))
02.00 D2,1 02.a Da.
C,002 +
|uo|
(B.c. 74 S 72 S 72 and 02,a S 22 by Assumption 3.2 with new constant C₁
x - 02 72w
(B.c. d 2 ca. Assumption 3.2
In summary, wi(() n² for some E [r,, (M). then vy(w)) for w E [0, 1]. addition, at by Lemma D.9 and Lemma D.7 Thus, for all 2 LL.
time T2. we have wx(1) 2 ex² for all
(e(() will never go below x²x² for t E [T2, s]. as desired.
To finish the proof, we will show the upper bound on (r()). Suppose 1 - -w(((x) S kgg Then,
(2((M)
(we(M)) = log
(D.221)
We (M
1
2 log
log(1 - (1((
(B.c. wy(()) 2 }
2
2
1
log
log(1 (1 - x²(3)?)
(D.222)
2
1
= log
log 1og(2x153 xE)))
(D.223)
2
1
> log
log(k* )
(B.c. k, S 1 and n&e3 (N:433)2 2 0)
2
2
1
log
+ 2 log
log
(D.224)
2
2
Thus, for any [L)
1
((wx(()) 2 C
log
(D.225)
2
50

---

for some universal constant C. In other words, letting wi := (i((1) and rearranging using the definition of we obtain
W1
1
(D.226)
61/2
w
or
w 55 S 55122
(D.227)
which finally gives - w - e. or wf 2 1 - O In other words, for all 2 ", we have wr(1) 2 O()) which implies that
D21
E [P2a[w]] 72
(D.228)
wmpt"
2 E |w²]] 7/2 0 wrpp
(D.229)
|1| 2 41] - 72 - (
2 w (14 4))
E
[w2
(D.230)
wripe
| 4.) - 12- - 0 (22)
2 (1 O(k))
E 2
(By Proposition D.6
wmpi
2 (1 ((K)) 0()) - -2 -0 (
(B.c. if [t] > L, then ws un(1) 2 1 0())
2 1 - 1 +K) 72
(B.c. 1/d is small by Assumption 3.2
(By Assumption 3.2 b.c. 72 small and large)
and this contradicts the assumption that for all E [22 s), Da, 2 0 and D2 S 0. Thus, for all t B [T2, s), 1 (((M)) 2 x²53, i.. (r(()) 1 x4e3
The purpose of the next two lemmas is to show that Phase 3, Case 2 occurs, then for all / 2 T2, we will have D2., S 0 and Da.: 2 O, assuming the conclusion of Lemma Le25 10.25holds We will later put these next two lemmas with the above lemma, Lemma D.25 in order to inductively show that the hypothesis of Lemma holds for t T2 (note that Lemma D.25 assumes that the upper bound on the potential difference holds on an interval [T2, s). and more importantly assumes that D2,1 S and D4.; 2 0 for all t E s)).
Lemma D.26. Suppose we are in the setting of Theorem 3.3 and suppose DL. = and D2,1 < for some t 0. Additionally, suppose that the conclusion of Lemma D.25 holds time t, i.e. for all 6 2 L. w.(() (1) 2 {²x?, and we (tM)) S 1 x²E3. Then, d D1.e s=t ds
Proof. Recall from Eq. (C.4 that
d + 6d +8
6d + 12
3
Pl.(()
(D.231)
d2
d2
meaning that
Ptd 41 + 0 =
(D.232)
Next we compute the derivative of Dz,r:
d d EE [Pad(w)]] = E [Pa,' (w) ve(w)] wmpp
di D4. di wmm
(D.233)
For positive particles 2 va we have Pa,d' (w) 2 0, and for w S v2 it may be the case that Pa.d' (wd'(w)= I 0. Furthermore, when Da 0 and D2,1 I O, we have
(e()) = -(1 - -1-=3) 220 D2.00 D2 + o -2 D2.1
|101
d
(By Lemma 4.4 and because Da, = 0)
22.a D2,e d
= =(1-w2) (203.) D2,1 D2.
|w0
(B.c. D2, S 0)
= (1 - w²) 20 12 wr (1 (1/dd)
(D.234)
51

---

Finally, to lower bound d. Da.,, we must show that there is significant fraction of particles for which 2 Va and 1 is large. Because we have (r(() 2 g²x² for all & 2 uL and (P( < u.) 2 by
de
Proposition D.6. using a union bound therefore know that
P Pumpe (lwol E [g2 1-=)) 0(74) 0(k) 2 1 0(74)
(D.235)
where the second inequality is by Assumption 3.2 since d is sufficiently large. Thus,
d Da.t= = E Paac' [w)]
(D.236)
dt
wmpp
P Pump (lwb 2 1/Vd) aimpe
E [Pa.d (w) ve(wr) |ll| - 1/vd [Pa. (w) v |w| 2 1/vd
(D.237)
+ I pr (110) 2 1/vd)
E
(D.238)
wripe
2.d 12,1
2 -00
+ PP Puupp (w) ス 1/Vd)
[Pad (w) vw(w) |m|
ス 1/Vd
03/2
d1/2
wmpp
(By bounding |Pa. (w) ve(w)] when |
- 1/VØ)
02- D2,e
-0
2,d
+ I wrppe (lw) E [ 1 - 1-x(0)
(D.239)
dc
E PPa.d (w) ve(w)] |w|| E [E2n² 1 n 501
(D.240)
wmpp
2
2.a
D2.1
(1 (1-1001 (774)
E
[Pad' (w) v2(w) |w| E x522 K x
d
wripe
(By Eq.
(D235
32 D2ic 2 200
+ (52x2)3 (1 -1- - 2 (2.
(D.241)
d²
where the last inequality is by Eq. (D222) and applying the above bound on 6y (w) from Eq. (D.234) for |o E 1s2n²,, 1, 43, and because 2 C3 by Assumption 3.2 Expanding the last line gives
d DusZ - (2.4 d²
+ +
(D.242)
dt
02 D2,el
2 2-0(
+ + "t
(D.243)
d²
and the right hand side is strictly positive since is sufficiently large by Assumption 3.2
Lemma D.27. Suppose we are in the setting of Theorem 3.3 and suppose Da. > 0 and D2, = 0
for some f 2 0. Additionally, suppose that the conclusion of Temmaa DD25 holds time Le. for all 6. 2 4L we (1) 2 g²x,, and w. (1s)) S 1 RAç3, Then, d D2.2 de smt 0.
Proof. Recall from Eq. (C.4) that
d
P2.(()= = d -
1
(D.244)
d 1-1
meaning that
d D2.1 dt dt wmpp
d E [P2.a
2d
E [w (w)
(D.245)
d
mmp
Suppose D2,1= = 0 and D1,5 > 0. Then, the velocity is
(w) -(1 u² )) (P)(w) (w))
(By Lemma 44
= -(1 - w²) (662, Dar + 0 03.0 Da.e lcc]
(By Lemma
d
= ((1 - w2) 46, 1.a Dr. (w²' + |uo|
(D.246)
52

---

Thus, for positive particles w, if 2 positive. Using this, we can upper bound d dt
1 vr(w) S O, and for w I 1 the velocity is potentially
va
va
D2, by upper bounding Eu-p, [w vr(w)]:
E [w- vr(wr]] = P Pumpe wmups
(w]
E wmppe
w ve(w)
|wl
1
(D.247)
+ P woupt
|wo|
E wmppe
w
(2))
|00|
(D.248)
<0(
1
0 D
+ I pumpe
(lu0)
w vr(w)
| N
1
d1/2
d3/2
wripe
vd
(By bounding w v((w) for S w - v2)
67
I 00
+ P
Pemppe
(lw) E (52 -- n2429)
(D.249)
d2
E wpups
|w|
E >s= , - K x
(B.c. for w 2
1 vo' v/(w) < 0.)
0 I <0 <0(00 oi
d²
+ -(T))
-
E wmpps
|w|| E |52 1 - K x459] w vr(w)
(D.250)
where the last inequality is because PP[1|] > IM) < ((4)) (by the definition of im. see statement of Lemma D.25 and [P[||] < (1.) S (()) by Proposition D.6 Simplifying further, we obtain,
0 22,a1Da.cl
E (w- ve vr(ww 00 =0 wrupe
((k) - (E3))) (1 - 11- 11-1(11Ax))) (1=)) DE, (By Eq. (D.246) and b.c. is small (Assumption 3.2)
d2
82 ParlDarl
2
-0 Dr, x2x² n 55 R
(By Assumption 3.2 1 - OKs) O(ra) ž 1)
00 02 Daul
07 D4,5 11
(D.251)
and the last expression is strictly negative, since d c3 by Assumption 3.2 and Da.t 2 0. This proves the lemma.
Now, combine Lemma D.25 Lemma D.26 and Lemma D.27 to inductively show that for all
t 1 T2. we have D2, S 0, Da, 2 0 and most importantly, (.(() can be bounded away from O and for E (M].
Lemma D.28 (Phase 3 Case 2 Invariants). Suppose we are in the setting of Theorem 3.3
Assume
D2,T2 < 0 and D4,72 Lee. Phase 3 Case 2 holds. Then, for all 2 T2, the following will hold:
1. D2. < and DI,; 20
2. Let iM be defined as in Lemma D.25 Then, (1(1) 2 1²2 for all E [1,, uml and we (1xs) S k²
Proof of Lemma D.28
For the purpose of this proof, define
= {s 2 T3 D2.e S and D4. 2 0}
(D.252)
and let tu = sup I, meaning that for E [Ty, tu). D2. S 0 and Da. 2 0. Assume for the sake of contradiction that to co. Then, by the continuity of D2.1 and D1, as functions of t, D2,, S 0 and D4,, 2 0. Since to = sup I, either D2.t. = 0 or D4.., = 0 (and one of these has to be nonzero since otherwise to = oc, which is a contradiction). By Lemma D.24 and Lemma D.23 for all E [T2, 40), for all i, L' E [1,, AM].
((wx(()) ((xx((1) 2log + log 1 + g
(D.253)
53

---

where C' is a universal constant. Thus, applying Lemma D.25 with = to. we find that for all tol, for all 2 4n wx(1) 2 <²x², and furthermore, for all E tol, we((ss S 1 x
[T2, E
To finish the proof, there are two cases to address: either D2.10 < 0 and Da,t0 = 0, or D2, = 0 and
Da > O. In the first case, by Lemma D.26
> O, meaning that for a sufficiently small at Da.t t=
open interval of time J containing fo. Da.t is increasing on this interval. This gives us a contradiction, since it implies that for some t/ > fo (with t/- to sufficiently small) D4, c nonnegative on [to, 16]. and since 12. < O, by continuity, 22.t < 0 for t E [to, tyl as long as to - to > 0 is sufficiently small. Thus, sup I to. which is a contradiction. In the second case, we can apply the same argument using Lemma D.27 in place of Lemma D.26
In summary, if Da 0 and D2,12 < 0, then for all t 2 T2, D2 S 0 and Da.t 2 0. As consequence of Lemma D.23 Lemma D.24 and Lemma D.25 for all E [i,, (M), (x(()) 2 5² x
and furthermore, wy (tx) - 1 ntg". for all t 2 T2.
We now proceed to bound the running time of Phase 3, Case 2, by dividing it into few subcases, and showing that the running time that each these subcases contribute is within our desired bound. The first lemma below bounds the rumning time spent in the subcase where the positive root of Pa is larger than 2 (and where Dr.t is large relative to D2,1).
Lemma D.29. Suppose we are in the setting of Theorem
3.3
and assume that Da,72 and
D2,T2 i.e. we are in Phase Case 2. Then, at most
E2ge log
22
time 1 > T2 elapses such
that the following hold:
1. I is the positive root of Pe (as defined in Lemma 4.4 then r 2 2"
2. D4,1 2 22,l
3. [(pt) 2 (02,a + +0 o22
Proof of Lemma D.29 Lett 2 Tr such that 1D,,ll 2 D2,l We can write
(((w) = 22, + 462,a D = Ds (w +r)
(D.254)
where r is the positive root of (ww)). We are now going to lower bound the expected velocity while r 2 2 by lower bounding (w) for large fraction of w.
For all 2 4, and 2 T2. we have (x(() 2 g2x², by Lemma D.28 Thus, Punep (w 2 (2n²) S
1- (k) by Proposition D.6 Additionally, because D2, S 0, we have E. Ew-a, [w²] S 72 + 0(1dd)
by Proposition D.38 meaning that by Markov's inequality, Pwowp, (|w] 2 3 S 72 - 1/d S 72,
where the last inequality is because d 2 c3 by Assumption 3.2 By union bound,
Pumpp (100l E x 1/3) 2 1 - O(12 + +x)
(D.255)
For particles w such that |w|
E (52 2 1/3), we can lower bound the velocity if r 2 1
[w)(w)] 2 (1 (P) + (w)
(By Lemma 4.4
2 > IP((w) Q(w))
(B.c. w S 1/3)
(D.256)
2 120 + 46 D
(1) 1) " + Aa (8)
(By Lemma 4.4
(2,1D2,r pl + 6Gf Da,el )ool
2 12 D2 40, i D Dr
(By Lemma 4.4
d
02 D2,1l
D2,el el + -o2. Ds.el
2 D1. /
r||
r||
+
+l|
0
(By definition of r)
d
1
02.
D2.el + (2adDDadd
> 02 Dis
622²
0
6 2
d
(B.c. w E 152n2, 1/3] and r 2 1/2)
2 20g
(D.257)
54

---

where the last inequality is because |D,1l 2 D2,el and d is sufficiently large by Assumption 3.2 Thus,
E (u((w))2 whipe
2 P w (1w) 153x2,1/31)
E [[x(w)) |w| E [c2x9,1/3)
(D.258)
wmp:
(11 - O + k)) PR,55
(By Eq. D.255
2 0 D2
(By Assumption 3.2 since 72, d sufficiently small, large resp.)
and
dl(p))
2 02,5
(By Lemma D.39
dt
2 (D (2D22
(B.c. Da.1 2 D2,))
2 5R + =
(D.259)
3 -00,05" (ptt)
(B.c. 02 x 0 by Assumption 3.2)
the amount time spent in the case where r(t) 2 2' Da.el 2 Dael and (pp)) 2 (02,a + o2 after 2 T2 is at most
Since L(Po)= 3 by Lemma D.8 and Assumption 3.2
03 tK log
72
by Gronwall's 's inequality (Fact) 1.13
Next, we deal with the subcase where Da,t is small relative to D2,t- Lemma D.30. Suppose we are in the setting of Theorem 3.3 and DA.T, = and D2,T, < 0, i.e. we are in Phase 3, Case 2. Then, at most 55 xx hold:
log 22 time t 2 T2 elapses such that the following
1. Da.l D2,
2. L(p:) 2 (02.a 6 o),
Proof of Lemma D.30 Suppose D4,el S E),2, at some time 2 T2. Then, we can lower bound the velocity for a large fraction of w's as follows. First, we provide bound on (.(w) (defined in Lemma 4.4):
(w(w) 20 2,0 D2 + 462 Da
(By Lemma 4.4
-20 12, + 452 D4,1/
(B.c. D2 < 0 and D4,; > 0 by Lemma D.28
-2022al D2, + O 12.(w²
(B.c. |D4,1 S E D2.1
-200 D2.1 w 0(50) (2,
(B.c. 02,d < o by Assumption 3.2
-262, D2. w (1 11-001 O()))
(D.260)
Next, we provide a bound on Qc(w):
[Q:(w)]
PAa (2) + Ad (3)
(By Lemma 4.4
D2.1 + - Da,al loo|
o
(By Lemma
2,d D2.1
O
[wo
(B.c. [Da,l S D2,1 and 6 < o3.0)
d
O D2, w
(B.c. 1/d < Tog logd = by Assumption 3.2
Thus,
(P((w) + Q.(w:) 202 D2,e/ /w O(E))
(D.261)
By Lemma D.28 have D2, < 0, meaning that by Proposition D.38
E
[w?] I 7/ + O(L/d)
(D.262)
wi-pe
55

---

Thus, by Markov's 's inequality and Assumption 3.2
PP Pwmppe (ww) 2 1122] x 72
(D.266)
Furthermore, by Lemma D.28 for all LL. w((() 2 x²x² for all times t 2 T2, meaning that Wt 2 5²x²² with probability at least 1 - (K) under pt. by Proposition D.6 Thus, by union bound,
P Purpe (lw) E [<? 1/21) 2 0(12) O(k) z
(D.264)
where the last incquality is by Assumption 3.2 since 72 and d are sufficiently small and large respectively. Thus the average velocity is at least
E wrupt
2 P Pump. (52 x S
|w| s 1/2). wmpp
E [l((w)) 52x² |w| S 1/2]
(D.265)
E [1) 152x² s |m| 1/2]
(D.266)
2 w~p:
M E [|u(w))] / s²x²² |w| s 1/2
(D.267)
wmp
E [11 (1- Pi (w) + (w)) g² < |w| S 1/21
(D.268)
wmept
[[P((w) + Q: (w) g S |w| 1 1/2
(B.c. |w|
S 1/2)
wmipt
E [ D2,1 u e²x² s | S 1/2]
(By Eq. (D266])
wmp:
2 22 D2,1 2²²"x²²
(D.269)
Thus, any time t where Da.1 s D2,ll we have
dl(p))
2 2,al D2. 12,115" k5
(D.2700
dt
2 0 (6 D2,1 2 +. I Da.e 1)
(B.c. [Da,tl D2, and o2, S 62,a by Assumption 3.2
$ --
c4 K L(p))
(D.271)
Since the initial loss is L(po) S oá by Lemma D.8. by an argument similar to that used in Lemma D.29 using Gronwall's inequality (Fact) I.13) we find that
[ 1( s (((()))) 2 (62,a + a S
72
log
(D.272)
Le. the time spent in the subcase in the statement of this lemma is at most desired.
2. log
(>=):
222
as
We have already considered the case where |Da,tl 2 D2 and r 2 1. Next, we must deal with the case where Da,ell 2 D2,l. but where r S _.2 We further split this case into two subcases. In the next lemma, we deal with the subcase where w/ ((x) 2 1 c. meaning a nen-nggligible portion of the particles are close to
Lemma D.31. Suppose we are in the setting of Theorem 3.3 and D 0 and D2.12 0, i.e. are in Phase 3, Case 2. Then, the amount of time 2 T2 that elapses while the following conditions simultaneously hold:
1. Du.l 2 D2.1
2. r \/5 where r is the positive root of Pt
3. (x((R) 2 1 1-
4. L(px) 2 (62,1 + o2,
2 1z log 2
at most
56

---

Proof of Lemma D.33 If (((()) 2 at any time, then the subsequent times when it could possibly be moving to towards 1 are:
1. During phase 2 - note that before phase 2, We (<R) S log so it is not possible for w, wx(4)
be larger than Ç since, by Lemma D.9. wy (4R) increasing during Phase
2. The times 2 T2 when Da,,e - 2),,
3. The times t 2 72 when D 2 22,1 and r 2 where r is the positive root of (:(w)
The only case not mentioned above consists of the times 2 T2 when |Pa.el 2 D2,1 and the positive root of (ww) is less than 2- In this case, ((x) will be moving away from 1. For this reason, we also do not need to consider the case discussed in Lemma D.32
Thus, we will upper bounds on the velocity and time during the first 3 cases listed above, and a lower bound on the velocity in the last. First note that the velocity can always be upper bounded as follows:
Uy (w))) S - w1 + w) (P((w) + Qx(w)
(By Lemma 4.4
D2 + 02,0 D4. 0
$ (1 w1 - w) 2 D2. + 46 Drelu + +0(
d
(By Lemma 4.4
(1 w) (02 + 07,0 Da,tl)
(B.c. |w| S 1)
Thus,
d (1 w) 2 (0 D2,el o7 Da, w) 2 (02 + o2,a)(1 dt
- w)
(D.273)
During Phase 2. By Lemma D.10 the rumning time during Phase 2 is T2
Ti x
25. log
12 At the beginning of Phase 2, we (4R) S log by the definition of Phase 1.
Thus, by Gronwall's inequality (Fact I.13). at time T2.
WT, (IR)
log d
exp
(02.a 02.0)
(6 + 0 n
log 7/
(By Fact 1.13 Eq. (D.273) and bound on T3 -T))
log ?2
exp
(D.274)
log d
Times 2 T2 when D4.el s E|D and I(p:) 2 (02 + 22. By Lemma D.30 this case
time to the overall running time. Thus, by Gronwall's inequality (Fact I.13), the additional factor that this case contributes to we(4x) is larger than or equal to
contributes at most 5dx log
22
exp
(62,a + 02,))
02, log
12 (22) 2 exp (- gd
C'
log
22
(D.275)
for some universal constant C.
Times 2 T2 when |Da,el 2 E/D2, r() 2 and T(pt) 2 (6 + d2. By Lemma D.29
this case contributes most 0, log
the overall running time. Thus, by Gronwall's
inequality (Fact L.3 the additional factor that this contributes to wx(4R) is larger than equal to
exp (6 + +2,a)
03 R R log
(2 (2)) 2 exp (-
gákō log
C
7/2
(D.276)
for some universal constant C.
57

---

Times 2 T2 when |D,,1 2 2l, r(t) - 2/2 and wR 2 1 Ç. In this case, we obtain lower
bound de (1 -UR), where have written WR we (AR) for convenience. First let obtain lower bound on P. (:))):
Po(WR) 2 22, + 402 2,
(D.277)
2 40 :R. - r) (wR + +r)
(B.c. is the positive root of Pt)
Ds -r)
(B.c. D4. 2 0 and wo - 9
2 D4.t
(B.c. and & S
log log d)
Next let us obtain an upper bound on Qu(w)):
1c((w))
Ad 01 + (
(By Lemma 4.4
02 12 + 02 D4.1
(By Lemma
d
0 Da.el
(B.c. D2 $ DDa,l/E and 02.0 S 63,x by Assumption 3.2
Ed
Thus, because
log logd
Ed
P((wR) Qu(w))))+))))))))))))(()))))QQ)_ > Da.
(D.278)
and when WR -
- (1 wR) 2 (v((
(D.279)
dl
z (1 wR)(1 + WR (Pr(WR) Q+(wR))
(By Lemma
2 (1 wx) Ds.
(By Eq. (D278) and b.c. WR
Furthermore, since |Da.el 2 D2,l. this means that [Da.t] 2 &c, since otherwise, |Da,el S and |Da,el S Çe, and the loss would be less than (02.a a) Thus, if time Tese is spent in this case, then by Gronwall's inequality (Fact I.13), this case contributes a factor of at least
exp(Tee Dat) 2 exp(Tex ol,
(D.280)
to - wg.
Summary of 4 Cases. In summary, by Eq. D.274) Eq. (D275 Eq. (.276) and Eq. (D.280).
1
16 log (2
C
(2
1 WR
exp
exp
log
(D.281)
log
E log (22) exp(Tew oZ,
exp
(D.282)
where Tesc is the amount of time spent so far in the case where DD,el 2 D2, r(t) 2 and
WR 2 1 -1-E g. This is fact a lower bound on - wy when L(pt) 2 (62.d d + og, since at all
other times when L(pt) 2 (03.4 + 0 oz, we have that 1 is increasing (i.e. wp is moving away from 1). In particular, if
Texe oe ( 10 log (22) + Ed
C
log 72 (22)=
C
log 72
22)
+
à2 1
12 log
(D.283)
then WR will be at least ç.
Finally, the following lemma deals with the last case, where the positive root r of P, is at most 2. and additionally, wx(R)) 1
Lemma D.32. Suppose are in the setting of Theorem 3.3 Assume DA.T, = 0 and D2.Ts < 0, i.e. Phase 3 Case holds. Then, the amount of time T2 that elapses while the following conditions simultaneously hold:
58

---

1. Daal 22
2. r 5
3. we((R) S 1 -
4. I(p))= 2 (02,d )2,
at most
log
12
For convenience and ease of presentation, in this proof we define 7: v12. and B 2 1.1 such that 74 3T4. (Note that by Assumption 3.2 [ is most universal constant.) We first show that there is always large fraction of particles which are far away from the positive root r of (.(w). We consider two cases: (i) S 73/2 and (ii) r 2 73/2.
Proof of Lemma D.32
Case 1: r S 73/2, Assume for the sake of contradiction that
IP Pampl (1rol 2 r 2r ST
(D.284)
Then,
[w s P Pwwpp (Iw) 2 r 70/2) + P Pampp (lu) S r + 73/2)
[w
|w| S + /4/21
wript
woups
(D.285)
7 = (r+
(D).26)
S 74 + (273/2)1
(By assumption that r S 73/2)
74 + 167
(D.287)
Thus,
D4.t E [Pa.a [w] 371
(D.288)
wmipt
I E |w-| + 0 (22) -3
(By Eq. C..4
wmp
~ + 167 + +0(2) 374
(D.289)
167 + +0 - (B 1)7^
(7)
(D.290)
< 17,6 - 1)74
(B.c. is sufficiently large by Assumption
3.2
<0
(B.c. (B - 1) > 1772 by Assumption 3.2
which contradicts the assumption that we are in Phase 3, Case 2 since DI,t > 0 by Lemma D.28
Thus, if < /3/2 then
P Pampp (luol 2 r + 2 4
(D.291)
Case 2: r 3/2, Suppose that
P Puopp (r- 3/2 <|w0| < r + 73/2) 2 1
(D.292)
Then,
I2 wruppt
> P wnup. (r- /3/2 < |wl S 3/2) wmmpe
E [wi |w| Ir 3//
3/2
(D.293)
(1 75) (r (3-33/2g3 3/2/2
(D.294)
Since D2, < 0 by Lemma D.28 this implies that Ew-p, [w?] <² + O(1dd) by Proposition D.38 Thus,
(1 - - - ((1/d)
(D.295)
59

---

and taking square roots gives
(1 -0/12/2 ) S + +
(D.296)
Thus,
(1 (1- -5/// (++ (
S 7//2
(D.297)
On the other hand,
E wrept
[w0]
P wopp (ww) - [ r
718/2) E -w |mm E [r - 7:3/2 r + 7 /
(D.298)
wmpe
P S (r + 3/2/4 +
Puwpe (1w| [r- - r 73/22)
(D.299)
(By the assumption in Eq. (D.292)
and since D4.t 2 0 by Lemma D.28 Eupp, [w=]] 2 3+4 (1/d) by Eq. (C.4 and because w Pt
are bounded by 1 in absolute value. Thus, by the above equation,
(r 7/2)t + 7 2 3r 3/-r-
(D.300)
Rearranging gives
(r ( + 75 + (
2 BTd
(D.301)
and taking fourth roots gives
r + 3/2 + 75/4 +0 1
d1/4 2 BL/a7
(D.302)
(by the inequality v u VI + VT). Therefore,
2 31 73/2 -5/4 0
1
(D.303)
d1/4
Combining Eq. (D.297 and Eq. D.303 gives
81/4 3/2 5/4
I /3/2
1
(1+)
(D.304)
d///
(1 11-15
Rearranging this equation gives
(g - V1
r T 273/2
+ 5/1 +
+
1
0 (
(D.305)
d1/4
-5
va
< 5/4 +0
1
1
0
1/4
vd
(B.c. T TS 1, we have 3/2 < 75/4)
S 5/45/4 5/4
VI-
(B.c. d is sufficiently large by Assumption 3.2
< 5/75/4 + 0
(B.c. T 1/2 by Assumption 3.2
< 5/75/4
(B.c. d is sufficiently large by Assumption 3.2
Further rearranging gives
81/1
S 1/1
(D.306)
VI
This contradicts Assumption 3.2 since 3 2 1.1, the left-hand side is larger than an absolute constant, while the right-hand side be made sufficiently small (since 72 C2 and C2 is chosen to be sufficiently small). Thus, in the case that r 2 3/2 we obtain
P womp (wwl & [r 3/2 + T ///2) 2 5
(D.307)
60

---

In summary, whether r 2 73/2 orr S /3/2, we can conclude that
P Peopp (/wo E (r- /3/2 r + T -0/2)) 2 7
(D.308)
Further assume that wy (1R) <²x². Thus, by Proposition D.6 with probability 1 (()) under Pt. we have g²k² w S 1 - By a union bound,
6, and note that by Lemma D.28 for > n. we have (1) 2
F Purpp (1/w) -r 2 7 3/2 and 5²k² V | S 1 - {) 2 7 0(() 2 2
(D.309)
Suppose w satisfies ||w| - r| 2 3/2 and 5²n²² < |w| < - E. i.e. the conditions mentioned in Eq. (D.309. and write
Pi(w)= 2 + a D L () +))
(D.310)
Then.
[o((w)]
N 2
w²1 PP((w)) + Q((w))
(By Lemma 4.4
1 + w| -1 w| (P(( Q.(w))
(D.311)
22 (ww) + Qw))
(B.c. |w|
S -
- - ( 02 ( D2,el el + o DD,el
25. 1
|w
d
(By Lemma 4.4
( 2.al D2 + + Da.c)
26(1 14,
73/2 7
-
0
(By triangle inequality and b.c. |w| 2 E²n² and ||w| - r| 2 3/33
0 09. (01,|Da.d) ))
2 E [00 ( D4, 3 22
Ed
(B.c. D2 S |D,,e/E and o2,0 S 62 by Assumption 3.2
25 . 462,al DT gg2
(B.c. 1/(Ed) log load and d is sufficiently large (Assumption 3.2
2 0 - nー
Da.ll
(D.312)
and the average velocity at least
E [wr( R PP wrpp (1|wl r 3/2 and g2x² S |wo| S 1 -) o 1. 55 k Da,el wmpp
(By Eq.
(D312)
2 5 -50 Das.!l
(By Eq.
(D309
2 04 50 Da.e!
(D.313)
6 ((p)))
(B.c. [Da,e] 2 D(, and 02, 2 03.0 by Assumption 3.2
2 0 11 s L(pt)
(D.314)
Thus, by Lemma D.39
dL(pt)
x -02 1.d 21 L
(D.315)
dt
Since the initial loss is L(po) S (6?.a + a by Lemma D.8 we can calculate using Gron- wall's inequality (Fact[I.T3) that the time spent when the conditions in the statement of this lemma simultaneously hold is most
log
12
desired.
To conclude, we show Lemma D.33 which gives bound on the running time during Phase 3, Case 2. Lemma D.33 (Phase 3, Case 2 Summary). Suppose we are in the setting of Theorem 3.3
Assume
D4,T= and D2,T, < O, i.e. Phase 3 Case 2 holds. Then, most O
log
22
02/ 11/2 10/2
time 2 T2 elapses while [(p)) 2 (62,d + 82 o2,
61

---

Proof of Lemma D.33 By combining Lemma D.29
Lemma D.30
Lemma D.31 and Lemma D.32
we find that the overall running time during Phase Case 2 most
log
log
12
1
log
12
log
2,45
2,d5
E+K
0 cf
4.d 12
11/3 11/3g8m4 g8
(D.316)
and this can be upper bounded by O
g0 log
22
where we have used the fact that
2,2
2,d x 1 (see Assumption 3.2)
D.7 Summary of Phase 3
The following lemma provides bound on the running time after T2 that is required for L(pt) x (02 + d..
Lemma D.34 (Phase 3), Suppose we are in the setting of Theorem 3.3 Then, To. T2 x
(log log d)
log
22
Proof of Lemma By combining Lemma D.22 and Lemma D.33 we find that the overall running time during Phase is at most
max (o
log
( (
log
72
))))
(D.317)
02
3,d
12
k
07
12
11/2 g8
and this is at most O Assumption 3.2)
log
22 where we have used the fact that 03,0 x o7.a (see
D.8 Proof of Main Theorem for Population Case
Finally, we prove Theorem 3.3
Proof of Theorem 3.3
We combine Lemma D.5
Lemma D.10 and Lemma [D.34 to find that the total
runtime is at most
(log log d)
(log log d)20
Ta.
log d +
log
22
+
log
72
(D.318)
2,
(o2. + 02,a)
2 2,
2
and we can further simplify the above bound to obtain:
1 log d + (log log d)20
T. S
log
(D.319)
2,072
02 2
where we have used the fact that 03 x Sh,0 by Assumption
3.2
D.9 Additional Lemmas
Lemma D.35. Suppose Assumption 3.2 holds. There exists some T. with r 2 log logd and r S 1, such that if p is the singleton distribution under which r has probability 1, then the velocity at r is 0.
Proof. Let p be such that has probability under p. for some r (0, 1). Applying the definitions of D2 and D4 to p. we have D = P2,(()) 72 and D = PI,a(r) - Y4. By Lemma D.3 the velocity of is
- t3,
(D.320)
The second factor is equal to
22, - -2)
(D.321)
62

---

and we denote this polynomial of r by F(r). Note that (2 = Pa.a(1) 1 (see Equation (2.20)
of Atkinson and Han (12).. Thus, F(1) > 0 by Assumption 3.2 since (2,'(1) > 0 and Pa (1, > 0
by Eq. (C.4). On the other hand, suppose r x log logloge Then,
d
1
|P2a(rr]= = d d-1
(By Eq. C..4
1
(D.322)
(log log d)²
Similarly, [Pa,(rr) S (Tog log Thus, P2.((r) - 72 < 0 and Pr,((r) -4 since d can be chosen to be sufficiently large according to Assumption 3.2 However, by Eq. (C.4), one can show
that P2,d' (r) Z log logd and Pa.a (r) (oog as long as d is sufficiently large. Thus, F(r) < 0
for x log log In summary, F has root between C for this value of the velocity of r is O.
log log and 1 for some universal constant C, and
(log log d)2n log (22) Thus, (02.d
Lemma D.36. In the setting of Theorem 3.3 Ta Ti x
2,
0 (2()) Ti)
(log log d)
log
c72
Proof. This follows from Lemma D.10 and Lemma D.34
D.10 Helper Lemmas
In this subsection, give some convenient facts that use in the following subsections. Proposition D.37. If0 S < 1/, then for all t 2 0. (1((1) < (2((2).
Proof of Proposition D.37 Assume this is not the case, and let & be the infimum of all times s such that (1)) 2 (2((2) Then, (x(41) = (t2))) meaning that for all t' 2 1, (11(41) = (t2))2). For convenience, let upp (1(41) (12)) Note that > since ( F 12- However, this implies that
for /' E (0, t). (r1( < (t2). meaning that for any 6 > 0, W, (11)) and wx(12) are non-identical solutions to the initial value problem on the time interval (t 8, + 8) given by the conditions ft) = Wh and df
= (f(()).. This contradicts the Licrdo-Lineelof theorem, and thus there cannot exist time such that wx((1) 2 (12).))
dr
Proposition D.38. Suppose we are in the setting of Theorem 3.3 and let 2 0 such that D2,1 0. S S
Then, Ewpp
[w
+
Proof of Proposition D.37 Suppose D2, S 0. Then, Eunop, (P2,((w)] S 72, which implies that
[221
S 72
(D.323)
w wm
d
and rearranging gives
d -
1
< 72 + +
1w
722 +
(D.324)
Wrupe
d
as desired.
Lemma D.39 (Decrease in Loss and Average Velocity). Suppose we are the setting of Theorem Then, dl dt p. ((p))
3.3
We note that similar result was shown in Lemma E.11 of Wei et al. 75 but for gradient flow rather than projected gradient flow.
Proof of Lemma
D.39 First, we can calculate
d
L(pt)
1 d 2 dt 3- (.) -()))==
-(==))))))
E
(Lo, v(x)) Sp. (x)
(D.325)
dt
63

---

Furthermore, we can expand
d dt
fp (I) as
d E
=- E
d'(u (()xx)) x) grad [(pi)
(D.326)
dt amps
ampp
Therefore,
d
(Foc(z) v(x)) d
[(p))
E an80-1
dt fox (z)
(D.327)
dt
(Fn((x) -0)))/ (u Z) grad, g(a)
E
E expe
(D.328)
z~ga-1
E grad [((pt)
(, - y(x))) x)01]
(D.3299
urep
am3e-1
grad L(p)) vul(pe))
(D.330)
umpt
E
(gradd,(pp))
amPi
(B.c. grad,. = (1 wu VV and (1 -/u'm T)2 = (1 wu
This completes the proof.
The above lemma implies that (p:) is always decreasing. Note that frequently, when we apply the above lemma, we will only consider the first coordinate of grad L(p.).
Proposition D.40. For any distribution p on R.
1w w w'13) 2 Var[w] 81, m
E
(D.331)
weup
Proof of Proposition
D.40
This follows from short calculation:
E
||w
w/ 19
EE
(w² 2ww' + (w')²) 2 E
[l 2( E wmpp
[w])
2 Var [w]
10,10 p
w0, wowdrrp rpp
wmip
weup
(D.332)
as desired.
E Proofs for Finite-Samples and Finite-Width Dynamics
In this section we give the proof for results related to the empirical dynamics, where the number of samples and network width are both polynomial in d. The end goal is to prove Theorem 3.4 the proof which given below.
Proof of Theorem
3.4
We have
L(pT...)
=
E [[pT... (x) )(x)))
(E.1)
< 3 E
[(fere.
(ore..(0 (x) - -fpr.. (z)) +3
[(prr.. (x) fpr.. (x))
(E.2)
z-g0-1
3
E B- [(fer... (x) -)(2))
(E.3)
(log (02. + o22a)dd (log 01)
(62 d + a $
+
(02 + 22.
(E.4)
d
m
where the first inequality is by Cauchy-Schwand inequallll and the second inequality is by Lemma 5.1
Lemma 5.2 and Theorem 3.3 Since e = log l the last term dominates because the conditions
of Theorem 3.4 imply that m 2 d3, and because 23
p-1 2 > 1, meaning that the first term at most
To complete the proof, give proofs for results in Section 5 We first give proof for Lemma 5.1
64

---

Proof of Lemma 5.7
For the first statement, for any fixed t - T, by Lemma E.13 we have
E (0 for (2)) (02 + 0 ad + log (log d)c
0 + δ2
+ d90lod)
(E.5)
m
with probability we will choose shortly. addition, for any t, E [0. Ti. by Lemma E.14 we have that l1 ue (x)l22 2 (0 + dt - t'l, which by Lemma E.12 implies that for any gd-1
(x) (x)( < E (x(u((x) x) aarr (x) x)]
(F.6)
x~po
((z2,d V N2 ف / Na.d) (02,a + oz 1')
(E.7)
In particular, if
(6 t in Jdiiinggs then for all gd-1 we have
002 64.dl ( d)
(x((x) Sn (x)
(E.8)
meaning that
2d + 57.
E 2 2~8t_, fox
(E.9)
dRllog d)
By the same argument, we have that
2.d + o (Rog
E (xx
(E.10)
Now, we perform union bound over all times such that S T and t is an integer multiple of d
(0 a ja lord for some sufficiently large absolute constant H O. Since T S 0 0 the
number of such times is at most HogHlogdd Thus, with probability at least - 35.0--4 C logd we have
(62 + o + log ( d)) - (x)))
02.0 + o..0 d02 d)
E -
(E.11)
m
for all [0. In particular, in order to have exponentially small failure probability, set 2 x244 and by our assumption that S (loggd d), we obtain
8
10 0 d (log (11)
E - - (x))))
(E.12)
for all < T, with probability least 1-c e- since log 1 de Ollg d)2). This completes the proof of the first statement of the lemma. Additionally, by Lemma E. 16
we have
(z)a 1[,
E
Jr. (x)) (62.a + (7,))
E (a,R)~
(1 we1
(E.13)
2~
which proves the second statement of the lemma.
The rest of the section is dedicated to prove Lemma 5.2 To begin with, we first introduce the following lemma which formalizes the decomposition of a ب - dellz discussed in Section 5
dt
Lemma E.1 (Decomposition of de f ūz 121)). Suppose Then, d. ûú u.112 A, + B, + C2 where A, 2(grad, L(p.) grad, à, ū.), B, -ggrad, (pt)) grad L(p)), dt ūz), and Cs = -2 2ggrdd L(pt) L(ad) us De).
in the setting of Theorem 3.4
Proof of Lemma
E.
By the chain rule,
dt 1/ ū, 11
2( u,
ūy Ds - u.) = =-(ggrad - grad, dn a.) (E.14)
dt dt
65

---

We can expand the difference in gradients as grad, Tr).) - grad I(pt) = (grad, [((ee) - -grad T.(ba) + (gradT. (0r) - gradT-((1i) (F.15)
+ (grad - grad (pt))
(E.16)
Thus,
d
di llū Jell2 2gradd, L(p:) grad L(fi), it Te) 2(gradLa (pr) (gradda((pp)) 0 -x))
(E.17)
2grad, (pt)) - ((pi), ae- u.)
(E.18)
C2 Bt + A
(E.19)
as desired.
Next, we give proof of Lemma 5.4 which gives upper bound for each of A,, B, and C..
Proof of Lemma 5.4 The proof directly follows from Lemma
E.8 Lemma
E.7 Lemma
E.9
Lemma E. and Lemmal
E.I
Combining the upper bounds Lemma 5.4 and the decomposition Lemma we have the
following Lemma F.2 which upper bounds the growth of ||ũ - well- Recall that Te.r is the total I log (see
runtime for the algorithm (see Theorem 3.3) and Ti is the time such that wr, (tu)
Definition 4.5)
Lemma E.2 (Growth Rate of ||ó, 112 With Dependeney on n). In the setting of Theorem 3.4 suppose that the inductive hypothesis in Assumption 5.3 holds for some and v until t for some t S T..e Further assume that vb < 3 and Φ > 2. Then, when S Ti we have
dt [ -0rli:l
wr113 < [462 2. + ( log
1
|| - url12
(E.20)
+ (62,a + R..)) d(logd) d))(1)
|| Tz 12
(E.21)
vm
+ (62, + +6%,)) (log (log) d(
1
/ d
+
d
2.5-221 24 20
1 ūelle
(E.22)
n
n
2-20-0
91-10
20 26
+ (03,0 + )g (h,a) (log on)
| - 1
(E.23)
and
d
1
A < (60 D2,11 [1+0
A
(E.24)
dt
log d
d(log d) 0(1)
(62 + o..))
(E.25)
Vm
d
22.5
21 20
(03.8 + 0% (log do
+
A
(F.26)
+ (02 + 02 (log (logd) 011)
o11)
1
+
22-20-0
11-10-20
A
(E.27)
db
When 1 [Ti, T.... we have
di | - < (0 a
2 a 2
(E.28)
d(log d)o(1)
+ (6%, + (..))
||
(E.29)
Vm
+ 10% + + 3 (log or1
1
+
V
d
d2.5-20 22-22222
11 uellz
(E.30)
do
n
22-00-0
20-6
64-400 4n)- 26
+ (62,0 + 8 (log 0()
00 - Tel1
(E.31)
66

---

and
d Λ $ (02,d 04,0)
(E.32)
dt
(6 + Rid)) d(logd)00)
A.
(E.33)
Vm
(2.5-20-26
+ (6 + (7,n) (log d) o01)
(
+
D
(H.34)
(62 + ( (log dl on
1
+
d2
22-20-0
d 10-10-20
A
(E.35)
do
n
Proof of Lemma E.2 This result follows directly from Lemma 5.4 and the inductive hypothesis. By the inductive hypothesis, we have A, S d= We apply this to the bound for Ce in Lemma 5.4 to obtain the desired bounds above. Note that in our bounds for a A 2. it is important that the upper bound for [[B.] in Lemma 5.4 instead of the upper bound for Bt.
Using this bound on the growth rate, we can show that the inductive hypothesis is maintained.
Lemma E.3 (IH is Maintained), Suppose
in the setting of Theorem 3.4 Suppose 1 > 0 and 1> Φ > 1/2 + th. Let n = d' (log d) where μ 2 max + 2á, 4 - 26 4 and C a sufficiently large universal constant. Suppose the inductive hypothesis in Assumption 5.3 holds for all t < T where T < Ta... Then, when m Z d (log d) and m S dº for a sufficiently large universal
constant C, we have that the inductive hypothesis holds for all & S T + Oi where qu 0 is constant that only depends on d, Ø, V. 02 Gs,d-
Proof. Suppose the inductive hypothesis holds for all t where T S Ti. Then, by Lemma E.2 we have
d | ūz < 333=x(43... D2 Tog 11 ue112
(1+ 0
(E.36)
dt
(6 22 d(oog d))
||ū 0el12
(E.37)
Vm
(0 2.a + +0% (log d) d()
+
||4 - Ty 1ell2
(E.38)
s [403. D2.1 (1 (1+0 1 do
log
[| del12
(E.39)
0 (02,a + 02 o 011)
|| - -ulll))
(E.40)
d
Here we have used the fact that dlog d) (1) where p 2 max(4 4v- 220 26) well as
the fact that Φ 5 as follows:
In the third term in the bound for d [ ue from Lemma E.2 only the 1 term
is important - this term dominates the V and d 222---22 terms, using the fact that
Jr >>>> 26.
In the fourth term in the bound for || 1el1 the d2-00 1 ū1 12 term is dominated by the 1 ج uell2 term this is because μ 2 2 26.
Since 2 4 4-44 - if we increase n by (log d) 01)) factor, then the d1-4w-20 12 ūe 11
term can be absorbed into the first term, as it is dominated by the 46 A62,al 12.
1
|
luga
11 term. (Recall that by Lemma D.8
D2 Z 72 during Phase 1).
67

---

In the second inequality above, we have used the fact that m 2 2log d) the term involving m dominated by the [log (lod) o(1) Additionally, by Lemma E.2. have
this implies that
1 - -eell term.
de
d A <
(2. 0 1
log ()og
(6 + +%,a) d(lg d)
A
(E.41)
dt
vm
(02 + (2,0)
+0(
d0.5+ d) d) 41)
(E.42)
(02,a + 07,a) 00.5+ (log d) o1)
$ (662,, (2, 0 log
(
D.
(E.43)
Here, apply the bound for A, from Lemma E.2 In the third term in this bound, the V
term dominates the d2:5-20-20 term, using the fact that @ > and 2 2 + 2Φ > 3, meaning In the fourth term, we deal with the d 44-26 26
term similarly to our ze l/ú; 8.l.. We can also deal with the d2-20-0 term similarly, since by our definition of μ., this will be at most 7 (log djory and will thus be absorbed into the first
f2.5-20 26-20-10
20
Vx"
calculations for the bound for d
term involving D2 Using μ 2 + 2 we then obtain V 2o.5+a as the coefficient of the A term (i.e. the first-order term involving Σe). Note that here we need have m 2 d (log d) ()))
in order absorb the term involving m into the 20.5+7 Ar term.
Now using Lemma we have
(0 + 02.0) 0 Tobto) )
exp (f
40 do (1+ 0
log d ) (F.44)
(log d)
462 min, D2,1l
and
(02,a + 07 do 462,a min, D2ee
11(1)) )
exp (f." 40 ol (1 (1+ +0 (1ogd) dt) (E.45)
A $ (log dogd)
By Lemma D.5 we have
exp (
T 462,a D2, S
(E.46)
1=0
and by Lemma D.8 we know that |D2,ll Z during Phase 1. Thus,
2 max,7 I of (log d) -2
(E.47)
Since 26 - > 2 and d is chosen to be sufficiently large after 72 is chosen (Assumption) 3.2 we
have that A A < 175 have
1l < gv Note that Amxx, is continuous and we
A Amxx, I max
1/ - Tellz
(E.48)
df
x dt
1
d
max x
di l u.ll?
(E.49)
lit
ūz
< max
| ie (ūg - ūy grad,, I(ū.) grad,, L(x)))
(E.50)
x
< max
Igrad, [(0r) grad,, L(u))ll2
(By Cauchy-Schwarz Incquality)
X
< (02, + +o7
(By Lemmal E.14
68

---

( do - (0 ( we still have the inductive hypothesis for Amax
Thus, for #r =
until time M1, that is, T+p S de. Similarly, our calculation above using Lemma 1.12 gives us
A S o (2og d)o()))
(E.51)
Thus, we have AT (Pllog d)om < 2 (where in the middle expression, choose smaller power of (log d) ((1) that the inequality holds). Observe that A, is continuous in and
d A.
A, A
(B.52)
di
El(ūz U1. 0/1 --ti
(E.53)
A,
[lllie ūt 12 [2]
(By Cauchy-Schward Inequality)
A,
(62 + dd
Elllūr ūz [2]
(By Lemma E.14)
A,
< ( + o6 d.d)d.t
(B.54)
Thus, for 422
20 8e (logdjoors
t0 dd we still have the inductive hypothesis for until
time 7 + 122 i.e. AT+r2 78. As result, assuming the inductive hypothesis holds up to time T, it holds up to time T + min(j1,,
Our above proof strategy applies to all times T S Ti. For the induction when T [Ti, T..c] follow a similar argument but instead use the bound on A, obtained from Lemma E.8 Since we know from Lemma D.36 and the assumption that 0 log loga) that
[ (62, ( toad & (T.e (i) + o7 S poly logd) d) T
(E.55)
O(1)
we know that this additional phase will at most add another exp(poly(log) log d)) factor to the growth of Amax.t which is smaller than any polynomial in Thus, Eq. (E.47) and Eq. (E.5T) still hold for this period, which finishes the proof.
Finally, we prove Lemma 5.2
Proof of Lemma
5.2 The proof directly follows Lemma
R.3 with o m-1 and 4-3
E.1 Upper bound on A
The goal of this section is to obtain an upper bound on
A.(x) 22grad, grad, (x(2) xx(x))
(E.56)
When there is risk of confusion, simply use Ac. ûr and ū,, omitting the dependence X-
We first obtain strong upper bound on this inner product during Phase of the population projected gradient flow. To obtain an upper bound on this inner product, we separately consider the contributions from the first coordinates of the vectors, and the remaining (d 1) coordinates in other words, the gradients with respect to w and the gradients with respect to For convenience, let grad, Lp) denote the gradient with respect to w and grad, L(p) denote the gradient with respect to z. The following lemma is about the contribution of the first coordinates of the particles to Ar, and is useful during Phase 1.
Lemma E.4 (Contribution of w's). Suppose we are the setting of Theorem and pt. be the projected gradient flow population loss, initialized with the uniform distribution , defined
69

---

in Eq. (.77). Suppose D2. < 0 and Da,t < O, where D2,t and Da.t are as defined in Section 4.1
Then, the contribution of the first coordinates to A, can be bounded by
graddd - grad (10 - w.)
(E.57)
2 4 22 (10 wx)² + 0 (D4, Da,a Wmax (wr - rTe)²)
(E.58)
+ DA.alDa: Ama ( x
(E.59)
o 02 D2 + 02 Da.el
(10+ )2
(E.60)
d
where Wmax is as defined in Definition
4.3
The constant at the beginning of the first term particularly important since it determines the growth rate of || 2.. and in turn affects the number of samples we will need. At least during Phase 1, the contribution of the first coordinates will be the dominant term in Ar, as we will see in the next few lemmas.
Proof of Lemma
E.4 The contribution of the first coordinate to the dot product defining A, is
(grdd I(p)) ((p)) grad L(x)) (wit w.))
(E.61)
2(v((a) ol (w:) (niz wit)
(F.62)
where v(w) is as defined in Lemma 4.4 We can further simplify o(wi)) - v(w;) using Lemma 4.4
Let P. and Qe be defined as in Lemma 4.4 Then,
u(ix)) v(w))) = -11 wi
(P. + Q+(())) + (1 (P (+
(F.63)
Let us upper bound the term corresponding to Qe first Since each coefficient of Qt bounded above by O Pa.al (.... (by the conclusion of Lemma 4.4 and Qi only has linear and cubic terms,
we have d
a 22.1 0Z D4.1
(1 - (1)-()) +1 (1- (10))))))111)2((mm)111111 (²?)(((m))) 110 wel
10 We 2
(E.64)
by Lemmal
1.16
v(.)))(L., we can be bounded The terms corresponding to (1)) in (o(1):)
as follows:
(-(1 - Pz (1)))) + (1- (1- w)) (w0 wi)
(E.65)
= 203 (i: - -.)² 4 Da,, (ver - w. ( + Ue + wi?)
(E.66)
- -20 D2.1 (mi + u.ي: + m²?) 40 D4. c ( ) + t. W. + m²?)
(E.67)
< 220 (D,, 22 + -6,, D (wr w.) ( + w + wi))
(Since last two terms nonnegative)
S 20 D2.e w + 0 110, + wh (w. - )
(Since W, w S 1)
Thus, the overall bound on the contribution of the first coordinates to A, is
2(v(10x) v(w))) (wit w.) 2 (-(1 ui P (1 w² P: P:(w))) (w:: w.) (E.68)
+ (1-
w Qr
-100 the
(E.69)
< 462 i2, (we
w.)² 0 D4 1 we
We )2)
(E.70)
22d D2,1
+ 02 D4.1
+ 0
(20: - (w)))
(E.71)
d
70

---

We can simplify this bound further in terms of Wmax and A max,. First observe that
10 wyr S || S max,t
(F.72)
Thus, by the triangle inequality, we have |We|| S Wmax and | S Wmax + Amxx meaning
2(v( v (()) (w0x (i) - 40 D2,1((0r w,)² + + D d0 + wel(th; )
(E.73)
2.d D2,1 + D ( - wx))²)
+0
(E.74)
d
S 402 2.d D2,1((ur w² 0 Da,t Wimax (10 - we
(E.75)
+ +0 02 D1, ( -0x)))
(E.76)
2.d D2 + 02,0 Da,1l
+
(ūs - -0,))
(E.77)
d
(E.78)
desired.
Next, we study the contribution of the last (d - 1) coordinates of the particles to Ar(x) during Phase We first introduce the following formula for the projected gradient of the population loss with respect to the last (d - 1) coordinates of a particle.
Lemma E.5. Let p be a distribution which is rotationally invariant, as in Definition 4.7 Let
grad,, v L(p) be defined in Eg. (3.4). In addition, let grad L(p) denote the last (d - 1) coordinates of g()) any particle u = (w, 2). Then, for any particle = (w, we have grad =
L(p).
I-w7 gr L())
Notice that w² + x² 1 for any u = (w,,) E Sd-1, and this equality is maintained by projected gradient descent the population loss, in Eq. (3.7 Also, by Lemma I.15 we have grad (pp) = cz for some scalar c, since p is rotationally invariant. Thus,
Proof of Lemma E.S
(w² 2²)
(F.79)
2wgrrdd in (p) grad,L(p))
(E.80)
2uggrad, ((p) 2c |||1
(E.81)
2wgrad ((p) 20(1 - w²)
(E.82)
Rearranging gives us
w
grad L(p).
(E.83)
1 w
By directly applying this formula for the last (d 1) coordinates of the projected gradient, we have the following upper bound on the contribution of the last (d (d-1) coordinates to A, during Phase 1.
Lemma E.6 (Contribution of z's During Phase 1.). Suppose we are in the setting of Theorem 3.4
Let 1. < Ti (where Ti is as defined in Defimition 4.5g. Let pt be defined uccording the population projected gradient flow as in Eq. (3.7). Let grad, (() be as in Eq. (3.4), and let grad [[(p) denote the last (d 1) coordinates of grad (p) for any particle u (w. 2). Then, we have the following upper bound:
-2(grad, ((p:) -oool-grrdaaa ((pi), 2 2
(E.84)
S 2 D2. [( + (0tl)) || uill + og.nl D1.e(( |1201) [
û ||
(E.85)
D2.1l + : L..el
Da.el
||ū, 1
(E.86)
71

---

Proof of Lemma E.
Using the formula for projected gradient in Lemma E.5
we have that
(gradgL(p))
grad (pt), & xz)
(E.87)
ri wi
We
grad L(pi) -2 1 oij
grad, L(t)) Ex, 20 21)
(E.88)
1 gradd ((p)) W.grad, 1)) L(pe)
2 T (3+ 2,) W U,grad, 11: L(pe) m
|| 2x| 2
1 r
(E.89)
The second term always negative for S. Ti, because Lemma D.9 implies that v has the same sign as Wi meaning that grad (pt) has the opposite sign. We bound the first term as
ú grad L(pi) W.grad, 1 ((pe) & (Zg - 2r)
(E.90)
1 =
w7 (P,((. + +Q((w,)) wz (Pr( Q.(().)) || - 22
w grgrad (pp)) uigrad ((pr) 1 wf
. -1| - (B.c. [del2 I || ll = 1)
ut
(By Lemma 4.4
S 20 (2, - 11 - 1112 + 40%, Da.: 1:0 - -,,
12
- 2
(E.91)
+ +1Qi Qi(ú))) 11 &
(By Lemma 4.4 and Def. of Pi)
< 26 D2.1 [// wlll - - + 46 i.d
D4,e |10 w/|l| || 2x122
(E.92)
+0 2 D2.c o. Da.e
100 vZ || - 2g l|
d
(By Lemma 1.16 1.16lnd and Def. of Q: from Lemma 4.4
$ 222 10 + 10 - a.11 + 02 Da. + 10, 1|
(E.93)
02,0l D2.el + o2 Da.el
|| - Tell2
d
(B.c.
line de $ l| uella and 12 2el12
S
|| un (l2
as desired.
We can combine Lemma E.4 and Lemma End get the following bound on At during Phase 1. Lemma E.7. Suppose we are in the setting of Theorem 3.4 Let Ti (where Ti is as defined in
Definition 4.3). Then,
A S 4 D2,1 (101 - -,, + + 0( D2,1
+ 02 Da,e Du,el)Waaxx)l 0e
(E.94)
+ o (02,a) D2,el + ,ā Da., Amux)
1 - mel12
(E.95)
+ +0 2dd D2,1 l + ,3 Da.el
02
|| uell2.
(E.96)
Proof of Lemma E.7 This directly follows from Lemma E.4 and Lemma E.6
Next, give a potentially looser bound for A, which holds even after Phase 1. The running time after Phase 1 is at most (log log d))(1) (aside from the dependency oz ås.d, etc.), meaning that even the following growth rate is enough to ensure that the coupling error grows by only a sub- pobyyoolll factor after Phase 1.
Lemma F.8. Suppose we are in the setting of Theorem 3.4 Then, for all 2 O, we have
Arr S (02.a o2,)) |ūee dcl
(E.97)
Proof of Lemma E.8 We first consider the part of A, that originates from the non-projected gradients, which is defined
A, = 2(V - (pL(p,), 3t - -m))
(E.98)
-22R [[fp(x) v(x))( (प. x) o'a,, (T], x - a, x)).
(E.99)
72

---

By the Cauchy-Schward inequality, we have
|A A gg ((fp(z)= - )(z))² Ex-sy Bense->((o' (ū, a) o' d'ū 2 ú x)
(E.100)
We bound the first factor in the equation above by the loss at initialization, which
2 Re-ga- [[ y) S (6 + 2.,))
(E.101)
by Lemma D.8 Thus, plugging Eq. (01101) and Eq. (E.210) (from the statement of Lemma E.
into Eq. (E. 100 gives us
A E (02,d + 22 116.11
(E.102)
Next, we bound the difference between A, and A We have
4 - A = - (grad, (p::) - L(pt) ar ú.) + (VaL(p.) - v.l(p)), & - ū.)
(E.103)
(P)((p)) grad; ūe ūe) - (VaL(p)) - grad, (p:c), 01 ū.)
(E.104)
For the first term of Eq. (F.104), we have
(Val(p)) ((ddgL(p)))) dr Tz)
(ūrâ, V Vel(pe), 0r Ue)
(E.105)
-(VaL(pt); (ū, 0 )d, (Ueli )ue)
(E. 106)
(Vol(pt). (t) -(a,, ūth)
(E.107)
=(Vol(p)), us) | 1el1
(B.c.
| |
|| 1)
2
||
rell (E.108)
E 1Kff (1) - -y(x)) (0. x 2)a, z~
2
||
E (z) )()))o (á, x 2)0,at x] 2~
(E.109)
and by an argument similar to the one used to bound A/, (here using Eq. E.211
from Lemma E.15
instead of Eq. (E.210). ) we can thus show that
(Ve)(pe) grad L(pr), , ir u.)ي S (6 + + 1|
(E.110)
Similarly, for the second term have
(Val(p)) grad, (ptt), 0. u,)l S (0 22 110.112
(E.111)
Combining Eq.
E..IO.. Eq. (E.III and Eq. (E.102) finishes the proof.
E.2
Upper Bound on B,
The goal of this section is to obtain an upper bound on
Bx(x) == ([grrdd,((p)) grad, de(x) ox())).
(E.112)
When there no risk of confusion, we simply use Bt, ût and ūt omitting the x dependency. In the next lemma, we first obtain bound on (x). and after that, we obtain a bound on [x (X)( The proof of the following lemma mostly straightforward calculation, and follows mostly from Lemma 5.1 and the definition of A.
Lemma F.9. Suppose are in the setting of Theorem 3.4 Suppose T oo, sorany any absolute
constant C. Assume the width m of P. P is at most ( d) Then, with probability at least 1 - e -d we have
(6% + 02 dd ()
E -
(E.113)
for all times t E [0, T). Additionally, with probability at least - e d² for all t E 10, T and x E po we have
B,() S (02.a + (dllog 011)
116116112-
(E.114)
73

---

Proof of Lemma E.9 We first decompose B, into two terms:
B. 2. - grad, [(tt); de - ū+) --2((T - ". VaI(pt) - (I - u (a (pte), ūt - (i))
(E.115)
(E.116)
-2(Va - VaI(pt). (I - u.ú,, (dn ū.))
(E.117)
-2(V VaL(pi): â de) + +2. apv) VaI(pe), d,d, (ā: (z))
(E.118)
-(((Val(Ae) - VaI(pt), àn - ū+) + +((Va((()) - Vat(pt): (1) (1 (We, ie))
(E.119)
(() - VaL(p:). i - + - VaL(p)), ūt) || ūt l
(E.120)
2
since | ll2 = || l|2 = 1. For convenience, (ûq. ūz) where the last equality is because 1 1 11 define & = 0r Ug. Then, we have
B, - ((V.(()) VaL(p:), (V,L(A)) V.L(p,), úx) 116:112
(E.121)
We first focus on the first term in Eq. (E.12I), which is the main contributor to B,
Analysis of First Term in Eq. E. 1.1. For convenience, we refer to the first term in Eq.
E.121 as
By. Observe that we can expand Bi
B = V2 (())) (pL( 6,)
(E.
- [1/(x)) fo(x)) (6),,
(By Eq. (3.4)
2 E [(/(x)) (p(a))) ((0)), (f( (:) - E (1/(()))))))))) - (s(a))o )) (6r, a))
2-g8 1
x~gd-1
(E. 123)
Let us bound the first term of Eq. E21 By Lemma 5.1 with probability e- for all S T we have
(02.) + 02 o2 (log (1)(1)
2~5411 2
(E.124)
m
Thus, we can bound the first term of Eq.
12. by
E [[[[(1))
((i) (6,,,
(E.125)
2-/Bd
E aadi
[[f((z) (x))))11112
E
x))
aro
(By Sccwaa-Schwarz inequality)
E [(f(()) - (1) + o%, 1/2 6.
(By Lemma E.15 Eq. F..2
2g d(log 0(1)) 0(1)
(02 + +g,,) 18:12
(By Eq. E. 24
Vm
02,a) d(log d)
2.
|
(E. 126)
Vm
Next, bound the second term in Eq. have
123
By applying the Cauchy-Schwarz Schwarz inequality,
[(fo(x) -
(E.127)
<
0- [1f(()) - 1112
angad [o'((i x)) (6,, x) 2/1/
(E.128)
$ (02,0 + )/
E ja ((ū,, m)) (5z, 2) 211/22
(By Lemma 5.1
2~ga
x (02 + + 1/2 (og.a + )3,4) 1/2 18
(By Lemma E.15 (Eq.
(2.11
$ +0
(2.)
74

---

Combining Eq. E2.16 and Eq. (E.129) with Eq. (E.123). we find that
(
d (1)) m + 11)116612.
[ < (02 o2
(E.130)
Analysis of Second Term in Eq.
E.121 For convenience, we write this term as B i.e. we define 1 suffices show that the first product in the definition of B3 is at most (02 a 02,0) up constant factors. To start, we have
B" := (V))( Va (p.), ūz)
(VaL(A) (.L(
E
[(fp(z) fo(x)) ((0e, x) (0,,
(E.131)
E ((f(()) fo(x))) () (ur,
(E.132)
E ال-ي"_1 1[f fa(x)) ((0e 2) (0,, m)
(E.133)
Both terms in Eq. EE.13 1.333) can be bounded as before. For the first term we use Eq. E. 1214
E (Vo(z) - fa(x)) (((t, (ūt, gogd
(F.134)
s E [([p(x)) - fo(x))))1/2
[o' (o'(( (a,, 2//2
zn_a
z
(By Cauchy-Schwand Inequality)
d(log d(logd)o1 d)
2 (62 + (2,,))
(By Eq. (E.124) and Lemma E. Eq. E.21
m
For the second term, we use Lemma E.16
E [(/b(z) -
(E.135)
2~88
2n z2gd-, (0,1([())))))))))))))))){()))( - 11(x)))3112
E
E 2)) 2j]
2~84-1
(By Cauchy- Schwwaz Inequality)
S (0 + 02 a
(By Lemma E. 16 and Lemma E.15 Eq. (E21
Thus, we can conclude that
B S (62,a+ + +G (dllog o)1)
+ A.) ||ō, 12
(E.136)
Overall Bound on Bi, Combining our bounds on B' and BV, and using the fact that ||ó:|12 S 2 to absorb the bound for B into the bound for B/, we find that
BB $ (62.0) + oq.a) dd(log d) (d(logd)o()))
+ A, |
(E.137)
Vm
with probability 1 - dd (recall that Eq. E. 124 held with probability 1- e This completes the proof.
Next, show tighter bound when we take the expectation of B, over the neurons, i.e. over X. Taking the expectation allows us to obtain a tighter bound because of the term
2
E [[fx(z) fo(x))o ((ū)), <6,, 2-
138)
For single ün the best bound we can obtain by using the Cauchy-Schware Schwarz inequality. However, the key point in the following lemma is that when we take the expectation over (û,, T. ~ P,, this term becomes
-2 ((((())
(E. 139)
20
75

---

together with third-order term which is most (0 + +3,,) which is significantly better bound than (02 a.d + 1,a) A,ll61l12 which we obtained in the previous lemma. This is because to simplify Eq. E.138). we can use
o ((ūe, x)) - a ((ūe, x)) + o(((e, - ūi, x) + ((0"(((e,
(E. 140)
meaning that we can essentially substitute o' o( x)) (6,, x)) zs a ((t,, x)) - ((T,,
.r) ) in Eq. (F.138)
Note that we need this better bound on the growth rate of D, since our inductive hypothesis imposes stricter condition on A.
Lemma E.10. Suppose we are in the setting of Lemma E.9 and in particular, suppose T satisfies the
same assumptions as in the statement of Lemma E.9 Then, for all S T, we have
(6 + 0 T,a
E X
[B(2)
D. + (0 + o
(E.141)
m
We again make use of Eq. previous lemma that B. B, + B,,, where
Proof of Lemma E. 10
E. E. and we recall from the proof of the
B 2 E [(f(()) fo(x)) ((ue ()),, Zo
-
E ((f(()) fa(x)) (( x) (6, x)
an
(E.142)
and
B = (V,L(A) - V. ((,),, (z) 118.112.
(E. (E.143)
The main difference between Lemma By Taylor's theorem, we can write
E.9 is that we now bound the second term of B; expectation.
((ūr, x)) or ((ūu, x)) + o' ((),, ((x, x)) + o"()) ((i,,2) (ūr, x))2
(E.144)
for some which is convex combination of (û, .)) and (ū, x). Rearranging gives
o' x(), x) = o' d'(,,, x)) - ūe, x) = ((úe, x)) - a ((ūt, + o"())
x), (E.145)
Thus, taking the expectation of the second term of B; over (û,, ūe) ~ T gives
-
E
E
[(f((z)
(E.146)
= = E
- -((t,,)))))
(F. 147)
(,,a)-,, a
E
E z-80 [(fa(z) - fa(z))o
(By Eq. (E.14))
(a,ū)~I,
2 E
(ff(zz) fp(x))
(E.148)
a
E
E
[[)(((>>> -
(By Eq. (3.1)
znga_,
E [V)(())) (o(())-(((()))) (A))))"(A)((,..)))))
(F.149)
( 8-
Next, let us bound the inner expectation in Eq.
E.. 14.). By the Cauchy-Schwarz inequality, we have
E [[fa(x) fa(=)))"
(E.150)
E [_,[[f((x= (/)
E (A,)
(E.151)
z-ga-1
2
$ (0 d + 2/2
E
(1) 4/1/2
(By Lemma E.
2ngd-1
< (62 + )3,,) 1/2T,
10" (o"((,,, x)) 2 x))
sj1/2)
(E.152)
76

---

Here to obtain the last inequality, we observe that o" is convex because P2,d" is constant function and Pa.d" is a quadratic function by Eq. (C.4 - additionally, we used the inequality (a+b)² 2 +² and the fact that À convex combination of (ūe, and (U,,
By an argument similar to the proof of Lemma E.15 E Eg-ge-1 (o0 2 x) 4)1/2 S (02,a + (2/2))/21 and E Ex-ga-i (" 2))
we
can show that
(6 + 02,d 1/2 l Combining this with Eq. E..4 we have that
E aa
[(fa()) -f(())f(o()))))))) ((0,,) x)) 2), 2 (02,d + 0,,,11
(E.153)
and thus,
2 E E ((f((=))f((x)))) (a,0)~I, z-84-1
(6,, 2 2 (0 + a,
(E.154)
by the definition of A. This gives a bound on the second term of By Combining this with Eq. bound the first term of By, we have
EE.166
(02 + R. d(log ())
E (Q,0)~I+
[B]] x
E
| + (6 +,,,)
(E.155)
Vm
(f,0)~T.
(02 + of dlog d) 01)
A. + (0 oZ
(By Cauchy-Schwarz)
Vm
Combining this with Eq. (3616)) (for which we also take the expectation over (ũ,, Te) ~ lt), we have that
(dloogg)
(jol + +.)) A
E BB (02 6%,)) (d,a)~T,
(E.156)
Vm
which is at most the bound that we have for By, and thus,
(62 + 0 d(log d)
E (4.a)Tr
[Bo]
A. + (6 d + + 1
(E.157)
Vm
as desired.
E.3 Upper Bound on C.
The goal of this section is to obtain an upper bound on
Cx(x) 2gradd, ((p)) grad, L(p),, ix(x) - xx(x))
1E.158)
This quantity is essentially the part of the growth of [|o,11 which is due to the difference between the gradient using finite samples and the gradient using infinitely many samples. When there no risk of confusion, we simply use Ct. ûe, and ūn omitting the x dependency. We bound this term with the following lemma:
Lemma F.11. Suppose we are the setting of Theorem 3.4 Assume that the number of samples J; is n. S d ffrr any universal constant C >> O. Assume that the width m S d for any universal constant C > 0. Let > O. Assume for all E [0,T] we have a bound B₁ on As with B S V Additionally, assume for all [0, that we have a bound B2 on Amax,t such that B2 Vd Then. with probability 1 (لाíin . all t [0, and x gd-1, have
|Cx(x) 2 (log d) o()), (11 d B
G2.5
d¹ 3
A B2lloll2t
T
(E. 159)
where 8, ūx(x) Tx(x).
Proof. We bound C, by expanding Lrad,L(A.) as the sum of several monomials, as well as expanding grad,L(pp) L(pc), and then applying Lemma 1.7 to each term from (p::) and the corresponding term
77

---

from grad, (px). 2.). For convenience, we use (p) to denote the loss of fa on a single data point E Sd-1 and grad, (p) to denote the projected gradient of (p) with respect to ü. We have
(grad, (p:). ûz ūz} = ((fa(x)) - ((u), )x, (T 0 ú ((+ - -m))
(E.160)
= =(fo(x)) - u(z))o ((ue, , 2) xx ūz Ut)
(E.161)
- -(lfo(a) - y(z))o x)),, ūt)) (1 (āc: út))
(E.162)
(((((t)) - v(x))o' x) )r, dn ne)
(E.163)
- ((ff(x)) y (x)))o' a)), u.) 1 üe 1
(E.164)
For convenience, define 6, == ûn Ty. Then we can rewrite the above as (grad, (10) (pc). 52) = ((f((z)) v(x))) 2) (x,
(E.165)
1| - ((Vo(z) - v(x))oo' ((āe, 2))x,, â.)
(E. 166)
2
For illustration, define C(g) as
C(g) = Σ
g(.,)
E 2m
[g(z)]
(E.167)
i.e. how close the empirical mean of the g(ar)) is its expected value. We will show that C(g) is small when g corresponds to each of the terms on the right-hand side of Eq. E.165 even when we take the supremum over the úe, which may depend on the z4.
Concentration for Mean of First Term in Eq. (6.165). As first step, we list all of the terms we obtain when expanding the first term on the right-hand side of Eq. (E. The a a'(iū, x)) factor consists of the following terms: (1) a Ō2,d d V (2, x) term, (2) a da,''v N4,a(uu, x)² term,
and (3) a d /Nadd (āe, x) term - here, we have stated these terms up to absolute constant factors for convenience. In the fø(x) - y(x) factor, fĄ(x) contributes (1) 02 NN, x)² term, (2)
a Ō / NN, term, and (3) a @s.a/No.d (i// x²) term. Here note that i is not a single vector, but ranges over the entire support of أ.- Thus implicitly perform union bound over the m vectors in the support of A - this does not affect the proof since the failure probability of Lemma 1.7 is | while we assume the width m is at most dC where C can be any absolute constant. Additionally, y(x) consists of the following terms: (1) 02,d 12..d N2,a(e1, x)) term, (2) a
04.d% / (e, 2x term, and (3) @s.anyy / Na.d (c1, )²² term. Finally, there a factor of (be,
d
For convenience, throughout this proof, let 8, = T In each of the above factors, for every occurrence of <). we substitute û, = ū, + 6, and further expand. This is useful since originally ân depends on the T, and thus, when we applied Lemma [.7 [.7we we would have to take the supremum over - however, now ū, can be considered a fixed vector with respect to the æ;, and while we need to take the supremum over o,, in each term where ó, occurs, we obtain an additional factor of ||6e||2. which reduces the error from uniform convergence. We now completely expand the first term on the right-hand side of Eq. 165 First we consider the contribution to C(g) of fa(x) and later study v(x). Up to constant factors, it suffices to obtain an upper bound on the concentration of
Σ
82 m d plligirie (5e, 2)p+1 (5,, x5), x) (14, x))
(E.168)
9+.=2
where we have let dmax max( 02.d). (ōs.))). Here, the sum ranges over p-erp+q= 1, and t 2, 4 since the 6, and ū, terms are due to the o' ((0, 2)) factor, while the & and i. terms are due to the fa(w) factors.
We now obtain concentration bounds for each of the terms in Eq. E. 168p. First, we consider the case where p + q 2 1, in this case, Lemma L. can be applied (since can choose PL, P2, P3, PA
78

---

in Lemma L.7 so that = 8, and + 1, and U2 = 8 and P2 = and P1 0 the
hypotheses are then satisfied since p1 + P2 + Ps P4 2 2). From Lemma we obtain the bound
22 max
(8r, x) (6 x) (ūe, (T),
IE mo
K6 2) (8, (Te, ()) (T/, x>*]
(E.169)
(V d
(Wa n
113 n
2
max (log droml)
|16 1 116:11
(E.170)
Here in applying Lemmal I.7 we have assumed that the number of samples is at most dC for some absolute constant C, meaning that the gotlog term in the conclusion is a lower-order term. Lastly,
we consider the case where 1+g 0. To apply Lemma L.7 we can let ūt P1 = L, U2 = ||..16.
6.
P2 1, and let P3 - P4 with w ür QI = r
and w2 1h Q2 8 since p1 + P2
the hypotheses of Lemma 1.7 are still satisfied. This leads to us obtaining only one power of | l in the bound for this term: if q then we have the bound
a
(6,, x) p+1 18,,118,, (we, 2²" (1h, x)
E xr S Ki
x) x) p+ (8% x) x)" (1),
+>*]
max
(E.171)
(V= + () |öell2
o (log d) o)1
(E.172)
Summing over the possible values of p. q, 8, find that the overall contribution to the concentration error of the term with fo is, up to logarithmic factors and a 62 factor,
2111
E 8'~p
Σ P.g.r.
(
11
18.112 +
1
188113
(E. 173)
n
(V2
d
E
Σ
1 1
|
11
||8:11
(E.174)
8'~p
n
72
P53,054
10111
E +
1
(E.175)
8m
PS3.954
d
pigtt
2
E
|
116%111 +
(E.176)
8'm
n
P53.954
Σ
E 8'mup
(E.177)
Y2
PS3.954
Finally, in order to simplify this bound, we isolate the dominant terms among the terms of the form ptg+i
|| Ex- |10% 12. with the following cases. First, consider all the terms where = 0.
For S we can bound these terms by " 11/122 and for q > 2 we can bound these terms
انواع A² B3 31 Thus, of these terms, the term with g is dominant, and n
by contributes
d
12
(E.178)
Next, consider the case where p 2 1. In this case, since the bound Bg on Amax, is greater than the term with p = 3 and : = 4 dominates, and it contributes
1
21/z.
dz
dt
B B3
B 11be11
(E.179)
n
n
However, we must lastly consider the case where p > 1 and q << 2 - the bound on the concentration
error may be larger for = 1 than q 2 since A < Vd' In this case, since B, > Va'
1 we must
79

---

consider the =39 1 and 3, terms, which respectively contribute
d
B 1
(E.180)
and
d²
B
16
(E.181)
n
Note that the p 3, 0 term dominates since our bound on A is less than V meaning we can ignore the p 3, q term. Thus our overall bound on the concentration error originating from the first term in Eq. 1.. is
max (lo d) 01))
(v 1116:112+
& T
R -A BB
B2 16/6/112
-BA 160
(E.
T
To deal with the terms originating from y(z), we observe that we can simply treat (e1, in the same way as we deal with (ūr, z), and thus the terms originating from y(x:) do not make any new contributions to the concentration enror.
Concentration for Mean of Second Term in Eq. (6.165). Finally, to deal with the second term on the right-hand side of Eq. (65), we note that we can follow the same proof, but instead obtaining an upper bound on the concentration of
1
Σ
2 max d
x) (ū,, x) (u,, x)"
(E.183)
and as before, the contribution of the terms originating from y(x) most the contribution of the
terms in Eq. 183 Due the additional | factor, can in fact ignore the contribution of the
terms Eq. E. 183 as the dominant terms will be less than the dominant terms from Eq. (68))8).
Concluding to Bound Ct. summary, with probability at least dellog for choices 8,, 8,. we have
[C(g) i o (log d)o0). (V 110el2 + & B3
2 A
+ da B 16.1
18.444)
where g(x) = (grad, (p)),, 5z). Since Ce is exactly equal to
R
g(x))
z-ga,, __,_ [o(x)]
E
(E.185)
up to signs and constant factors, we therefore have
(V 11eell2 + d²
d
d'
|Crl & 02 (log d))
A
B
+
BBA
110.13
n
n
n
(E.186)
E,4 Additional Lemmas
Lemma E.12 (Uniform Bound on a and d'). For any t [-1, 1], we have (o(t) S (62.al / /N2.d [ōa.al / Na.d and additionally, (a'(t) S (/2.dl Ne.d + l NN.d. Finally, we have lo"(t) S
Jō2aly Na,d + / /Nadd
Proof of Lemma E.12 The first statement is by the definition of a, and because (P2,(()), (Pa,a(() < for any 1 E 1-,, by Atkinson and Han [2]. Eq. (2.116). The second and third statements are by the definition of and using Eq. (C.4.
80

---

Lemma E.13 (Error from Finite Width), Let be distribution on S and let p be a distribution on sd of support at most m. where the elements of the support of P are sampled i..d. from p. Additionally, let > O. Then, for some absolute constant C > 0,
E (ff(z)) f + + log (log d)
6 +
(E.187)
d d)
m
with probability at least 1 5, where So and f& are defined as in Eq. (3.
Proof of Lemma E.13 We can write fĄ(2) Euruport ( x)]] and fz(x) 1
E
Now, let B E (0, 1), which we will choose appropriately later, and define the function ς [-1,1] [-1,1] by
m
t
t t
(B(()
B
t> B
(E.188)
-B
t t<-B
Additionally, define fo(²) Eu- (p (( ((u, x]]]], and define fz(x) o Let us analyze the error incurred from replacing So be Fo and fp by fg. Observe that
E Trr-ga- [(f - - f((x))
E (f)(z))² x- fo(
+ (fox)))))) (E.189)
E angel1
We can simplify the second term on the right- side by expanding fo and fp:
angaa- fo(x)fo(x)= =
((1,,))))(
(E.190)
anddd-1
ui Mamp
E
E ()) x)),
(E.191)
To simplify the inner expectation, we use Proposition L.5 to find that yp (p (2), (ug, with probability at least 1 - 2 2exp(- -. Thus,
E ar8d
(Dala))²
{x(x){((x))
(E.192)
angd
E
jo((un ( x))) - ((uu,
(E.193)
u1 12ppppp-e 2-81111
(122dl V No.d | NA
E
E
( - of ((),
ugmup 8-1-1
(By Lemma 12
(02 + N1, 2 exp (- )-
(By Proposition L.5 and Lemma 12
2
Similarly, show that
(-Bd))
B2d
8 2~gd-1
E (f(x)))))²
-
E (F)())) S <exp {
2
Ne. + Na,)
(E.194)
and therefore,
(- B&d
E F.()())) 2dd
(62 +
(E.195)
2
By the same argument, we can show that
E 2~ 3- - exp exp (- B2d
B²d)
(E.196)
2
Therefore, it suffices to show that En-gg-i [(f(x)) f (z) is small with high probability over W1,..., um- Observe that for | < B, we have (P2 & B² + O(1dd), and [Pa,a(t) S BA +
O(B² /d) + O(1/d)). Thus, if we choose B 2 1/vd, then for |d| < B we have [P2a(11) S B and
[Pa,((t) < B In particular, we have
(a()) & 102,ll 2 + 1 NA,.
(E.
81

---

Now, letting E g and using Hoeffding's 's inequality, we find that if ui, My are i.i.d. samples from p. then
Σ
(PB((u).)) x)) - -E [o((p st
(E.198)
ump
with probability at least 1 - exp Na, a a for some universal constant C. Now, by Corollary 4.2.13 of Vershynin (74), for any > O, there exists W C gd-1 of size at most (
Cmt
such that for any x E Sd-1, there exists E Sd-1 such that || - ull2 S y. Thus, with probability at for all E N, we have
least 1 - 2 c
Cmt² 10 Na. RB
exp
7 Na
(pn((())
E (o((sn(( ump
w))) SL.
19.199)
If Eq. (9.199) holds for any y N, then for any E g since lo'(t) S 1ō2,al v N + J / Na.d for any E [-1,1] (see Lemma E.12), if we let y E N such that || yll S y, then
we have
((
E (o((p(((,, s++ (102,al v /2. + 10 Nu,ay. (E.200)
ump
In summary, with probability at least - 2 2 5 d [[F (x) fx())) < 1 (02 N2,1 + Na.a)?
Cmtta exp N2 N2. B00.B No.8
we have
(F.201)
Combining Eq.
E.201 Eq. EE.15 and Eq. E..9.). we have that
[(fp(z) f2(xr)) S 12 + (0 N2 + oe.d N
(E.202)
+ exp (- (-Ba1)
Bdd (62 + Na..)
(E.203)
with probability at least 2 2 exp
Cint?
Choosing B = C' (log d)² for va
aB Na,
sufficiently large universal constant C², and using the fact that N2 S d2 and No.d S d' (by Eq. 10) of Atkinson and Han 12] we have that
22 + o (02, + oa
E Jt
fz(x))
(E.204)
(log d)
with probability at least 1 - 2 ( ()) exp (- (- 2 (6
mif
We can rearrange the failure
probability to find that, up to constant factors, it is most exp
Finally, choosing 7 ny ay and choosing so that the failure probability is at most we obtain (-Ω
ml?
+ dlog 9)
(0 +0 (log dj
E ((6(x)) - l (0 + o (log d) 16 (d E + log +)
0 ad o2
7
d d)
(E.205)
(63,a + 02 (log d) 16 (d² log d + log
0 + 03.0
+
d d)
(B.c. 7
with probability at least 1-8. 8, as desired.
Lemma E.14 (Bound on Gradient). Let E g and p be distribution with support on $d-1 In addition, let (p)( and grad, L(p) be defined as in Eq. (.4). Then, we have [.,((p)]22 Z (63 + 02 o2,2)d
d and [igrad,, (())]2 (02 + + dd".
Similarly, let ((p()) and grad, [(p be defined as in Eq. (02.a + 0 dd and LL & (0% + ddd
3.5 Then, we have | [V.((p)]22 x
82

---

Proof of Lemma E. in Sd-1:
First, we will bound (.L(p) for any u E S and distribution p with support
Molll2 = [()((a) (( 211)
(F.206)
R z~ B (5(()) v( ((,,)))(
(F.207)
By Lemma E.12 for any u, E 8d-1, we have (w, & 1 N2.1+ Na Addition-
ally, by Eq. and by the definition of y(x)), we have (,((z) (2) - / 00 /Naad
by Lemma Thus,
[..((p)l2 x N No S (6 ad
(E.208)
because N2,d S d and Na.d S dt by Eq. (2.10) of Atkinson and Han 12]. By the same argument, for any E gd-1
I uu V (p)lll S +
(E.209)
because un T is an orthogonal projection matrix. This completes the proof - the proof of the analogous statements for (p)) and grad, ((p) follow from the same arguments.
Lemma F.15. Let u, u' E Ed--1 Then, we have
E 2-
B_ll(o' [(oo((l ((TT)) x) d'(''' 2)))((² u²xx) (62,a + +3,a))1 u'll2
(E.210)
and
wm.gd [o'("" 2) x) (0 d + o..))
(E.211)
Proof of Lemmna E.15 For convenience, let 8 ;= u' - u. Then, we have
E znga-1
1(0'(( (u x) o' (u - xT
(E.212)
S 22 N2d angd
[P2dd (u x) - P2.d' Pa.'(ul' x)) x u²= x)²1
(E.213)
+ Na Na.
ao (Pad (Pad'(uT= _ x) Pad'(ul (TTT))))(( x1)
u¹ x)²
(By definition of a and Pa.d and b.c. (a + b)2 x a² + b²)
$ 3 N2,
[6)x))11 1 No.
wwgd [((w)) Jo)3 - (u (=))))(((n) To TT x)
angd
(E.214)
62
4,0 N4
Ro [uu x u x]
(By Eq.
C.4
and expanding the term with Pa.a')
S 00.00/1 N2,
1(0 x))] + Na. Σ
- [u T x) PP) (T (6 x)4
E
angd
p+g=2 2
(E.215)
Na.d
+
1(TTx))
(E.216)
d²
Here the last inequality is due to the identity a² - 13 (a b)(a² + ab + b²). which gives ((TTTx)) 2z)33 - (mTz)) x)))² = -(n² x²)² + (w x)(u Tx) + (u 21)) x u uT 2)²² - expanding the first factor fully gives terms of the form (u² (TTx))((u z)2 where and q are both even and p 4.
Thus, applying Lemma [.17to the final expression above gives
22 E,,llo' x) d' d'l'' x))) (u n" )
E
(E.217)
2d Na.d
|101 + Na. Σ S N2. d²
1
1101
|
d²
da
カto=2
(By Lemma
I.17
5 5 + 33.00011
(B.c. N2, S d² and N4. S d' (see Equation 2.10 of Atkinson and Han
12
83

---

which completes the proof of the first statement. The proof of the second statement is similar we expand using the definition of a and then apply Lemmal 1.17 each the terms. We omit the details of the calculation.
n
Lemma E.16. Let p and of be two distributions on S which both have support size m < oc, and consider a coupling (u, u') (p,p') of the two distributions. Additionally, let So and fo be defined as in Eq. 3.1
Then,
f (62, + 62,a)
E (u,"')))
||u v112
(E.218)
Proof of Lemma
16
We write the left-hand side
E
(0f()) (x())) xmgee-1
E (( (uw))an (o((u,x)))
oo''' ([}[,, ((1)))) -
(E.219)
E (o(( - o(('u(
(E.220)
E (u,"")~(ppp) xm81-11
E
(o((u,))) (( (1,'u,,z) .n)
(E.221)
For any u, w' E Sd-1 and a E gd-1, by the mean-value theorem, there exists À which is a convex combination of (u,ar) and (u', 2)) such that
a((u a)) ((u', , x)) o' ('))((u, (u',,))
(E.222)
Using a similar argument as in the proof of Lemma E.15 we can show that
E ---8111
d'()) - (u', 22) 23) S (0 + a - v'112
(E.223)
The proof proceeds by expanding (A)) into each of its terms and using the inequality |a + 6/3 x |a| + [6]$ (which holds for any two real numbers a, b) in order to consider only terms which involve one of or u' the resulting terms are of the same form as those which appear in Eq. (E.216). We omit the details of the calculation. Thus,
z~ z~ga (0,()) -
E (w.')~((p-P) pagd-1 (o((u, ((),x))) - - ( $ (02.a + )7,5)
(E.224)
(une)~(mp)) I|ue 1lu
(E.225)
as desired.
Proof of Theorem 3.5
In this section, build on our analysis from Appendix to show that projected gradient descent with (ol learning rate can achieve low population loss. In the rest of the section, for convenience, we define ò, Ue un., and omit when it is clear from context.
Lemma F.1. Let p be two distributions gd-1, and let I be coupling between and p'. Additionally, let u1, U2, v E gd-1 Then, we have
[|u u' x/121] 1/2 l wellz). (F.1)
V., L(p) - -- L(y)(p)) 1l u (62. 22,0) E uNNT Proof. We can expand the left-hand side as
E
(fo(z) - v(x))oo((u -) -(a x) x)
(F2)
z-84-1
We first obtain bound on
E Zo
[f)()) 2)(u- x) for (x) (u2 x) .x)]
(F33
84

---

By the definition of fp. we can rewrite Eq.
F.3 as
I
E
[u4 arjo (un x)(v x) o a))o x)
x])
(F.4)
a
By Eq. (..4. is even fourth-degres polynomial and a' is an odd third-degree polynomial. Thus, the inner expectation can be written as a linear combination a constant number terms of the form
2~884-1 (u4 2)(2u - 2) (v x) (ul2 (u2 x)²(v =)]
(F55
where i = 0, 2,4 4 and j = 1.3. We can also estimate the coefficient which accompanies each P2.(P,, + & that N2.d Ocd²) and Na, = ((d") by Equation (2.10) of Atkinson and Han [2]. and that the 7tht degree term of P2.d has coefficient which ( and the 1hh degree term of Pa.d has a coefficient which ( From these facts it follows that the 2th degree term of P2.d = N2 P2.d has coefficient which is O(d'/2) and the jth degree term of Pa.a = VNa,al'aa has a coefficient which is (dd//2). Thus, each term of the form given in Eq. (F.5) has a coefficient which is at most (03 + )3. (d(+++1)/ in absolute value. We can now upper bound each of the terms of the form given in Eq. E.5) as follows:
individual term Eq. (.5). We first recall the facts that a(t)
E
(ul
(11 xr) x) (ul2 (22 xV(v m)]
(F.6)
[(u) x)²² (ul2 x)² (n) z) #)]
E
(F.7)
+ zm
E
(u/z 2)²² (m) 2) (2z 2)) (v x]]
(F.8)
[(1n) - ub6) 2 (u) (ui x) (uls 2 (u1 x) (v x)
E
(F.9)
1m
(uly x)) (wu u2) w (ui = 2)--1 (v x))
E
k=U
(F.10)
1-1
j
u2112
|lu U2l12 -
km
(By Lemma
I.17
M/ w/2
||u uall2 dii+++1y/2
(F.11)
1o
Since the coefficient of this term is at most (6 + 6% ad in absolute value, this term contributes at most
(o2.a + + [lus - ublle + llur - uellz)
(F.12)
Thus, taking the expectation over u', 21² ~ I, we have
E --
[f(()) 2) x)- (z) x) x)
(F.13)
S (0 32,
[|uu - u'1l2
||u u2lla)
(F.14)
u,a'~
1/2
(0 02,a)
E
||u 2'(13a
||
uelle
u,a""
(F.15)
A similar but simpler argument can be used to bound
E
(( (un (v x) y(z)o'(u2 2)(u-
(F.16)
Z
85

---

by expanding y(z) and (z)). Thus, we obtain
Vu, L(p) - v
Vus [(p) 2l S (0 + o7,0)
E uaiar
[Iu u'
1/2 + 1lu u2ll2)
(F.17)
as desired.
Lemma F.2. Suppose we are in the setting of Theorem 3.5 Assume that n 2 d8 (logd) d) for
sufficiently large universal constant > O. Additionally, assume that maxx [|6:(x)la va Then, we have
(grada, [(pp) grad [(Pnt)) del S (02,a - 22 max (1)x
(F.18)
Proof. We can write the left-hand side of Eq. F.18) as
(grad,, ui [(pp)
grad Ryg [(p:)) -8,
(F.19)
(J,,(z))) - u(z;))o' v.) - (on( - ((zi))oo((nn æi) az
x:))
8e.
(F.20)
We proceed by first showing using Lemma L.7 that we do not incur much error when we replace the empirical average over the by an expectation over ~ gd-1, We can separate the right hand side of Eq. (F.19) into two parts:
S
(a.())) (ūe- xxi) - Sonn
x.)) (ōr xi)
(F211)
and
S2 ((xz)) (ūe x4.) - v(zx)o 21)) (8: vz))
(F.22)
1=1
We wish to show that Si and S2 are very close to
Mi = E 2-ga-1
(x)o (ūz x)) fr (I)) (āig x)) (6: x))
(F.23)
and
(u(x:)) -.. x.) - ylz;)o (ūgp zi) (6: zi)) -
M2 =
(F.24)
respectively, with high probability - then, will obtain bounds on Mi and M2. We will show that S and Mj are close the proof that S2 and M2 are close is similar.
Expanding into Monomials Our proof strategy is expand the following quantity, for x e sd-1,
G(x) = (r)o (ū, x) - Jan. (x)o (x)o'(ūn 2))
(6.-2) .x)
(F225)
and obtain concentration bounds for each of the terms. In this proof, let denote the natural coupling between p. P. which is the distribution over triples (ū,z (x), (x)())), where is sampled from (xI..... Xm). Then, we can write G(x) as
G(x)= = & By M4 r lotal x) (ū, x) ail x))' (f xx) (0, (0.- x)
(F.26)
For convenience, let us define
g(x) (a(')) 2) x)) - a lihe x) (unet 2)) (5. x)
(F.27)
Clearly we have G(x) = "f [1)]] Now, we consider the effect of expanding g(x) by expanding and o' into monomials that involve (öt æ), (ūe x), (ilee x) and (ūns x). Since
86

---

is an even polynomial of degree 4 and o' is an odd polynomial of degree 3. we can write g(x) as linear combination of terms of the following form:
(Thh ( (ūe- x²²² (i/ x)" (Uzt =z)²² =x))a)(( x)
(F-28)
where i = 0,2,4 4 and is = 1,3 - each such term also has a constant factor of O /N /Nzza). We now consider the contribution of Eq. (F.28 for each value of i, dg. In the case where 11 = O. we can rewrite Eq. (F.28) as (8, . z)² when in = 1, and
(āo. x)² + (ūx x)) (ūgz xx) + (ūgz x))) (ōe xx2
(F.29)
when d2 = 3. In the case where 11 and l are both nonzero, letting 8; = ii, - ugyt we can write Eq. (F28
(Th x (ūe 2)²² (ihgs =)" (Unt x)²²² (8e x)
(F.30)
= ((0b x)²² (ilye z)th (wt x)²² (8: x) + (iine 2))² (ūx - 2))²² (ūnt x) (ō+ x)
(F.31)
(F.32)
Σ
(+) x x)) (ite thet (ū x) .x)
(F.33)
1 20.
+
Σ
(i/re . x))x 2)d0 (ānp 2)) (ūe x - unp 20)6 .z)
(F:34)
is 111
10,422
Σ
(i)
x) (tilee x)²² (ūe x)²² (8. x)) .x)
(F.35)
18,4420
Σ
(U/r 1)) (āg - 27/0 (agp 2))² (3, 2²²²
(F.36)
is 444-- 4,420
In summary, Eq. (F28) can be written linear combination of terms of the form
(ū) (ant x)' (8, ()) x)
(F.37)
where i3 + 14 = 1,3 and ig = 1,3, and terms of the form
(il, x)th (ūs 2 (ūnz x) (ō. xx2
(F.38)
where 1 = and is + i4 0, 2.
Contribution of Eq.
F.38) to Concentration Error In Eq. F.38 we can write ū, une + 8 and
thus write Eq. FF38 as a linear combination of terms of the form
(ihe z)5x (ā,t ( ())8)x)))+)))) 2
(F.39)
where i, 0, 2, 4 and is i 0, 2. Next, in order to use Lemma L. we write únt = Unz 8, and expand (recalling the definition of & from Lemma 5.4). Thus, Eq. F:38) terms of the form
is a linear combination of
(Th/ x) (6/e x)3x (Unt ar x="' (8. x) 2
(F.40)
where i + i 0, 2, 4, and is + is + is = 0.2. Finally, we obtain concentration bound for Eq. (F440) using Lemma I.7
n Σ (the x.)) (5/ 23)" (Unt 3s ( 23) mz)"a(E,
2+0s
(F.1)
1~884- (ahys 2)31 (6'p ( (6 2) 24) 2+
(F.42
0m 1 1 16:
11 12t 2t
d diiotiitio/244
S <log (log
(F.43)
tia
n
(logg
87

---

Here, when applying Lemma 1.7 Bays and Eyet take the role of wi and w2, while / 1/ l22 Sye/ [| ll and / ///l/eel2 take the role of u1. U2, U3 and 44 (we let ug 24 6 ll and choose pa and p4 so that pa + P4 2 + is). Note that we implicitly perform union bound over uge and 82 drawn from the support of fú.. - this does not affect the failure probability, since the failure probability for the result of Lemma 1.7 is quiionar and is assumed in Theorem 3.5 that de for some universal constant C. When applying Lemma L.7 at later points in this prool, we implicitly perform this union bound.
In order to compute the final contribution of each such term to the concentration error between S₁ and M1, must first (1) incorporate the coefficients of a and o' with respect to the Legendre polynomials P2,d and Pi.d (2) take the average of ||8 llé with respect to X (which is implicit throughout this proof), and (3) use the fact that n Z (log d)o(1)
Recall that o = 02, N2. + ōs.dy Na.ala,d. Using Eq. C.4) and the fact that N2,1 x df and N1.d x d² (by Equation (2.10) of Atkinson and Han [22). we find that, up to constant factors, the coefficient of the -fd-rrder term in (t) can be bounded above by max(ō2.a- Ōs,a), the coefficient of the 2nd 2oddor term in (t) can be bounded above by max(02.a 04,a)d, and the coefficient of the 4th order term can be bounded above by ô o.,dd. Similarly, up to constant factors, the coefficient of the 1odorder term in o' (t) can be bounded above by max(02,a. &1.a)d and the coefficient of the 3rl-order term in o' (t) can be bounded above by max(ō2,d: a..)dd Thus, each term of degree originating from a contributes a factor of max(62,a. (4.d) to the coefficient of Eq. (F.88.. and each term of degree k- - 1 originating from o' contributes a factor of max(02,a, as.a)d to the coefficient of Eq. (F38). Finally, for each term in Eq. (F.40) of total degree i 82 + i3 i4 + is 2, note that and of together contribute a factor of degree i + i2 + is + 14 + is + 1, since in Eq. (F.27). aside from the factors contributed by a and o', only a single factor of 8, x is present.
Thus, to obtain the final contribution of Eq. (F.41) to the overall error between M1 and S1. we multiply by an additional coefficient of max 44 ddii tittstttt Taking this into account, the contribution of Eq. (F.41) is
(V==+
dlistiatis/2441
(6 27 (log d) O(1) 18 11
- 18, 16:el12
(F.44
dO(logd))
n
Next, we simplify this using casework on ig. For the case iz S since is + is S 2, the bound is at most
(03.0 + 02 or 11. f1 112 1|oell2tss z (0 a + 5
(F.45)
since +is)/2+1 < d2.5, and because n 2 ( d) o)1)) and we eliminate a factor of 11 112² [10 112" 11 because all of the particles have la norm at most 1. Additionally, for the case i2 2, after taking the average of 11 & 11,% across all X, the bound is at most
d (o o2,a) d) 0(1) A 16.13 (
(F:46
dBllog
n 2 (03 + 2% 4 01 l1 d + +t)
n
(B.c. n 2 dd (log o)(())) and is + is S 2)
S (02 + 2 11
(B.c. A, Va and i2 S 4)
where we have Xye S Va as nt I T,... In summary, have shown that terms of the form Eq. FF38 contribute at most up constant factors, the concentration error between S₁ and M2.
1 because by Lemma E.3 we have that Assumption
5.3 holds as long
(03 +
by using ü, 8, + 6,p Wytt ueg = &ez + The and 7i = 8 + 5nc + uhet Thus, we can write Eq. E.37 as a linear combination of terms of the form
Contribution of Eq. FF37 Concentration Error We expand Eq. F.37
(ō. - x) tii (8/ 2 (fnr x)x ()) x)xx (Une ( 2)
(F.4))
88

---

where i1 - + is + + is + is = 6 (since the degrees of each of the monomials must be 8) and
is S 3 (since each of these terms originates from (i/ xr)* 2z)44 (ū, 2:) (ōe ==)()) x) where
we obtain a concentration bound for
d2 S 3). Using Lemma I.7
Eq. (F.47)
6 xz) 1+1 (8) x.) (āgz xx) (6'2 xi) (īnt 1z) (inee xz)a
(F.48)
E
(og 2:)t (ō; x)) + 2))a(0pp (6 (ūne 2) (Thp 2))²
(F.49)
16el12 +i1
10 1+1 |6 11 (V
d
d'-liitiotistu))?
S (logd) 0(1)
d' f2
n
d dj
(F:50)
As discussed in the previous case, after we consider the coefficients of (t) and of (t), we find that the term in Eq. (F.47) contributes
d
(62 + 0, (log ()) [Bell2* +
18:11
i2
11 11,,,,
n
d d)
(F.51)
Now, suppose is + is 2 2. Then, i + i2 + ig + is S 4, meaning the contribution to the concentration error at most
(0 + T. 12 (62,a + (g,) max (x))12
(F.52)
x
using the fact that n 2 (log d)c for a sufficiently large constant C, and l10:112 S and [|óæell2 S 1. On the other hand, if is + is 1. then i + is + i3 + i4 2 5, and since is S 3. this implies that i + 12 + is 2 2. In this case, the bound at most
(0 2. + o (log o) 18el12+ 18:112
||5
11
18 11 (V
d d to2
n
dlog 4) (F.53)
2 (62,a + (1. 1+ 10:1122
1 11 (B.c. i is + is in S and n 2 (log c)
1 16 d2'd
1
S (02 a + +,a) max (lxx(x))12
1 1
d +/2
(B.c. maxx (1xx(x)la S +/2 and || S 1)
Thus, if i 2, then the contribution of this term to the overall concentration error is at most (62.a + a..)) maxx (xxx()) On the other hand, i4 = 1, then we can take the average of [|8,ella across X. and by Lemma E.3 we have that Assumption 5.3 holds for rt I T..., meaning that Ex 11 S Aye S Va In this case, the contribution of this term to the overall concentration error is at most (62.a + -2,a) maxx [l
2o الا = (62,a + o%.2) maxx (1 Finally, S
when i 2 2, we can take the average of 1 112 over X- we have Ex 11 113 d. and therefore the contribution of this term to the overall concentration error is at most
(62.u + a..) max [|x:(x)1 d++(1+ S (62.a + aZ max (x)(
(F.54)
In all cases, the contribution of Eq. (F37)) to the overall concentration error between S₁ and M1 is at most
(62. + 27.0) max (x)13
(F.55)
Overall Concentration Error
In summary, the overall concentration error between S and Mj is that of G(z). We have expanded G(x)) into various monomials of dot products involving x. and shown that each monomial contributes most (02
(..)) maxx || (x)1l the concentration error. Thus, with high probability.
MM S11 S (62,a + oF.a) max (x)13
(F.56)
89

---

In order to bound MM S2l, an almost identical, but simpler, argument can be used, with y(z) in the place of fp. (z) or f (z). the modified argument, ii tyy and uye will all be replaced with 61. Thus, in the analysis of Eq. (F.40). we can replace thne with 6 and assume that in = 0 (since otherwise this term becomes due to the 8, x factor) - we can then proceed similarly to the subcase where i2 < 1 in the analysis of Eq. (F440). Additionally, the term analogous to Eq. (F.47) does not arise in the case where frz (4) and f (r) are replaced by y(z), since ii and &yt are both replaced by C1, so the corresponding term is O. We also note that the Legendre coefficients of ((x) have absolute values which are less than those of a. Thus, we have
M S1 + M2 - S S (02,a + o2,a) max 11xx())11
(F.57)
Bounding the Expectation To complete the proof, it suffices to bound [M - M2]. We have that
M1 M2 = Vu. (((e) 5, - Van, T-(pet)) 8e
(F.58)
and by Lemma
we have
[11a' Wne 113] 1/2
M M2 < (62,a + o7,0)
E
+ 11 - uip l12) |5ell2
(F.59)
(0) (,1
x (0ž,a + oa,a) max 110: (x)(x
(F.60)
x
Combining this with our bound on 1M S21 |M2- S2l. we find that
(grada, T(p.) grade,. I(pn:)) del S (03,a + (g, max (x)x x
(F.61)
as desired.
L
Lemma F.3. Suppose are in the setting of Theorem
3.5 Suppose s,t 2. 0. Then,
[grrad [((xe) - grad,, [(ox)lla S (62,a + d
max [ u. (x:)ll2 ic(m
(F.62)
Proof. We have gg a, I(pe) grad a, T((Pa)l2
(F.63)
-I(1 à )V L(pe) - -1- (. l(pa)ll
(F.64)
= -1( unūi²) T)
1
Σ
(wx) yo xz)x
(F.65)
n1
i=1
- (I-
Σ (so (2:) - o' a.)
(F.66)
[1(o.(x;)) v(x;))o (ū, x;)(( - unūe )are
(F.67)
- (x)) -
(ox))) oo' (0] x:)(1 û, u. )a:ll
(F.68)
We bound the right-hand side by several applications of the triangle inequality. First, observe that for any gd-1 we have
1 - f2. <
(((((()) x) - (()((x)) x)1
(F.69)
(102,al Ne.d + 10a.aly Ni x =x(x)) 2z]
m ==1
(By Lemma E.12
(102.l / Ne. + | Na.a)
[( (xx)l22
(F.70)
m
$ (1ă2,a / N2d + lōs.l Na.d)
max ((ūe(x)) - (xxxll2 ie[m]
(F.11)
90

---

Additionally, for üt and üs with the same initialization X, we have
(oū x) - o' o' x) S (ázz,l V 2, + |ō4,al / Na,a) | x - ui'0l se
(By Lemma E. E.12
2 ((zz,ol N2,d + o v NN,a) (haa)le |å - uo
(F.72)
x (102,a) V N2.d + ( V Na.d) max [[te(x)) (xx)ll22 sEfm)
(F.73)
Also, for üt and üs with the same initialization X, we have
[I ú )x; (I 0,ú, )aell2
|1(dg z;)u, - (ūg Dz f.lūlll2
(F.74)
[1 Zi ûs xz) xz) 2 + lūg x;ill - 0,1l2
(F.75)
max [(x((x)) (xi)l2... ke(m)
(F.76)
Finally, for E gd-1 and any distribution P on sd-1, we have
(f(#) v(z) $ /ó2,al V N2 + l0,,dly /
(F:77)
since we can bound [x((]] using Lemma E. and because (ignoring ha,a since is equal to oo.4)
y(x) 22 + a.d and using Assumption 3.2
Thus, we can simplify the right-hand side of Eq. F668) using the triangle inequality, and combining
Eq. (.77I. Eq. (F73). Eq. (F.66) and Eq. F.77
[1.a.. (2:.) - y(xx))) o'' (á, a=)(1 - û,ú, )a; - (Jp. - v(a:))o (ū, xx)):)(( úg Jail12
(F.78)
S 1m (xx) - (x:))llo - i, )willz
(F.79)
(xx) (00))))(lo(( xx)) o' a: a:) - u )o:: l
(F.80)
+ (x,) - y( lo (0 x. xx)
un )r: - (1 ú. â Jana || (F.81)
S (02 o,,d .E(m)
max [x(())) (xi)ll
(F.82)
+ IIfe. (2,, - y(xx)) (0, xx)) o' d'
[
Uz )aralle
(F.83)
+ +1f (z.) (Zx)) o o'((0 (0, x;) xx) -,0, )a; (1 û, á
)a
(By Eq.
(.711. bounding |o' æ;)1 by Lemma
12
S (02,0N2.. + 62 22 N1.d)
max .E(m]
[(e(x.) - (x(x))l2
(F.84)
+ (6 No + 02 Na.a) max (xxx(x)) û, (xi)
(F.85)
ve(m)
+ +15, y ( o (0] o: ax))ll(T
in )ar (7 ú. ú Jan |a
(By Eq. (:77) and Eq. (F733)
S (22 + N4.d)
max [ a, (xx)ll2
(F.86)
xE(m]
+ (0 N2.d + N4..) (e(m]
max (xx())) - 0, (x+)
(F.87)
(0 No,1 + 0% N4.a)
max (xx)))) û. (x:) (ml
(By Eq.
F.76
Eq. F.77 and Lemma E.12
S (6 + N
max (x())) ú. (xx)ll2-
(F.88)
.E(m]
Therefore, Eq. (F gives
|grad, a, [(p.) grad a, I(p.)ll2 S (6., + Na,.) ie/m]
max [[ú((xs) - (x(x))ll
(F.89)
as desired.
91

---

Proof of Theorem 3.3 To show that projected gradient descent achieves low population loss, we show that does not diverge far from projected gradient descent. Suppose P and Po are both equal to unif(xI..... Xm)) Then, by induction over I, we bound the difference between (x()) and (x) for all X (xi,... Xm). At O, we have (x(x) uo(x). Let be nonnegative integer, and assume that for all s I we have maxx (x)x S va
By the definition of projected gradient descent, we have
ūx(x) - 1) grad in (x) [(pt)
(x)1(x) [xxxx) grad ( T.(x:)ll2
(F.90)
Note that || - grad,, (())) ll2 2 | since grad,, ((r) is orthogonal to in Thus, letting II denote the projection onto the l unit ball, we have
|1 1n(c+1)1l2 = || 77 grad [(p.)) I(
(F.91)
$ 1 - grad,, (p:)) - 1n(e+1)l2..
(F.92)
Also, we have
(1(++1)(()= = ax)(x) - m((+1)
grad ((p.)ds.
(F.93)
9/2
Thus,
0 17 grad,, ((p)) (n = 1 grad [(px) int + / grad L(px )ds
(olet))
2
(F.94)
= 1 une 112 + |1 grad T(Pp))
(t++1) grad a. [( )ds
2
(F.95)
+ 2(ūx - unp)) p((++1
(grad,, u. L(p:) - grad ((p)))ds
(F.96)
= =110,11 + +0 00 + 62 222))22))
(F.97)
+ 2no. (grad,, I(px) grads., ((pxt))
(F.98)
/ ((((t)) nt
+ 23,
(grad I(pne) grad, D(pa)ds
(By Lemma
E.14
< 818 + (17)(63,) + 222))22))
(F.99)
+ ( + -2 max (x)
(F.100)
m((+1)
+ 25,
(grad [(p))t grad, i(p_)ds.
nt
(By Lemma F2
92

---

Further simplifying using the Cauchy-Schwarze inequality we have
[ - 7 grad,, ((pe) ( l2 - 10:12 + (1 02,02
(F.101)
+ +0(1163,r + ,2 max (x)(
(F.102)
max + 21|16/112 (.m.1((++)
l|grad an [(prt) - grad
[(pa)lla
(F.103)
$ 110:12 + +0 o2.))
(F.104)
+ +0 + ,3,a) max (x)(
(F.105)
+ 2n/11:112(03.. + ad)
(F.106)
max +E(ot.n(++1) 4E(m)
max |lū - (n)))
(By Lemma F3
S 110:112 + + (7 + 0 2d
(F.107)
+ +0(1662,+ + )7,4) max 1,.(x)11
(F.108)
+ +2/118/122(22 , "1 + 0 ad)
(By Lemma E.14
$ 118:112 + +0 +
(F.109)
+0(1163, + 23,0 max (x)(
(R.110)
Thus, we have
max 100+1 < (1 (1+ (10001662 + +oia))) max +xx) + - +22
(E.11)
which means that for some universal constant C,
max (|x21
1+ Cn(o2 (1+ +ol.)) . .72 02,0)00
(F.112
1n + +o2a)) t+1
$ 10 + 22,0
(F.11)
(1+Cn + +.2,)) -1
H+1
(1 Cn(02 + o2,4)
$ 30 + /d.
(F.114)
Cn(o2 + (2,a)
$ n00 + 02 o2a)da (1 Cn( + o
4+
(F.115)
$ n(63, + 02 dd cnll (00,
(F.116)
Thus, as long as 17 (02,++02.)) aa for sufficiently large universal constant B, and choosing 7) that T, T. is integer multiple of 17, have max, ||o, (x)ll2 S J/2 because if < T=s
so
then "
max (1844 < n )d dd t0
(E.11)
S <n d 0log
(E.11)
< ( + (
(F.119)
93

---

This completes the induction, and therefore for all t S Ttis, we have maxx (x)( S Va Addi- tionally, by Lemma 16 and using the coupling between Pr and Pyt which is the joint distribution of (ūe (ūx(x), Bod (x)), we have
"
0 + 87d
E 2 ((f - fpm (x)) S (6 - (7,a) max (1)x 2
102. + a7 2
d
(F.120)
which completes the proof.
G Missing Proofs in Section 3
G.1 Proof of Lemma 3.
In this section, we prove sufficient and necessary conditions for 72 and 7/ when the population loss is O.
Proposition G.1 (Necessary condition). Let fp(x) = Eumplot lo(u x)] be a (not necessarily symmetric nor rotational invariant) two-layer neural network and y(z) = h(e] x) the target function. Define
hE.d Vk 2 0. Suppose Ex-8a- (f - (((()))))(x)(}>)) = and 62.a * 0, ôr,d # O, we have
7kk 0u.
12 $ 14 +
(G.1)
vd
74 S 12 +0
(G.2)
Vd
12 1
(G.3)
Proof. Similar to the proof of Lemma 4.2 we have
[(fp(x) - h(x ez)))²= =
E
E
lo(u [o(uTx))] x]] h(xx ea))
M(TTe)))
(G.4)
amgd-1
umup
8k Ph Phalu x3]
Pha(x" Tei( 2
E
E
(G.5)
1
amp
k==
k=0
Proa(oTTi))) eii))²
E
ŌR.d E [PR,auu x)] - PM,(l whip'
(G.6)
2n-
k=O
Continuing the equation by invoking Lemma
C,5
we get
(önd umpp
E (u (T]) h Ph.aPralle Raa(( Te ea)))
(G.7)
E (on.d E
(Pna(uTx>) x))] a e1))
(G.8)
anddd1
ump
kdd
E. uuplnpp
(Pea(uTT)) ul) 2% Praau es)
(e)) + 2 Pr. Teill)
(G.9)
K=0
Therefore, Ex ((5((x) h(xx" Te [())]]] = implies that
E u,u' up
|P2alu" TTul w/) - 22222.a e1) + P2 ei)l = 0,
(G.10)
E u,x
[Paalu" ul) 2YaPaa e1) 2 Pa ex)] 0.
(G.11)
First we prove Eq. every u, v E gd
G.3h. 1. By Lemma
C.6 there is a feature mapping @ Sd-1 + No. such that for
(ō(u), ((v)) N2a22.. v),
(G.12)
|1 = N2.1-
(G.13)
94

---

Rewrite Eq. G.10 using Φ2 we get
E - -200)] 0.
(G.14)
Or equivalently,
7202(c1)
E [d)( ump
(G.15)
As result,
2b (e1)
E Euppp Eumploz((]]] [[2[1)]
Euep [|02(u)]lz 1.
72
S
(G.16)
/Na.d
N
/22dd
2
Similarly, we also have 74 S 1.
Now we prove Eq. G.I.. Recall that
d
12 1
P2,a(1) d
(G.17)
d-
(t,)()) (d
+ 2) (d +
6d + 12
3
(G.18)
d²
de 1
12
Combining Eq. G.10 and Eq. (.17 we have
d
E P2 T ul) - 2/222 61) + P2 e1)
(G.19)
d
up
1
E
[u u2)2 2722- E1) 7 (
2 272 + 7
(G.20)
a u up
=
02 7/2 11 z (1-
(G.21)
urup
Consequently,
E urup
[1 02
22e
62
d - 72
(G.22)
Similarly, combining Eq. (G.II) and Eq.
G.18) we get
d-11 = (d+ 2) - 4) u.u (P u') 47444 - P
(G.23)
3
E
[Iu [uuT'y uu'² 2m ex)
(((TTe)) + (d 2) + +4)
(1- 274 + +
(G.24)
u,"'>p'
6d + 12
E [(uT - 2n e2)2 ))()Te))
e1)
(G.25)
(d+ () + 4) axx-
Consequently,
3 (d ( () + (1 74)2 + (d + 2) + 4) (74 + 1)2.
6d + 12
E 1 2
(G.26)
Now for any fixed & E gd-1, our key observation that by Jensen's 's inequality,
E. wrep
[(u T E) < E [(u {)*] [)*]
(G.27)
ump
On the one hand, by Eq. G.22) we have
[u T E)
72(e T))2
=
1k -20 82 [22
(G.28)
ump
ump
(1- 72)
E 02 amp
720
11111
(G.29)
d
95

---

On the other hand, by Eq. G.26 we have
E [u E) (((TT)))
E (u - {
(G.30)
ump
ump
6d 12
< E 1 01 me 11 1 $ $ (d + () 4)
(1/ + 1/ +/ ha- 11).
(G.31)
ump
Finally, combining Eqs G.27
G.31 and set
Eι we get
11
12)2
6d 12 (d+ 2)(d 4)
72
$ 7/ + 2
(hrs + 11 lh 1)).
(G.32)
It follows directly that
(
12 74 +0
(ia
(G.33)
Finally we prove Eq. G.2 For any &, E gd-1 we have |u T El S 1. As a result,
E [ E)*]
< u~p
[[u E)²].
(G.34)
umpp
Combining with Eq. G.29 and Eq. G.3 and setting & e1 we get
74 S 0(1/V)) + E (u (()T())== 0(1/V)) + E [u T E) ((1/Vd) + amp
(G.35)
amp
Proposition G.2 (Sufficient condition). Let [p(x) = Eumplo(ul (op(ou(u))) be a two-layer neural network and y(z) h(e x) the target function. Let Q/k- ho.d oo a is degree-4 polynomial, and
hi.d wk 2 O. Suppose ha.d O, vk & 20,
$ 24 - - 74 72 - 0 (1/d), S 0
(G.36)
(G.37)
(G.38)
Then there exists symmetric and rotational invariant neural network such that En-za- (fp(z) y(x))²
Proof. Recall that by Lemma 4.2 have
for every symmetric and rotational invariant neural network p
02.1
0ā,1 +
(pp) = 2
(2.d(w)) - 72)
E umup
[Pa,a 7/
(G.39)
ump
2
where (w, w E 1-, and v1 mg2d-2 exists a one-dimensional distribution such that
Therefore, only need show that there
E [P2a(w)) = =72, wo-je
(G.40)
E P (Paa(w))] 74- wmy
(G.41)
Recall that
d 22 1
(2.(() d
(G.42)
d-1'
Pt,a(1) (d + 2) + 4)
6d + 12
3
2
(G.43)
d -1
de
J
96

---

Consequently, Eq. G.40
and Eq. (G.41) is equivalent to
d --1
1
E [w0]
12 d'
(G.44)
way
d² -1
6d + 12
(-TJ) 72 + 2)) 2d + 44)
d-1
3
E [w0] wmi
+
(G.45)
(d+ + () + 4)
(d 2) 4)
Let B2 - -2 + 3/2 and 34
(d+2][d+1) 74 +
[d+2)(++]]
6d+12 ا + 4)
[d42)d+1)
Then we
have 1122 10-7% 72| O1/dd) and 184 -4/ - (1/dd) Rewriting Eqs. (G.36) G.38)
(G.36)-G.33)
using 82 and B4.
we have
8/ $ B1.
(G.46)
84 S $2,
(G.47)
82 S 1.
(G.48)
By Proposition I.14 there exists a distribution u such that
E [ B2,
(G.49)
E [w=1 woo
= B4.
(G.50)
Hence, by setting the marginal distribution of w equal to μi", we have
[2)] = 72.
E wwyu
[Pa,([u)]] = 74
(G.51)
w~
which proves the desired result.
H Proofs for Sample Complexity Lower Bounds for Kernel Method
In this section, we present sample complexity lower bound for kernel methods with any inner product kernel. That is, the kernel K : gdd x gd-1 - R can be written as K( 2) = <(z 22 for some one- mieddmmnsional function k : [-1, 1] + R. Inner product kernels are rotational invariant and do not specialize to any coordinate. In particular, NTK of two-layer neural networks is an inner product kernel ge. Du et
30 Wei et 76))
To prove Theorem 3.6 we only need to work on the degree-k spherical harmonics component of the target function and its estimation because, by the orthogonality of spherical harmonics, any lower bound for the degree-k components directly translates to the original setting. The following lemma states the lower bound for general & 2 O, while Theorem 3.6lonly requires the case k = 4.
Lemma H.1. Fixed any k Z 1, let Pr.auu x) for some fixed E Sd-1, When d larger than a sufficiently large universal constant and < (8k))-(3k++ (In d)k++2), with probability at least 1/2 over the randomness of i.i.d. data {I1,... ,In} drawn uniformly from Sdl11, any estimation g of the form g(x) Dim 3.P (a, x2;) must have constant error:
E z E_ (g(x) g² 3/4
(H.I)
Proof of Lemma H.1 is deferred to Appendix H.2 where our key observation is that PR.a((z,.) )) cannot correlate too much with g* when x; is uniformly randomly drawn from the unit sphere. Indeed, (P (Pi.a((a) ( "))""))))))))))))
(s,a(<r u) concentrates within +1/Nk,d (Lemma C.7p. Hence,
can upperbound (g.g*) = Σn B, Px, u) & 181l2yn/Ne.d However, we also have llg/l2 = Enje(m) 3,3, Pk.a (z. x,)) Z | because the matrix (Px.a xz)) xz))).ee concentrates around (Lemma H.3 Since llg 112 3/4 requires (g. > 2/ we must have 2 Nu.d d
H.1 Proof of Theorem 3.6
The following theorem states that for every 2 0, to achieve a constant population error, we need least Ced-*(In d) (k++2 samples where Ce = (8k) -3k++1) is constant that only depends on Hence, Theorem 3.6 is direct corollary of Theorem H.2 by taking A 4.
97

---

Theorem H.2. Let y: g + R be function of the form y(z) h(g. x)) where E Sd-1 is fixed vector and h [-1,1] 1] - is a one-dimensional function. Let S x S - R be any
inner product kernel. When d is larger than a universal constant and n < d" (8k)- (In (lnd) ()- with probability at least 1/2 over the randomness of i.i.d. data points {xi)E= any estimation
of the form [(x) >i=1 3 (xi, 2) must have a significant error:
1 - .112 (hx
Proof of Theorem H.2 In the following we first fix a degree k 2 1. For every >> 0, let Yi be the space of degree-/ spherical harmonics and II, the projection to Lt By the orthogonality of W L.d. we have
by fll2
Σ
(y f)ll [ f)
(H.2)
By Atkinson and Han [12 Section 2.3], the projection operator II, is given by
(Ll)f)(z) = VNd.d
&~gu _,, (pa((=T))((E)))
(H.3)
<) Consequently, have (x)) /NR,d Rewge-1 (P E)) Recall that
T(d/2) 1
pa(l) A (1 (3) T((d 1)/2) Vi
(H.4)
is the density of U1 when (u1,-.. ua) is drawn uniformly from sd-1, and hadd Exequ (t))(())(()) By Funk-Hecke formula (Theorem C.1 get
(x) au,aa q-), Vo Ed-1
(H.S)
Similarly, we can write the inner product kernel K as K(x,z) = k(x 2). Define Ri.d Eespa (t)) Then have
n P.Pk,a('' (aeT))))))))) xi), Vx E
(Ixf)()) = hikdd
(H.6)
Now we apply Lemma H.l to the function (hx,d) ( (I[ny) = Pk.a(lg., (aa(((g..)))) :)). As a result, with
probability at least 1/2 over the randomness of {a, HEr for any function f of the form f(-)
E 3 K,K(2,,)) :), we have (((h,.))-' 3 2 3/4, which implies ly fll2 2 (hv.d) O
H.2 Proof of Lemma H.1
Proof of Lemma H.1 First we prove that when ||g*||2 = 1, llg - -1 < 3/4 implies (g.g*) 2
| To this end, by basic algebra we get
3/4 > ||g - g" 112 Illll + 113 - 2 ((
(H.7)
Consequently,
(g.9") 2 |19113 +))
1
11112.
(H.8)
where the last inequality follows from AM-GM. In the following, we prove that with probability at least 1 1/2, Eq. (H.8) impossible when n < (8k) (3k-1) (In d) -x++2).
Recall that Lemma C.5 states that for every 2, = E gd-1 and k 2 0.
(), ()) = Pk.a((z; (xx 2)).
(H.9)
As result,
n llall3
n
Σ 1
3,8, PRa(((a)))))) Σ
3,3, Ph.a((a,,
(H.10)
i-=1
98

---

Invoking Lemma H.3 with probability at least 3/4 over the randomness of (Za))e|m) we have
119122 =
2 118|
(H.11)
4==1
For the LHS of Eq. H.S).
(0.9*) = Σ 8i Pkea((an 1), Pk,a((u,))))
Σ
Be P.Pk.c (k.d( u))
(H.12)
==1
i=1
1/2
1/2
1/2
n
Σ
33
Σ Pa.a((2s, u))?
= |111|2
Σ (k.a(Frs u))?
(H.13)
By Lemma C.7 when < dh (8k) (3k (In) (nnd)((+))) -(k+2), with probability at least 3/4 we have
n
1/2
1/2
(9.9 9 ) s | Σ
Pa.alfar u))?
$
||0||2
n max Pk,d((2s, u})?
(H.14)
se[n]
$ 11111ly n32k In (ddn))
113112 Ind'
(H.15)
d
Combining Eq. (H.Il) and Eq. H.15). there exists universal constant do such that when d > do. with probability at least 1/2 over the randomness of {x1, .. Inn.
(g.9") || 11gll//2.
(H.
Recall that ||g 90 3/4 implics (g. g²) 2 Consequently, with probability least 1/2. when n dth (8k) (3k+ (In d) ( (k+2) and d > do we have llg g0113 3/4.
O
Lemma H.3. Let ...... Fn be i.i.d. random variables drawn uniformly from gd-1, and Pkád the degree-k Legendre polynomial. For any fixed Z 1, define the matrix Mj. E TRn where [Ma]is ji = Pk.a (I,, x,))) There exists universal constants dy such that, when n < d (8k) (8k)-(3k+3)) (In d) (k+2) and d > do, with probability least 3/4,
Amin (Mx) 2 1/2.
(H.17)
Proof. In the following, fix t 2 First, we prove that
E[Amin (M+)] 2 7/8.
(H.18)
To this end, we invoke Theorem I.4 by constructing a matrix A such that Ny.. 1 ATA = M..
By Lemma C.6 there exists feature mapping Φ: Sd-1 + Re. such that for every u, v 8d-1
(d(u), (bv)) -Nk.PR.auu (uTT
(H.19)
[ø(w)) Nk.ds
(H.20)
(H.21)
We set A, = o(x;), V1 S n. Consequently,
1 Ne.d
A T A = Mc.
(H.22)
|| 11 Nk.d:
(H.23)
E E[A
(H.24)
In the following, we upper bound the incoherence parameter p defined in Theorem 1.4
p
Nk.d
E
Σ
(A,
(H.25)
jE(n),i
99

---

By Eq. 1.19
we get
1
E
max isn
Σ
(As, Az}²
= Nk.d
E
max
E
(
(H.26)
Nk.d
isn
jE(m]
jE(n],st
n, E
max max Pk.d (2 (z)1,
(H.27)
371
Invoking Lemma
we get
Nd.d
max max Pk.a (a.a isn 371
x (32k In(dn))**
(H.28)
follows that p X (32k In ( Now that we have established the conditions, by Theorem 1.4 we have
BlIM - - n1(n (32k (ndn))
(H.29)
Na,d
Note that Me,d 2 (k+d-2) 3) 2 (d/k)* when &: 2 2, and clearly N1.d 2 d. Hence, there exists universal constant dg such that, when n < dt (8k) -3R++ (In d) (k+2). (32k In(d))) n(8k:)2k In(dn) ([(k))=+(((d)*++)
n(8k)2h In (nd
(H.30)
Nk.d
d"
de
In
Hence, there exists universal constant do such that when d 2 do.
E Mmii (M) 2 V n(Inn) (32k In(dn))* 2 21-0(1) Ind 7/8.
(H.31)
Nk.d
Now we prove Eq. have E[[] 1/8. By basic algebra, also have
Let = 1 Amin (M) be a random variable. Note that by Eq.
we
Amin (M) < Mi. (( = (k.a(1) 1,
(H.32)
which implies 2 0. Therefore, by Markov inequality we get
Pr(Amin (M) S 1/2) Pr(t 2 1/2) s 2 E|t] $ 1/4.
(H.33)
I Toolbox
L.1 Rademacher Complexity and Generalization Bounds
Lemma 1.1 (Special Case of Contraction Principle (Lemma 4.6 of Adamczak et al. (4)). Let F be
family of functions from to Tk, and let up be an T--lipschitz C-Lipschittfuunction with yp() Then, for any S (I1,..., NNN C D)
E Sup
Σ (p((((=)) < I. E.su G GEF
Gf(F))
(L.1)
G
where the expectations taken over i.i.d. Rademacher random variables ,
In the following, for a function family H. let Ry (H) denote its Rademacher complexity and Rs (H) denote its empirical Rademacher complexity (given fixed dataset S) - see Section 4.4.2 of Ma
153 for a definition of these concepts.
Lemma I.2 (Empirical Rademacher Complexity of Products). Suppose F and 9 are two two families of functions from D to IR, with D RRd, which are uniformly bounded by 0. Then, for any set S= (=,,..... ] C D,
Rs(F 9) < B. (Rs(F)+ + Rss()),
(I.2)
where F. F-G = SoolsFF.GO 9). As a corollary,
Rx 9) B (Rx RN (G)
(L.3)
100

---

Proof of Lemmal 1.2
We can write
Rs(F 9) = E sup
(f(1))(z))
(1.4)
a FEF
N
(f(x.) + +g(z))) - -(((a))))((()))))) - 9xx;))
E sup
c FEFWEE
N
4
(1.5)
N
(f(x)) + g(xx))?
(f(x;) g(x;))
sup WEg
64"
+ E
c
JEF
sup
N
4
(I.6)
Using Lemma I.1 and the fact that the function x: + x² is 2B Lipschitz on the interval [-B, B), we have
N + g(z)
f(x)
1
N
f(x)) - (x,))
RF $ 2B
sup
4
+2B. sup fET.gEg
fEFgEGG
N 2ml
4
N Am
(1.7)
B (Rs(F) Rs(g))
B (Rs(F)+ Rs(9))
(1.8)
B (Rs(F) Rs(9)),
(1.9)
as desired.
Lemma L.3 ISymmetrization - Analogous to Corollary 4.7 of Adamczak et al. (4). Let X XI...., XN
be independent random variables. Let F be a family of functions from D Rdd R. that is uniformly bounded by B, 2 1, and let G be a family of functions from D C R to R. Then, for nonnegative integers P1, P2, P3, P1 and a fixed g G that is bounded by B2 1, we have
N Σ
sup X Sa
(L.10)
N
(I.11)
S poly(p1, P2. P3. P3.P1, B th B2) RN((F)
(I.12)
Proof of Lemma 1.3
By Theorem 4.13 of Ma 53
E
sup fa
N (1AF)))P[((C()PPP33))))))4/1/111 1=1
Σ
(L.13)
X:
(I.14)
I RN F FPa. FP+.g) g)
(By Theorem 4.13 of Ma 53
2 max Bp tp B2) (RN(FPL FP= jPs FFA) + AN ((g))
(By Lemma 1.2
2 max(BBitttttottp B2) RN FP HP' .
(L.15)
2 max(BP:ttattin) Bz) max B t max B BRstp)
(I.16)
max BT1) (JPI) + RN + (FP) + Rx (JP*))
(By repeated applications of Lemma I.2
I (axxBBpitttttteenn Br) (Rx (Rr(FM)) + RN(FF2) + (N (x(F))
(I.17)
101

---

Finally, we simplify (F for each i. For nonnegative integers p, since F is uniformly bounded by Bi, the function + x² is p -ipschiizz on [-B1, B1] for 2 1 and is 1-Lipschitz for 0.
Thus, by Lemma I.1 we have RN (FP) min(1, Rx and
E
sup
N (1Ck)//44444K)P4444)))4444441143
(I.18)
Xr
E |113:)/|1440330P133333:331111111
(I.19)
2 max(Bi
mnx(BBiintettm-n B2) max , . " ,pBBtt) R, (F)
(I.20)
2 Ps, Britpatestps B2) Rx(FF
(I.21)
as desired.
I.2
Concentration Lemmas
Theorem L,4 (Theorem 5.62 of Vershynin (3). Let A be matrix with independent columns
{Ay}} and m 2 n. Suppose A₃ is isotropic, iee, B[A,A]] = and || = / for every
j E [n]. Let p be the incoherence parameter defined by
E
max
Σ
(Ai, A,)²
(I.22)
m
iSn
jE(n),jzt
Then there is universal constant Co such that
E
A
7
||
Cov
p pnn
(I.23)
The following proposition gives the tail bound for inner products of random vectors in gdl (c.f.
Theorem 3.4.6 Vershynin (74)).
Proposition 1.5. Let be vector drawn uniformly at random from the unit sphere Sd-1 For any fixed vector e E Sd-1, we have
Ed
Vi 100, 0, Pr(lu Tol S 2exp
(I.24)
2
By the symmetricity of and we can assume that 01.0,... without loss of generality. Let x = (uT (TTvv- + 1). Then we have x Beta( d-1 2 d=1). Hence, the desired result follows directly from Skorski 63 Theorem 1].
Proof of Proposition 1.5
Lemma I.6 (Modification of Proposition 4.4 of Adamezak et al. 14. Let XI...., XN be i.i.d. random vectors drawn uniformly from Vdgd-1 and let p1, P2. Ps, Ps. q1 92 be nonnegative integers. Suppose Pl + D2 + Pa + 2 2. Then, for fixed wi, we E Sd-1 as long as N S d for any universal
constant C O, we have
Σ (X, un) (xx, ua) un) Px) (X, wx) 01 [X,, w2)) 1
sup
Sdd
N
1=1
(I.25)
E Xnvadg-11
(X, u1) ((X, 12) (X, un) Px w1 | [XX, w2)
(I.26)
S Cp.pp.prrnn (log d1
(I.27)
+ Cpr.posss d) +21)
P2.
N
Elog d)
(I.28)
102

---

with probability at least 1 driingdi Here, Ppp.p.p.p 9..1. is sufficiently large constant which depends on Pi. P2. Pa. P4: 91 92 and P1.p..s.ps a constant which depends on P1,P2. P3. P4.
Proof of Lemmal 1.6 We largely follow the proof of Proposition 4.4 of Adamczak et al. 4 with some minor modifications. First, let E1 be the event that (X,, w1)l S (log d)² for all 1,..,, N and define E2 analogously for w2- By Lemma 1.10 the sub-exponential norm of (X,, w1) is at most an absolute constant K O. Thus, by Proposition 2.7.1 of Vershynin [74], we have
P PKK wi)l 2 (logd)²) - 2 exp (- || (IX,,,))l.,/ (1.))))))))))
(log d)²
(1.29)
exp ( - 22llg d)²)
(I.30)
2 ()ogg
(1.31)
Thus, by union bound, with probability at least 2rilng for all i I,...,N, we have (X,, will < (log d)², and (x,, S (log d)² (by performing a similar union bound using the sub-exponential norm of (X,, W2)). In summary, En and Es simultaneously hold with probability at O.
least 1 d)(log
1 since N I d for some universal constant
Now, as in Adamczak et al. []. define B 1, which we will specify later - this the amount by which will truncate (1), (Xe, U2), (Xi, uz) and (X;, ua). For convenience, define the function pp IR + given by
t
t [-
(t(()
B
t> B
(I.32)
B
L<B
Define B' (log d) and define y similarly to (p.. but with B replaced by B'. If Ej and Eg hold, then for all u1, u2, u3, u4 E S we have
N
(X,, un 1 (X, 22) P3) ua)
(X,, wi)) |e
(I.33)
E
(xx,
uz) 1" (x 1"d ((X, w/ wz pp ((x, w2))
(I.34)
(Xi,
P, " (*,, w1> w1)1"'(X,, w2))
(I.35)
E
u2) (x, ua) (X, u4) 1r /y (X, wi w())| p ((X, )2
X-
(B.c. Fg and Feg hold)
(x) (2)
ua) |1, ((x; W2)1
wi>)
(I.36)
N
((X, Pi| uz) 12 us) uz) u4)
w2)
X-vdd
(I.37)
+ Cp CPr....P4. R(B")
(By Lemma L.8
Thus, the rest of the proof will focus on obtaining an upper bound on
U== sup Ui, Ma, 144
(x,, PT) u2)1 uz) 1e uz) u2) w1)) ((x w2))
N
(I.38)
E Xwvaga-1
(X, m2))P" u2) un)lP*(p (ex)e//rx(((X,1( wl))| (g ((X, w2)
(I.39)
103

---

First define
U'
sup
u2)) (p))) 24))
(I.40)
( p0 (2))
(1.41)
E X-y
u4))
(I.42)
(PB p ((X, 22
(I.43)
We will first obtain upper bound on U² then show that U and U are close. We apply Lemma L3
with = {ar + (z(( u)) u E sd-1), }. and with fi corresponding to y ((z, uz)), and finally
with ( = Ip w1)) Io w2) 19, to find that
E U < poly(p1, P2. Ps. PL, BP BPiteatsste (B")a (N(F)
(By Lemma
I3
< poly(p1, P2, p4, B trattttt (B')att))
E sup
epp (( u))
X uesd
N
==1
(1.44)
$ poly((1, P2, P4, BPI (B))tt9))
E sup
cXx,, u2
Xas wesa-1
N
(By Lemma I.I
since yo is I-Lipschitz)
S poly(pL.P2.P3. Pr, Bei +B))
cX2
(I.45)
X
poly(p1, Ps,
Ps, P4. BuLtttttst (B')art))
(By Lemmal
L9
Now, show a high-probability upper bound on U², we apply Lemma 4.8 of Adamezak al. (Talagrand's concentration inequality), and we ensure that all of the conditions are satisfied:
We set XI...., Xx in Lemma 4.8 of Adamezak al. of Lemma [.6
4 to be defined in the statement
For convenience, define
@B,B'' Ma. (3)) = (1p un)) "allillsnsss ((r,, u2)) |p ((r, ux)) k (pn((1; ((x, u
(I.46)
(( wi) ko ((z, w2))
(I.47)
Then, we let F = {x + BBB an
(x)
E X X
[9B an uw, (X)
U2, Ug E gd-1)
(I.48)
where wa and We are fixed (i.c. as defined in the statement Lemma L.6
We let +22 - (B') + which is uniform bound on all the functions in
Observe that for all f F. Ex f(X) O.
We let supjer f(X)) and 0² = sup EN E(X)) Observe that o² <
N²N.
Thus, we apply Lemma 4.8 of Adamezak et al. with
d (log d)2 allog d)² E U' (1.49)
apoly(pl, P2, Pa, P4, D tpttpptss (B' ( N
X,
104

---

to find that
P
U' E U'
Ps Ps, Brittsttsta (B()at))
(I.50)
X;
N
with failure probability at most
exp ( (- 2 2(02 2a EUN + 3atN
N
12
sexp
(I.51)
12 12max (2 + 2a N E EU .atN)
N2
I.
exp
min
(I.52)
12
0² 2aN EUU
N2
IN
exp
min
(I.53)
12
a2N + 2a IE EU
N2 a²N + 2aN I EU vdN
exp
min
(I.54)
12
ON
exp
min
a² + 2a EU VdN
(I.55)
12
PN
ON
S
exp
min
a² 2a EU VdN)
(I.56)
24
&N
exp
min
d, 2a 2aE VaN)
(I.57)
24
S
exp
min
(d,IN, VdN)
(I.58)
24
exp
min
d, vdN
(I.59)
24
I exp((S2(vV))
(I.60)
Observe that Talagrand's concentration inequality only gives us bound on supfer Σi f(X.). while we wish to obtain a high-probability bound on supyez Eiz f(X,)). However, using the same argument, we can obtain similar high-probabilityy bound on supjer Σ E f(X))
- fEF Df=1 Thus, obtain lower bound inf E f(X.) which the negative of the upper bound that we obtained for supjez Σ (X),,, meaning that the upper bound we obtained for sup Enn f(X,) also holds for sup Σ f(X:). In summary,
U' & poly(p1, P2, P3, ps, Bor Boit (B')a192)) (log d)² V N
(I.61)
with probability at least - exp( 12( ())))).
Now that have obtained high-probability bound U let analyze the difference between U' and U. for convenience, let 9 01..2.1122110 N2.. Ma. (x) (xx ui)lPi (a,, P2) ua) | (ar, u4) ( (ppr(((,, w ( ((r, w2) i.c. we are sim-
Recall the definition of JB. on Mi. w1.
from above.
Additionally,
ply removing the truncation by B. Then, following the proof of Proposition 4.4 in Adamezak et al.
[4. we have
U U
(I.62)
sup
9B' M4, (X,) 9B, B' M3
(x1))1
UUK uل 2
01,02,00 us, U3.. -
j=1
(I.63)
N
+
sup
an wo - 9B.B', Mi, (x))11
Utk(,, 45)1l B)).
Mi 02, ug 68
N
N:
j=l
(I.64)
105

---

where {C} denotes the event that the condition C holds. Here the inequality holds because the difference 9B' 22,,, (X;) 9B.B' Ma, ar (X,) nonzero if and only (U, uy) 2 B3) nonzero. We have obtained a high probability bound on U' Let us now obtain a bound on the third term on the right-hand side of Eq. (.62 (here, Cp P2. is constant that depends on pt, P2, P3, p4, Q1, 92):
DR, F. "ww. (X.) (1(6 B)) gn ww Ma (X:)
(I.65)
j=1
$ 21) u2) 22) |2,, (X,, u2)1 (X,, w1) 18 (X,,, w222 (UIK,, u,)1 Bt)
(I.66)
E (X,, ppp u2)1 (X,, u3) 1(X,, u4) [ w w1)l
[ (Xx, w2) 202 (I.67)
p (UI((.)))) B)) 1/2
(By inequality)
& ap (U 2 B)) u2
(I.68)
j=1
To obtain the last inequality, we apply the Sauchy-Schwarzz inequality multiple times to write
IIKX, 2m) (X,, u2) | (X,, us) 2 220| 2p4 1 22) (X,, w2) /2/////2 (I.69)
2
product of moments of (X,, uأ)) and (X,, wy)] and by Lemma 1.10 each of these moments be bounded above by constant that depends on P2, P3. P4, 02. Additionally, by Lemma L.10
and Proposition 2.7.1, part (a) of Vershynin [74), we have P(K,,,, u1)l 2 B) S (B(B), since (X,, u has constant sub-exponential norm. In summary.
X; 19 an uig Ma (Xi) - 9B.B a1 Me we (X)11 (ILIUUU) UUK.,, 2/)1 BI)
E
(I.70)
1=1
S Sprppppppeee Q(B)
(1.71)
To complete the proof, we bound the second term on the right-hand side of Eq. [62 Following the
proof of Proposition 4.4 of Adamezak al. 4 we apply Lemma[I.I]] (a variant of Theorem 3.6 of
Adamczak et al. [4]) to find that, with failure probability at most e () for all m-sparse vectors E N-1, we have
N
2N
z X
S
Vd + /mllg
(I.72)
m
Let Exuarse be the event that Eq. 1.72 holds. For convenience, let A E |RdxN the matrix whose 7th column is X,, Additionally, for any subset C [N], let As be the matrix whose columns have indices in S. If S c [N] with |S| m, then assuming Espare holds, the operator norm of As is at
106

---

most vd+ + Vm
22
up to constant factor. Thus, for any set S [M]] if Esparse holds,
Σ EES
(X, u1 pL X 12) (X, u3) p (X,,, u4)l (o w (p:: 102)
sup M1, 01
(1.73)
21 < B')
P1
sup 21 EB4-1
KXi, "1))"" 1T (X,, ua) 1r (X; ua)
P1 P14t2+P5*74
iES
(1.74)
11 S (B') 193
sup
Σ
{X,, U1)
11lealpalz 1pa (X,, U2)
PI 124
2p2
01.02 24 E8d-1
(1.75)
K us) PL PP {x,, Us)) P1 "1+22*25724
(B.c. || 2 ||]|| for 2 and x E Rd, and P1 + P2 + P3 + P4 2 2)
91
PaPPP.S.PP (B')o1t7ata312)
sup
(X,, 21 )11 + (xx,, u2)1
SES
(I.76)
+ (X us) 1 + (X,, un))
(By Weighted AM-GM Inequality, with Cp. p1.P2.73.p1 constant depending Pl, P2, P3, P3.P4)
S CpI.P.PO.Pd (B') P1 42 (vā+ v log
2N |S]
(I.77)
where the last inequality is because Exparse holds.
Following the proof of Proposition 4.4 of Adamczak et al. 4. we now define SB = fi E [N] (X;, 2 B for some j E [4]]. Note that Sp depends on un, u2, 03, u4 and we will now show that if we select B to be a certain value which is not too large, this will still ensure that Sn is small. Observe that
Σ ((x,, 1 + (X,,, w2)1 + 1X,, uz)) + (x,,, ((,,,a))))) un)))) 1/2 iES
( iES
B²)
1/2
2
B
B
(I.78)
and
Σ (X,, +1],,(++(x(,,,, 2 (X,,, u2)12 1(x,, ug)l2) 1/2 S vd + vSS log
N S (B.c. Esparse holds)
SES2
Combining these we obtain
N
Sp S vd++
S log
(I.79)
Therefore, if we select 2 log
we can eliminate the second term on the right-hand side of
Szl
Eq. 1.79). finding that |Sal S B Combining this with Eq. we obtain
1.73 if B R log N and Esparse holds
Σ 1X u1) u2) u2) (X,, Pn un) ((,,, ko ((X,, (2))) 122)
sup 1
VESn
(I.80)
S Cpt.pr.p F1
01 tot
(vd+ viS²l log N)
(I.81)
(vá+ Vd
x Cppprapsp (B')FI P11211311
log N
(B.c. |SBl S B
B
x Cpppr.pa.p (B') p1tpot Vd.
91292
(B.c. B B2 log
107

---

Raising both sides to the (p1 + + + p1) power, we finally obtain
sup (1
Σ
u1) u2) P=) Pa) (XX, (1))) (X;, w2))
iESn
(I.82)
I (p (B))nt estratratiss
(I.83)
with probability - (()())) where Cy PP..P a constant which depends on P1, P2, Ps, P4-
In summary, assuming Esparse holds and the high-probability bound on U holds (and these both hold simultaneously with probability at least (V(()) we obtain
+
d + Cpp.pi.p..
U S poly(p1,P2, Pa, pa, Bortratnanny (B)) (log d)2
(B')u+
N
N
(I.84)
92 -BBB
(I.85)
by Eq. [.62p. Finally, by Eq.
1.33 we have
sup W1 My 6
(Xi, ui))"
"x' wi) 801 (Xx,, w22) 1
(I.86)
E X~vaga-11
(X,u)))1) (X, uz) (2) us) 13 ('X, ua) |P (X, wn) m (X, w22 92
(I.87)
U Cp e Q(BB)
(I.88)
as long as the events Ej and Eg hold, and these events simultaneously hold with probability at least 1 d(((ogaa Thus, by our above bound on U we have
(X, (I),s P²) (x)) ua)l wn) (X,, w22)
sup 4g Sdd-1
(I.89)
22) (X, us) us) (X,
1XX un) (X, wi) 10 |X w2)
Xavidga1
(I.90)
S poly(p1, p2, ps. P4, Beitpttsstn (B*)o++0)) (log d)2
(I.91)
Po (B'
((
(I.92)
-m 2(B")
(I.93)
108

---

Setting B' = (log d) and B 2 max(log N, (log d)?) x (log d)² (recall that N < dc for universal
constant C)2 we have
Σ (X, (2) uz) ug) (*) w1) (X,,, w2) /
sup Mi."M: 1
(I.94)
E X~vaas
u2) uz) "3) [ un) | (X, wi> m (X, (02) 92
(I.95)
d
Pappara (log d) dl + t taa)
(I.96)
N
Cpr.po PP.....--0002 d d)
pp (log (log +()))
N
(I.97)
with probability at least dfillng ay as desired.
Lemma L.7 (Corollary for Uniform Samples from gd-1). Let EN be i.i.d. random vectors sampled uniformly from Sd-1, and let s..P2.3,,Mo Q1 q2 be nonnegative integers. Suppose p
p2 P3 P4 2 Then, for fixed W1, w2 E gd-1, as long as N < dc for any universal constant
C > 0, we have
N
sup
Σ
Ka 11) u2) (2 "a) Ka,, uz) (M) (ar,, " W22]
N
41, Mg, My EBd
(I.98)
KK,, un u1) (2) (ar, uz) uz) (ar, u4) (00,,,, wi) Kar, w2)1 (0)
E Sm-
(I.99)
(Cpr (c (log d) O(n tratsstt + 21 F2 24 1e
N (I.100)
Cpepp. (log d) (Qa d 12 1
CpI. Pa.. d d)
(1.101)
with probability at least pactio dy
Here, Cp..pa.ps.P.nn is constant which depends on P1,P2 p3, p4, qt q2, and CrI.p constant which depends on pı, P2; 12,13,3 P4-
Proof of Lemma 1.7 This is direct corollary of Lemma 1.6 obtained by dividing both sides of Lemma 1.6 by
Lemma L8 (Moments for Truncated Dot Products with Random Spherical Vectors), Let be
positive integer, and for = 1,..., P, let E Sd-1 and Pa be nonnegative integer. Additionally. let
B 121 1. Finally, let X be random vector drawn uniformly from v Vdsd-1, Then, for some absolute constant e we have
P
P (X, u.))) < C Ca....pp..c
E
(X.
(x.
-E X
I
(I.102)
1=1
where PB is defined as
t
t [
(B(()
B
t> B
(I.103)
B
t B
and Op....pp is constant which depends only Pi.... PP.
109

---

Proof of Lemmal 1.8 in Adamezak al.
We use argument similar to one used in part of the proof of Proposition 4.4 First, for E [P], we can write
u.) ( u)) + (K(x,))) u4) BP (X,u.)2B
(I.104)
Thus, in the expression
P I u.)l II (eB((X,)))
(L.105)
if expand the first product using Eq. (.104), then obtain terms of the form Il F. where F; is either ( IP) or (I(X, ux) pt BP (I(X,, ua)1 2 B). Observe that the term where F. = ( u,)) for all [P] cancels with the second term in Eq. (I.105). Thus, in order to obtain an upper bound on
P u)) - II [
P
E
I
X
u.)))
(I.
suffices to upper bound the terms Il. Fi where at least one of the F3 is (I(X, ui)l BP) (L)))(((X, u,)l 2 B). Observe however that
()(X, u.)l BI,XX u 2 B) & (X, (' u,) 2 B).
(L.107)
Thus, because 1 (X, u.)) - (X, u.) in order to upper bound the terms II F; where least one of the P: is (I(X, us) BP, ( (1(( 2 B), it suffices to obtain an upper bound on the expected value of
1((X, B for some i E (P)) II (X, u. S Σ
(I(X, w 2 B) II (X, u.)l
(1.108)
Without loss of generality, let bound the expected value of the first term in the sum above:
P
E un)l B)J E III (X,,) 1/2
F 1(KX, u1)1 2 B) II 1m)
(By the Cauchy- -Shhh--Schwaz Incquality)
P EI(((,, un)l 2 B) II EIXX, uy) 2 p:] P1|z4t
(By applying the Cauchy- inequality P 1 times) pr EiC(KX u1)l 2 B1
(I.109)
where the last inequality is by Lemma I.10 and Proposition 2.7.1 of Vershynin 74 Finally, since the sub-exponential norm of (X, u1) is at most absolute constant by Lemma I.10 we have that
B[I(X, w1 B = P([(X, un
e
(1.110)
X
for some absolute constant
0. Thus, by Eq. 11.009) obtain
E X
uill > B)
II (X, u))
S C Cp -cB
(1.11I)
for some absolute constant > 0. This completes the proof.
Lemma 19, Let N, d N. Let XI...,,N, i.i.d. random vectors drawn uniformly from v Vdsd-1 Additionally, let Ei,..., EN be i..d. Rademacher random variables. Then,
N
V
(1.112)
X.
1=1
110

---

Proof of Lemma 1.9
We first bound the second moment of the norm:
N
E
E
(x,(X,,)))
(1.113)
X,6
N
x.
N
E
|x
2
(B.c. B[GG]] 0 for i + j)
N
(B.c. Xill2 vd with probability 1)
N
Nd
(I.114)
N2
d
(L.115)
N
(xx,. N EN a: (3) 1/2 S SV/ as desired.
Thus, Ex,e/ Eimn Gx.] <
N.
Lemma I.10 Lub-Exxoneettil Norm of Uniform Distribution on vdgd-1). Assume d 2 C. fooor some sufficiently large universal constant C. Then there exists an absolute constant K such that, X is random vector drawn uniformly from vdBu-1 and u E gt-1 is a fixed vector, then ((X,v)li, < K, where ||. ||,, denotes the sub-exponential norm of a random variable. (See Definition 2.7.5 of Vershynin (74).) As a corollary, for any p 2 L, there exists a constant Cp depending on p such that for any Sd-1, 1d-1, -Exxvass- (X, u) Cp.
Proof of Lemma 1.10 Let X be drawn uniformly random from the uniform distribution on ddsd-1 and let E gd-1, Observe that the distribution of va (X, v) is pa- Thus, the density of V8 (X, v) p(t) ( (1 - (2) - = by Eqs. (1.16) and (1.18) of Atkinson and Han 12
r(d/2)
By change of variables, the density of (X, v) is
T(d/2)
1 1
p(1) VFI((d - 1)/2) 1))2)
(1
2
(1.116)
d
Thus, for any & 2 0,
0
P(K 8) 12 22)
p(s)ds
(1.117)
T(d/2)
52 422
1
25
2
ds
(1.118)
(TT((d 1000111 1)/2)
d
v d
52 2=3
24
ds
(By Stirling's formula)
2A
d=3 T ds
(B.c. 1+x.e+)
27
e 5 ds
(B.c.
d-3 2d
2
1 for d sufficiently large) 3
Thus, for some absolute constant C > 0 and all t f22 0,
P([[X, v) 2 t) SC J. ds S C f e e- ds = C(
100
3C
log ac
t e Se 3
(I.119)
Suppose / > (18C)1/3, Then, 1 2 log 3C meaning that - 1 log
30
- and therefore
12 6
P([(X, wi1l 2 t) Se-
(I.120)

---

Thus, by the definition of Saub-Gassiin norm (Definition 2.5.6 and Proposition 2.5.2 of Vershynin [74], the nab-Gaussian norm (X, v) is bounded by a constant, i.e. for any v E S we have e)(x,,)lles I K for some absolute constant K >> 0 (here I1 denotes the sub-Gaussian norm of a random variable). By Propositions 2.5.2 and 2.7.1, and Definition 2.7.5, of Vershynin [74] this implies that [|(X, u)llen & 1, since sub-Gaussian random variables are always sub-exponential (specifically, using item (ii) of Proposition 2.5.2 and item (b) of Proposition 2.7.1 of Vershynin 74). The last statement of the lemma follows directly from Proposition 2.7.1, item (b) of Vershynin
Lemma 1.11 (Modification of Theorem 3.6 of Adamczak al. Let d C for some absolute constant C, and < N S e Let X1.... Xx i..dd random vectors sampled from the uniform distribution on v /dgd-1, Finally, let t 2 1. Then, for some absolute constant e O, with probability at least 1 e e ctvá for all m [N].
N
sup 2684-1
Σ
S st((vt+ v log
2N
(1.121)
m
(supp =/sm
Here, we use supp to denote the set of nonzero coordinates of 2.
Proof of Lemma 1.11 The proof is the same that of Theorem 3.6 of Adamezak et al. while Theorem 3.6 of Adamczak et al. A is shown for the case where ,I,...,X. XN are sampled from an isotropic, log-concave probability distribution, the proof can be generalized the X; are sampled from Vdgd-1 To see why, observe that the proof only uses the f'ct that the distribution of the X; is log-concave in the following places: (i) when applying Lemma 3.1 (also of Adamczak et al. [4]), of which the conclusion is that max;5N || l2 S Co v d with probability at least 1 K for some absolute constant Co. (ii) when applying Lemmas 3.3 and 3.4 (also of Adamczak et al. 44) the proof requires that (| 1, I &b for some absolute constant v > 0 and
the case where
any y gd-1, However, property (i) trivially holds when the X; are V since |Xi||2 Vd. and property (ii) holds by Lemmal 1.10
L
I.3 Helper Lemmas
Lemma L.1., Let f: 00) - Rg such that
d
dt [112 - prf(() + (t)²
(1.122)
where Pt, q 0.frr (0) Then, for T we have
S(T) < (S()) + maxt Pt) ) exp (%/ udde) mine 9:
(I.123)
Proof.
df(1) dt
df(1)² Pt. + 4: [(1) S
maxi Pr
+ St 2 f(1)-
2f(() dt <
(1.124)
2 2
2
2
Thus, we have
d ((t) + max, Pt min, qr.
91 2
f(t)
max; Pr min, gr
(I.125)
As result, by Fact 1.13
(f" q dt
f(T) < (F(O)
max; Pt min; gr
exp
(I.126)
2
Taking the square of both sides finishes the proof.
1 112

---

Fact I.13 (Gronwall's Inequality [33.. Let a < b, and let f [a, b] + Rzo and g [a, + R be
differentiable functions, such that
((()) S (()S(()
(1.127)
for all t E (a,b). Then, for all t [a, b). f(t) - f(a) exp(s" g(s)ds). Additionally, suppose f.g satisfy
(1(1) 2 (1)S(()
(I.128)
for all L E [a, b). Then, for all (a, b}. [(() 2 exp(" g(s)ds).
Proposition 1.14. For any 82, such that 0 5 32 $ 31 B2 S 1, there exists a one-dimensional distribution μ that satisfies
E
f2,
(I.129)
wm
E [
Bar
(I.130)
wm
supp(u) S 1..
(L.131)
Proof. Let p= B3/B2. By the assumptions we have 0 p
Consider the distribution
82 8
1
32
(xx)
3
3//2
1/2 +
B4
6z=0=
8
(I.132)
where dz=r denotes the Dirac measure at x = t. First all, since 83 s 34, the distribution p(x) is
well-defined. And can verify that
E
[
8
(5)
1/2/33 32,
(I.133)
B1
32
E
[w'] =
(B4 B2 //211= =B1-
1/2
(I.134)
B4
Note that 0 S 54 S B2 implies 2//2,, 3/ 1/2 E [0, 1]. Consequently,
supp(n) C 1-1.1
(I.135)
Lemma I.15. Let p be a distribution which rotationally invariant in Definition 4.1 Let V (p) be defined as in Eq. (3.4) and let V L(p) denote the last (d-1) coordinates of V (()) Then, for any particle u (w, 2l, have that V. Lp) scalar multiple of z. In particular, L is defined as in Eq. (3.4) and grad L(p) denotes the last (d 16-1) coordinates of L(p). then
grad. (p) is scalar multiple of z.
Proof. From Eq. 3.4). we have
Vab(p) = E ((())))((z))o x)x]
(I.136)
Since p is rotationally invariant, af a' E 1 such that (e1, x) (e1,,'), then fp(x) - y(x) fo(x') y(²)). Thus, (fp(z) v(=))o'(u') (TTT x) only depends on (e1, .)) and (u, x).
Let Pe,.u Iaxd denote the projection matrix into the subspace spanned by e1 and Then,
2- [_,(((( y(=))o' (u 2)(2 Pe,,.))]
E
(I.137)
because if u E gd-1 is orthogonal to é and u, then the distribution of (v, x) conditioned on (e1. and (u, x)) is symmetric around Thus,
VaL(p) = E z- [[f((()())))))))))))))()))))[[ - y()))o((u P) = Pe (V.L(()
(I.138)
and we have that v.T(p)) is in the subspace spanned by and In other words, there are scalars Gg Cg such that
V.L(p) c101 Czu and taking the last (d 1) coordinates of both sides of this equation gives V VaL(p) = C22, as desired. The last statement of the lemma holds because grad, T.(p) (T uu T V V.((p) which also linear combination of e and
(I.139)
113

---

Lemma 1.16. Let P be a polynomial of degree at most k > O, where k is bounded by universal constant, and let wi. W2 E [-1,1]. Suppose the coefficients of P are bounded by B. Then, (P(w1) P(w2)] S B B|
Proof. Let c; be the coefficient of the 1t degree term in P. Then, the lemma follows from expanding P(wi) P(wz) as
P(w1) P(wz)]
w w
(I.140)
i=0
cey [1 w2
[22 H-)
(I.141)
5=0
(2 1)
lu1 w2
(B.c. |11 [22 S 1)
==0
k
B(k +
W1 W2
(I.142)
B(k + 1) W1
W2
(1.143)
2 Blw1 W22
(B.c. k is at most an absolute constant)
as desired.
Lemma 1.17 (Powers of Dot Products on 1d Let u1, u2, u3, u4 u5 Sd-1 and q. 8, 1.
Then,
(12,, x) (us, 2)11 S Opanan d(p
11.144)
Here, Oparss a constant depending on p, 8,T,
Proof of Lemma 1.17 By repeated applications the Cauchy-Schwarc incquality, we have
E
[1111 (ue, (us, X3,, x)
(I.145)
X~V/U8--1
$
K(,, 12/2/
E Xnvagaa
[Kuu, ( X) x2 (us,, x) ( 22
Xn
(1.146)
[311,
E Xnv 13
E X~vadg
[|
( X (us,
X
(I.147)
11 X-Vddg--
E
X~ K2)4,
E
E
I
15/2/8
X
(I.148)
E
[lu4, (15,
(1.149)
E X~vaga
KK,,
E X~vaaga
E X-vaga
[K 1/////88
(L.150)
E / [Mus, X 16
X~ [l x) 2661//11
(1.151)
where each line results from a single application of the Cauchy-Schware inequality. Thus, by Lemma I.10, we have
X~vage- (u X) (1,, XX) (Y) S Cpanaet
(I.152)
Dividing both sides by d( we obtain
E -- [K () 2) (14, x)1²1 (0),u Cpansi Nepotrtoto//2
(I.153)
as desired.
114


Reviews:
[{"review_id": "MfZIYlw2Op", "reviewer": "Reviewer_aN4G", "summary": "Analyzed the (projected) gradient flow dynamics of a two-layer neural network in the mean-field regime in learning a specific degree-4 single-index target function. The main contribution is a polynomial-time convergence guarantee and a sample complexity that outperforms kernel methods. This differs from the naive mean-field analysis where the network width can grow exponentially. ", "questions": "I would be happy to update my evaluation if the authors could address the concerns and questions in the Weaknesses section.  \n\n#####################**Post-rebuttal Update**#####################\n\nThe authors addressed some of my concerns; I have therefore increased my score to 5.  \nIn the revised manuscript, please include a detailed comparison against relevant prior works, as done in the rebuttal. ", "limitations": "See weaknesses. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Most existing mean-field analyses of two-layer neural network focus on the optimization aspect, with the goal of showing global convergence of the training loss, but the generalization properties and sample complexity in learning certain class of target functions are largely unknown. The only exception that I know of is (Abbe et al. 2022). So this submission definitely tackles a challenging and interesting problem. On the technical side, the analysis differs from many previous works on learning single-index model with two-layer neural network, where the link function mismatch is typically handled by training the second-layer parameters. ", "weaknesses": "\nI have the following concerns. \n\n1. The problem setting is restrictive and convoluted, which is rather underwhelming given the promise of a *clean mean-field analysis* with no unnatural modifications. (i) The training algorithm is the unmodified projected gradient flow, but time discretization is not studied. (ii) The target function is assumed to be an even degree-4 function which is quite restrictive. Moreover, the trained neural network also uses a somewhat unnatural degree-4 activation function, and the Legendre coefficients are restricted by Assumption 3.2. The fact that this submission is lengthy and technically demanding despite these strong assumptions raises the question of whether the same analysis can be extended to more general problem settings. \n\n2. The comparison against prior works is not sufficient, and consequently, the significance of the results cannot be easily evaluated.  \n* In the context of learning single-index model, the sample complexity of gradient descent is decided by the information exponent from (Ben Arous et al. 2021). This definition is originally defined in the analysis of online SGD, but the authors should discuss whether the same mechanism is present in the ERM setting. Specifically, my current reading is that this submission assumes an information exponent of 2, due to the factor of $\\sigma_2^2\\gamma_2$ in the denominator of the stopping time $T_*$ in Theorem 3.3.  \nPlease clarify if this is the case.   \n(Ben Arous et al. 2021) *Online stochastic gradient descent on non-convex losses from high-dimensional inference* \n\n* Related to the previous point, it is also well-known that trained two-layer neural network can outperform kernel models when the link function is a high-degree polynomial with low information exponent. For example, the staircase functions in (Abbe et al. 2022) can be learned in linear sample complexity. While these prior results typically employs a layer-wise training procedure, the similarity in the main message may undermine the significance of the theoretical results in this submission. Therefore, I feel that the authors should use more space to highlight the technical differences and challenges (no bias units and retraining of second layer, etc.).   \n(Abbe et al. 2022) *The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks* \n\n* The simplification of population dynamics into some low-dimensional object via symmetry has appeared in many prior works, such as the dimension-free PDE in (Abbe et al. 2022) and (Hajjar and Chizat 2022).  \nThe authors need to explain the difference and new ingredients in Section 4.1.  \n(Hajjar and Chizat 2022) *Symmetries in the dynamics of wide two-layer neural networks* \n\n* The claim that prior mean-field analysis only proved \"generic exponential growth rate in the coupling error\" is not entirely accurate. For the noisy gradient descent setting, recent papers have provided uniform-in-time propagation of chaos estimates, for example see (Chen et al. 2022) (Suzuki et al. 2023). These results hold for finite-width network and discrete-time algorithm, but the current submission only handles the finite-width error. It would be a good idea to comment on the possibility/difficulty of  handling the discrete projected gradient descent algorithm.   \n(Chen et al. 2022) *Uniform-in-time propagation of chaos for mean field Langevin dynamics*   \n(Suzuki et al. 2023) *Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very helpful comments, and for noting that our submission “definitely tackles a challenging and interesting problem.” We now respond to each of the reviewer’s concerns.\n\n# Discrete Time\n\nWe were able to extend our results to time discretization, and we will include this result upon revision. Specifically, we can prove that projected gradient descent with 1/poly(d) step size and a poly(d) width neural network can achieve low population loss in poly(d) iterations as a relatively direct extension of our existing results (please see the next paragraph).\n\nTo extend to discrete time, we emulate standard bounds on the discretization error of Euler’s method. To bound the error per time step, we bound the smoothness of the empirical gradient, using concentration bounds similar to those we used in the analysis of the empirical dynamics.\n\n# Generality of Activations\n\nWe agree with the reviewer that our activation/target function deviates from realistic settings. We chose to keep the activations and target functions simple to not distract from our main objective of studying unmodified gradient descent in a non-trivial setting which requires avoiding bad stationary points. We note that while many works such as Abbe et al. (2022) study more general activation functions, they use modified algorithms (such as the layerwise/two-stage algorithm used by Abbe et al. (2022)) which avoid the question of whether gradient descent (GD) can avoid bad stationary points. To our knowledge, our work is the first to study unmodified GD in the mean-field regime beyond linear/quadratic activations.\n\nAlso, the framework that we use for analyzing the population gradient flow can potentially be generalized to higher-degree activations. We divide the population gradient flow into three phases, as described in Section 4. Phases 1 and 2 would likely have very similar analyses even if the activations/target function had higher degree. Analyzing Phase 3 would be more challenging, since the velocity function would have more roots.\n\nSeparately, we believe our techniques for coupling error of the finite-width/finite-sample trajectory can generalize to higher-degree polynomial activations with relatively modest effort, to obtain sample complexity better than NTK.\n\n# Comparison to Ben Arous et al. (2021)\n\nIn the setting of Ben Arous et al. (2021), the goal is to learn a student neural network which has a single neuron - this is significantly simpler than our setting where we consider a neural network with $\\text{poly}(d)$ width. It is plausible that the sample complexity of GD in our setting is also $d (\\log d)^{O(1)}$, since the lowest order term in the Legendre decomposition (which is comparable to the information exponent for the case of spherical data) of our activation and target functions has degree 2. We leave the tight sample complexity for GD as an open question for future work.\n\n# Comparison to Abbe et al. (2022)\n\n### We study how gradient descent can escape bad stationary points - Abbe et al. (2022) use a two-stage algorithm, only training the hidden layer for O(1) time, which bypasses this challenge. \nAbbe et al. (2022) use a two-stage algorithm for the dimension-free population dynamics. They train the first layer for O(1) time, and show no guarantees on the error obtained by training the first layer. They then train the second layer weights, which is a kernel regression problem.\n\n### We have a much tighter bound on the coupling error which allows us to analyze the training for O(log d) steps, while Abbe et al. (2022) have a much looser error bound which is super-exponential in the number of steps and could only work with O(1) training time. \nTechnically speaking, in Abbe et al. (2022) the upper bound on coupling error is based on a loose Lipschitzness bound (also see Theorem 1B from Mei et al. (2019)), while we tighten the bound by comparing the growth of the error with the growth of the signal.\n\n### We do not use fresh samples in every iteration, unlike Abbe et al. (2022) - ours is a more realistic setting. \nThis setting causes the weights to be correlated with the samples, thus requiring a more intricate induction on the coupling error together with a uniform concentration bound from Adamczak et al. (2010).\n\n### Our target function does not satisfy the merged-staircase property (MSP). \nThe lowest degree term in our target function has degree 2. The more recent work of Abbe et al. (2023) generalizes the MSP to functions with more “leaps,” but they apply non-standard projection steps while training the first layer (separating the coordinates into large and small coordinates and applying different projections to each subset), rather than studying unmodified GD.\n\n# Uniform-in-time propagation of chaos\n\nWe will cite Suzuki et al. (2023) and Chen et al. (2022) (though the work of Suzuki et al. (2023) was posted after the NeurIPS deadline). **However, it is likely highly non-trivial to apply current analyses of mean-field Langevin dynamics to obtain good test error and sample complexity.** Assuming Theorem 4 of Mei et al. (2018) is tight, the inverse temperature $\\lambda$ has to be at least proportional to the dimension $D$ for Langevin dynamics to achieve good test error. However, this causes the log-Sobolev constant in Suzuki et al. (2023) to be $e^{-D}$, and hence their runtime is $e^D$. In comparison, **we are able to extend our techniques to discrete-time projected gradient descent with $\\text{poly}(d)$ iterations.** Also, Chen et al. (2022) do not study finite data or discrete time.\n\n# Dimension-free dynamics\n\nWe do not claim that our reduction to one-dimensional dynamics is novel. The main novelty of our work is in our convergence analysis of the 1-dimensional dynamics and the coupling error between the population and empirical dynamics.\n\n# References\n\n(Adamczak et al. 2010) arxiv ID: 0903.2323\n(Mei et al. 2018) 1804.06561\n(Mei et al. 2019) 1902.06015\n(Abbe et al. 2023) 2302.11055"}, {"review_id": "qLoJDUd3Lr", "reviewer": "Reviewer_c2Ri", "summary": "This paper studies the statistical efficiency of the projected gradient dynamics on the sphere for (polynomial-width) two-layer neural networks under the mean-field regime. In particular, this work proves the sample complexity of $O(d^{3.1})$ for learning the single-index model with an unknown quartic link function.", "questions": "Could you specify the dependence of $m$ stated in the next sentence of Lemma 5.1: ``network width $m$ is sufficiently large polynomial of $d$?'' What does the highest degree of this polynomial depend on?  ", "limitations": "The limitation of the paper is well addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The results can be of interest to the readers and are technically sound. The remarkable contributions compared to most related works are that this work\n\n- studies not unnatural optimization dynamics but the standard projected gradient flow on the sphere,\n\n- shows the separation between two-layer neural networks in the mean field regime and kernel methods by providing the sample complexities.\n\nTechnically, the improvement in particle complexity is also worth noting. In a naive way, the approximation accuracy to the distribution by using finite particles exponentially deteriorates in time, but this study prevents such deterioration by making use of the problem and model structures. Hence, I think this work certainly contributes to the context.", "weaknesses": "- This theory deals with not standard activation such as ReLU, sigmoid, and tanh but a special type of activation function (i.e., fourth-order polynomial).\n\n- In my understanding, Abbe et al. (2023) [2] also studies the standard optimization dynamics and shows the superiority of the mean-field regime against linear models. A detailed discussion of the relationship to [2] would help clarify the position of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and for noting that our “results can be of interest to the readers and are technically sound.” Below we address the reviewer’s questions and concerns.\n\n# Generality of Activation Function\n\nWe agree with the reviewer that our activation/target function deviates from realistic settings. We chose to keep the activations and target functions simple to not distract from our main objective of studying unmodified gradient descent in a non-trivial setting which requires avoiding bad stationary points. We note that while many works such as Abbe et al. (2022) study more general activation functions, they use modified algorithms (such as the layerwise/two-stage algorithm used by Abbe et al. (2022)) which avoid the question of whether gradient descent can avoid bad stationary points. To our knowledge, our work is the first to study unmodified gradient descent that goes beyond linear/quadratic activations.\n\nAlso, the framework that we use for analyzing the population gradient flow can potentially be generalized to higher-degree activations. We divide the population gradient flow into three phases, as described in Section 4. Phases 1 and 2 would likely have very similar analyses even if the activations/target function had higher degree. Analyzing Phase 3 with higher-degree activations/target functions would be more challenging, since the velocity function would have more roots.\n\nSeparately from our techniques for analyzing the population dynamics, we also believe our techniques for analyzing the finite-width and finite-sample setting, by coupling the infinite-width/population loss and finite-width/empirical loss trajectories, can generalize to higher-degree polynomial activations with relatively modest effort, to obtain sample complexity better than NTK.\n\n# Comparison with Abbe et al. (2022)\n\n### We study how gradient descent can escape bad stationary points - Abbe et al. (2022) use a two-stage algorithm, only training the hidden layer for O(1) time, which bypasses this challenge. \nAbbe et al. (2022) use a two-stage algorithm for the dimension-free population dynamics. They train the first layer for O(1) time, and show no guarantees on the error obtained by training the first layer. They then train the second layer weights, which is a kernel regression problem.\n\n### We have a much tighter bound on the coupling error which allows us to analyze the training for O(log d) steps, while Abbe et al. (2022) have a much looser error bound which is super-exponential in the number of steps and could only work with O(1) training time. \nTechnically speaking, in Abbe et al. (2022) the upper bound on coupling error is based on a loose Lipschitzness bound (also see Theorem 1B from Mei et al. (2019)), while we tighten the bound by comparing the growth of the error with the growth of the signal.\n\n### We do not use fresh samples in every iteration, unlike Abbe et al. (2022) - ours is a more realistic setting. \nThis setting causes the weights to be correlated with the samples, thus requiring a more intricate induction on the coupling error together with a uniform concentration bound from Adamczak et al. (2010).\n\n### Our target function does not satisfy the merged-staircase property (MSP). \nThe lowest degree term in our target function has degree 2. The more recent work of Abbe et al. (2023) generalizes the MSP to functions with more “leaps,” but they use a layerwise training algorithm, and apply non-standard projection steps while training the first layer (separating the coordinates into large and small coordinates and applying different projections to each subset). Thus, Abbe et al. (2023) does not study unmodified gradient descent.\n\n# Precise Order of Width\n\nUp to logarithmic factors and other factors depending on the Legendre coefficients of the activation, the width m can be $d^{3 + \\gamma}$, where $\\gamma$ can be any arbitrarily small, but positive, universal constant. We will explicitly mention it in the revised version.\n\n# References\n\n(Adamczak et al. 2010) Quantitative estimates of the convergence of the empirical covariance matrix in Log-concave Ensembles\n\n(Mei et al. 2019) Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit\n\n(Abbe et al. 2023) SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics"}, {"review_id": "fAtI754zj1", "reviewer": "Reviewer_ZhLf", "summary": "This paper studied the projected gradient flow on two-layer neural networks in the mean-field regime with polynomial width and quartic activation function. With data sampled uniformly on the sphere, the authors proved that to learn a single-index model with an even quartic link function, this neural network needs $n=O(d^{3.1})$ training samples. However, we know for any inner product kernel ridge regression requires $n\\gg d^4$ to learn a quartic target function.  This provides a concrete example of practical feature learning of neural networks outperforming the kernel methods and NTK regime when the target function has a low-rank structure.", "questions": "1. [Ba et al. 2022] also study the feature learning and single-index model when $n\\asymp d \\asymp m$ with two-stage training processes.  And [Arnaboldi et al. 2023] and [Berthier et al. 2023] also studied the gradient flow dynamics of two-layer neural networks in high-dimension to a single-index model. Some comparisons with the current paper could be made.\n2. In line 88, you mentioned that a similar coupling error was shown in [49] but [49] is for random feature regression with proportional limit. Can you explain more about it?\n3. Line 147 typo.\n4. In Theorem 3.4, the sample complexity is $n\\ge d^\\mu$. Is there an explicit formula or upper bound for the power $\\mu$?\n5. In Theorem 3.5, you considered the limitation of the inner-product kernel with data points drawn from the unit sphere. Does the data distribution have to be uniform unit sphere distribution? Can Theorem 3.5 be extended to a general rotational invariant kernel or Euclidean distance kernel?\n6. Line 233 double \"Appendix\"\n7. Line 274 $w$ should be $w_t$?\n8.  In the analysis, Phases 1 and 2 are controlled by the quadratic components in the activation and target functions, and the dynamics are analogous to a power method update. Is there any relation between this and the PCA warmup proposed by [Chen and Meka 2020] before SGD training?\n9. In Assumption 3.2, we need $\\gamma_4\\ge 1.1\\gamma_2^2$. Why do we need this assumption and coefficient $1.1$? What if $\\gamma_2=\\gamma_4=1$?\n\n===============================================================================================\n- Chen, S. and Meka, R., 2020, July. Learning polynomials in few relevant dimensions. In Conference on Learning Theory (pp. 1161-1227). PMLR.\n- Ba, J., Erdogdu, M.A., Suzuki, T., Wang, Z., Wu, D. and Yang, G., 2022. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35, pp.37932-37946.\n- Arnaboldi, L., Stephan, L., Krzakala, F. and Loureiro, B., 2023. From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks. arXiv preprint arXiv:2302.05882.\n- Berthier, R., Montanari, A. and Zhou, K., 2023. Learning time-scales in two-layers neural networks. arXiv preprint arXiv:2303.00055.", "limitations": "I think the authors have adequately addressed the potential negative social impact of their work. The conclusion lays out some suggested next steps and in doing so highlights certain current limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, I found the paper well-written and easy to follow. The paper presents interesting insights into more accurate models of neural network training. The results obtained by the authors are as sharp as can be reasonably expected given the problem in certain cases. Unlike previous analyses of feature learning with two-stage training procedures, this work shows that neural networks can learn both feature direction and nonlinear link function when training in a more practical way with the projected gradient flow. The methods of the proof may provide further insights to obtain more general results on feature learning.", "weaknesses": "1. The main limitation is that the activation function and the target link function are special, only quartic functions. It would be nice to check if the proof techniques can handle more general activation and target functions.  Another thing that can be improved is the precise order of the width: how wide the neural network is sufficient for this feature learning.\n2. There should be a comparison between the results of the current submission and [10]. [10] uses information exponent to determine the sample complexity for online SGD to learn a single-index target. When $\\gamma_2=0$ in Assumption 3.2, [10] implies that sample size should be $n\\gg d^{3}\\log n$ which is similar to the results in the current submission. Further explanations are needed.\n3.  And it should be more convincing to visualize Phases 1-3 of the training dynamic in Section 5 if there is some simulation of synthetic data or real-world data for this finite-width training dynamic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and for noting that our work “presents interesting insights into more accurate models of neural network training.” We now address the reviewer’s questions. We will incorporate presentation-related comments in the revision and include a simulation to illustrate Phases 1-3.\n\n# Generality of Activation Function\n\nWe agree that our activation/target function deviates from more realistic settings. We keep the activations and target functions simple to not distract from our main objective of studying unmodified gradient descent in a non-trivial setting which requires avoiding bad stationary points. We note that while many works such as Abbe et al. (2022) study more general activation functions, they use modified algorithms which bypass the question of whether gradient descent (GD) can avoid bad stationary points. To our knowledge, our work is the first to study unmodified GD in the mean-field regime that goes beyond linear/quadratic activations.\n\nAlso, the framework that we use for analyzing the population gradient flow can potentially be generalized to higher-degree activations. We divide the population gradient flow into three phases, as described in Section 4. Phases 1 and 2 would likely have very similar analyses even if the activations/target function had higher degree. Analyzing Phase 3 with higher-degree activations would be more challenging, since the velocity function would have more roots.\n\nSeparately from our techniques for analyzing the population dynamics, we also believe our techniques for analyzing the finite-width and finite-sample setting, by coupling the infinite-width/population loss and finite-width/empirical loss trajectories, can generalize to higher-degree polynomial activations with relatively modest effort, to obtain sample complexity better than NTK.\n\n# Precise Width\n\nUp to logarithmic factors and other factors depending on the Legendre coefficients of the activation, the width m can be $d^{3 + \\gamma}$, where $\\gamma$ can be any arbitrarily small, but positive, universal constant.\n\n# Comparison with Ben Arous et al. (2021)\n\nBen Arous et al. (2021) consider a single-neuron student network - this is significantly simpler than our setting where we consider a neural network with $\\text{poly}(d)$ width. Still, it is plausible that the sample complexity of GD in our setting is also $d (\\log d)^{O(1)}$, since the lowest order term in the Legendre decomposition (which is comparable to the information exponent for the case of spherical data) of our activation and target functions has degree 2. We leave the tight sample complexity for GD as an open question for future work.\n\nOur Assumption 3.2 excludes the case $\\gamma_2 = 0$, due to the condition that $\\gamma_4 \\leq c_1 \\gamma_2^2$. Our analysis of Phases 1 and 2 makes use of the fact that $\\gamma_2$ is nonzero because the particles grow uniformly in magnitude while the second-order term is dominant.\n\n# Other Related Works\n\nBa et al. (2022) perform one step of gradient descent on the hidden layer, followed by linear regression to fit the second layer. Meanwhile, we train the hidden layer for a long period of time.\n\nArnaboldi et al. (2023) bound the error between dimension-free dynamics and the true dynamics. We also obtain one-dimensional dynamics, but we do not claim that it is novel. Rather, our main contributions are our convergence analysis of population dynamics (Section 4.2) and of coupling error between population and empirical dynamics (Section 5).\n\nBerthier et al. (2023) obtain a super-exponential coupling error, which gives a super-polynomial coupling error in our setting.\n\n# Exponent of sample complexity\n\nIn the sample complexity $d^\\mu$, we can set $\\mu = 3 + \\gamma$ where $\\gamma$ is a positive, but arbitrarily small, universal constant.\n\n# Data Distribution for Kernel Lower Bound\n\nWhile our current proof requires a uniform distribution over the unit sphere, we believe that our proof technique is quite general and can be used to prove similar results for other data distributions. Essentially we only require that the minimum eigenvalue of the matrix $(K(x_i, x_j))_{i,j\\in [n]}$ concentrates to its mean with constant probability, and $K(x_i,z)^2$ concentrates to its mean for a fixed $z$. Thus, we leave the technical details as future work.\n\n# PCA Warmup\n\nOur phases 1 and 2 are not directly related to the PCA warmup of Chen and Meka (2020) - they explicitly threshold points $x$ based on $y(x)$ to identify the low-rank subspace, while we show the neurons automatically achieve high correlation with $e_1$ in Phase 1 and 2. Additionally, in the rank-1 case, Chen and Meka (2020) perform Riemannian GD on a teacher-student setting where both the teacher and student have only one neuron and the same activation, while our model has poly(d) neurons.\n\n# Loosening assumption that $\\gamma_4 \\geq 1.1 \\gamma_2^2$ \n\nWe make use of this assumption in the analysis of Phase 3, Case 2 (omitted from main body due to space constraints). In Phase 3, Case 2, we have $D_{4, t} > 0$ and $D_{2, t} < 0$, which by the assumption $\\gamma_4 \\geq 1.1 \\gamma_2^2$, implies that the distribution $\\rho_t$ cannot have all of its mass at a root of the velocity function. Thus the population dynamics will continue to make progress. More generally, we could instead have assumed that $\\gamma_4 \\geq c \\gamma_2^2$ for any constant $c > 1$.\n\nOur proof could likely extend to the case $\\gamma_2 = \\gamma_4 = 1$ (and more generally $\\gamma_4 = \\gamma_2^2$), using a similar proof by contradiction to show that if all the particles are close to the root of the velocity, then this root must be $\\sqrt{\\gamma_2}$, or else a contradiction is obtained. We exclude this case to simplify the analysis while still having a nontrivial setting where unmodified GD achieves good sample complexity.\n\n# Citation to [49]\n\nThis was a typo - we intended to cite Mei et al. (2019).\n\n# References\n\n(Mei et al. 2019) arxiv ID: 1902.06015"}, {"review_id": "gv5vgfnHHH", "reviewer": "Reviewer_HQjX", "summary": "This paper studies the global convergence of gradient descent for training two-layer networks in learning a high dimensional quartic function. The authors show that GD converges when the sample size $n = O(d^{3.1})$ and the width of the neural network grows at most polynomially in d. The authors also showed that any kernel method with $n \\ll d^4$ cannot achieve the same order of accuracy. ", "questions": "The authors established the results for learning the quartic function. Does the proof generalize to a more general class of functions? If so, can the authors highlight the further work to be done? If not, it would be good if the authors can comment on the major challenges. ", "limitations": "None", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "First of all, this is an excellent paper on the non-asymptotic mean field analysis of GD training for two-layer neural networks and its separation from kernel method. Albeit the simplicity of the target function, the convergence analysis requests new ideas and tools. One crucial ingredient of the proof lies in showing that the projected gradient flow (population) dynamics can escape saddle points. The other key ingredient is bounding the coupling error between the empirical particle dynamics and the population dynamics. For the later, the authors tame the usual exponential growth rate in the coupling error with a delicate investigation on the relationship between the growth of the error and that of the signal. ", "weaknesses": "I do not have many comments on the weakness, but have few questions, which are mostly relevant to future considerations. See below.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review, and for mentioning that “this is an excellent paper on the non-asymptotic mean field analysis of GD for training two-layer neural networks.”\n\n# Extending to more general activations\n\nThe framework that we use for analyzing the population gradient flow can potentially be generalized to higher-degree activations. We divide the population gradient flow into three phases, as described in Section 4. Phases 1 and 2 would likely have very similar analyses even if the activations/target function had higher degree. Analyzing Phase 3 with higher-degree activations/target functions would be more challenging, since the velocity function would have more roots.\n\nSeparately from our techniques for analyzing the population dynamics, we also believe our techniques for analyzing the finite-width and finite-sample setting, by coupling the infinite-width/population loss and finite-width/empirical loss trajectories, can generalize to higher-degree polynomial activations with relatively modest effort, to obtain sample complexity better than NTK."}]

Author rebuttal:

Meta-review: The paper studies the dynamics of gradient descent for learning a single index function on the sphere. It performs a mean field analysis, which yields a generalization guarantee for O(d^{3.1}) samples and a polynomial-width network. In contrast, spherical kernel methods (including NTK) require d^4 samples to guarantee generalization. At a technical level, the analysis is able to show that gradient descent avoids suboptimal critical points, without requiring layerwise training. 

Reviewers expressed a uniformly positive evaluation of the paper’s contributions: it shows a separation between neural network training and kernel methods, with polynomial network width, without requiring unnatural modifications to the training procedure. The main limitation seems to quartic activation function, and its assumptions on the target function. Major points of discussion on review included the relationship to previous works on learning single index functions and the aforementioned limitations. After considering author responses, reviewers uniformly recommended acceptance, based on the quality and novelty of the paper’s theoretical results.
Decision: Accept (poster)