Generalization of Overparametrized Deep Neural Network Under Noisy Observations

1 INTRODUCTION Over the past few years , Neural Tangent Kernel ( NTK ) [ Arora et al. , 2019b ; Jacot et al. , 2018 ; Lee et al. , 2018 ; Chizat & Bach , 2018 ] has been one of the most seminal discoveries in the theory of neural network . The underpinning idea of the NTK-type theory comes from the observation that in a wide-enough neural net , model parameters updated by gradient descent ( GD ) stay close to their initializations during the training , so that the dynamics of the networks can be approximated by the first-order Taylor expansion with respect to its parameters at initialization . The linearization of learning dynamics on neural networks has been helpful in showing the linear convergence of the training error on both overparametrized shallow [ Li & Liang , 2018 ; Du et al. , 2018 ] and deep neural networks [ Allen-Zhu et al. , 2018 ; Zou et al. , 2018 ; 2020 ] , as well as the characterizations of generalization error on both models [ Arora et al. , 2019a ; Cao & Gu , 2019 ] . These findings clearly lead to the equivalence between learning dynamics of neural networks and the kernel methods in an reproducing kernel Hilbert spaces ( RKHS ) associated with NTK . 1 Specifically , Arora et al . [ 2019a ] provided the O ( n−1/2 ) generalization bound of shallow neural network , where n denotes the training sample size . Recently , in the context of nonparametric regression , two papers Nitanda & Suzuki [ 2020 ] and Hu et al . [ 2021 ] showed that neural network can obtain the convergence rate faster than O ( n−1/2 ) by specifying the complexities of target function and hypothesis space . Specifically , Nitanda & Suzuki [ 2020 ] showed that the shallow neural network with smoothly approximated ReLU ( swish , see Ramachandran et al . [ 2017 ] ) activation trained via ` 2-regularized averaged stochastic gradient descent ( SGD ) can recover the target function from RKHSs induced from NTK with swish activation . Similarly , Hu et al . [ 2021 ] showed that a shallow neural network with ReLU activation trained via ` 2-regularized GD can generalize well , when the target function ( i.e. , f ? ρ ) is fromHNTK1 . Notably , the rate that the papers Nitanda & Suzuki [ 2020 ] and Hu et al . [ 2021 ] obtained is minimax optimal , meaning that no estimators perform substantially better than the ` 2-regularized GD or averaged SGD algorithms for recovering functions from respective function spaces . Nevertheless , 1Henceforth , we denoteHNTK1 andHNTKL as RKHSs induced from NTK of shallow L = 1 and deep neural networks L ≥ 2 with ReLU activations , respecitvely . these results are restricted to shallow neural networks , and can not explain the generalization abilities of deep neural network ( DNN ) . Similarly with Arora et al . [ 2019a ] , Cao & Gu [ 2019 ] obtained the O ( n−1/2 ) generalization bound , showing that the SGD generalize well for f ? ρ ∈ HNTKL , when f ? ρ has a bounded RKHS norm . However , the rate they obtained is slower than the minimax rate we can actually achieve . Furthermore , their results become vacuous under the presence of additive noises on the data set . Motivated from these observations , the fundamental question in this study is as follows : When the noisy dataset is generated from a function fromHNTKL , does the overparametrized DNN obtained via ( ` 2-regularized ) GD provably generalize well the unseen data ? We consider a neural network that has L ≥ 2 hidden layers with width m n . ( i.e. , overparametrized deep neural network . ) We focus on the least-squares loss and assume that the activation function is ReLU . A positivity assumption of NTK from ReLU DNN is imposed , meaning that λ0 > 0 , where λ0 denotes the minimum eigenvalue of the NTK . We give a more formal mathematical definition of ReLU DNN in the following Subsection 2.2 . Under these settings , we provide an affirmative answer to the above question by investigating the behavior of L2-prediction error of the obtained neural network with respect to GD iterations . 1.1 CONTRIBUTIONS Our derivations of algorithm-dependent prediction risk bound require the analysis on training dynamics of the estimated neural network through ( regularized ) GD algorithm . We include these results as the contributions of our paper , which can be of independent interests as well . • In an unregulaized case , under the assumption λ0 > 0 , we show randomly initialized GD converges to 0 training loss at a linear rate . As will be detailed in subsection 3.3 , this is the different result from the seminal work of Allen-Zhu et al . [ 2018 ] , where they also prove a linear convergence of training loss of ReLU DNN , but under different data distribution assumption . • We show that the DNN updated via vanilla GD does not recover the ground truth function f ? ρ ∈ HNTKL under noisy observations , if the DNN is trained for either too short or too long : that is , the prediction error is bounded away from 0 by some constant as n goes to infinity . • In regularized case , we prove the mean-squared error ( MSE ) of DNN is upper bounded by some positive constant . Additionally , we proved the dynamics of the estimated neural network get close to the solution of kernel ridge regression associated with NTK from ReLU DNN . • We show that the ` 2-regularization can be helpful in achieving the minimax optimal rate of the prediction risk for recovering f ? ρ ∈ HNTKL under the noisy data . Specifically , it is shown that after some iterations of ` 2-regularized GD , the minimax optimal rate ( which is O ( n− d 2d−1 ) , where d is a feature dimension . ) can be achieved . Note that our paper is an extension of Hu et al . [ 2021 ] to DNN model , showing that the ` 2-regularized DNN can achieve a minimax optimal rate of prediction error for recovering f ? ρ ∈ HNTKL . However , we would like to emphasize that our work is not a trivial application of their work from at least two technical aspects . These aspects are more detailed in the following subsection . 1.2 TECHNICAL COMPARISONS WITH HU ET AL . [ 2021 ] Firstly , in the analysis of training loss of regularized shallow neural-net , Hu et al . [ 2021 ] begin the proof by decomposing the difference between two individual predictions into two terms : one that is related with the gram matrix evaluated at each iteration of the algorithm and the perturbation term . Henceforth , we name this decompostion as “ Gram+Pert ” decomposition . This decomposition can be checked with the equality ( E.2 ) in the supplementary PDF of Hu et al . [ 2021 ] . The key ingredients for the decomposition are ( i ) the simple gradient structure of the shallow neural net , and ( ii ) the partitioning of the nodes in the hidden-layer into two sets : a set of nodes whose activation patterns change from their initializations during training , and the complement of the set . This construction of the sets peels off the ReLU activation in the difference so that the GD algorithm can be involved in the analysis . However , because of the compositional structure of the network , the same nodes partitioning technique can not be applied for obtaining the decomposition in the DNN setting with ReLU activation . To avoid this difficulty , we employ a specially designed diagonal matrix Σ̃ and this matrix can peel off the ReLU function for each layer of the network . ( See the definition of Σ̃ in the proof of Theorem 3.5 in the Appendix . ) Recursive applications of this diagonal matrix across the entire hidden layers enable the Gram+Pert decomposition in our setting . It should be noted that the diagnoal matrix Σ̃ had been employed in Zou et al . [ 2020 ] , which analyzed the behavior of training loss of classification problem via ReLU DNN under logistic loss . However , since their result is dependent on different data distribution assumption under the different loss function from ours , they didn ’ t employ the Gram+Pert decomposition . Thus their technical approaches are different from ours . Secondly , Hu et al . [ 2021 ] directly penalized the weight parameter W by adding ‖W‖2F to the objective function . The ` 2-regularization solely on the W has an effect of pushing the weight towards the origin . This makes ‖W ( k ) −W ( 0 ) ‖2 ≤ O ( 1 ) 2 , allowing most activation patterns of the nodes in the hidden layer can change during the training , even in overparametrized setting . Here , W ( k ) denotes the updated weight parameter at kth itertaion of algorithm , and ‖ · ‖2 denotes the spectral norm of the matrix . Nonetheless , this doesn ’ t affect the analysis on obtaining the upper-bound of MSE in shallow neural net , since the network has only a single hidden layer . In contrast , in the DNN setting , we allow the non-convex interactions of parameters across the hidden layers . To the best of our knowledge , a technique for controlling the size of ` 2-norm of network gradient has not been developed under this setting , yet . We circumvent this difficulty by regularizing the distance between the updated and the initialized parameter , instead by directly regularizing the updated parameter . This ensures that the updated parameter by ` 2-regularized GD stays in a close neighborhood to its initialization , so that with heavy over-parametrization , the dynamics of network becomes linearized in parameter and we can ignore the non-convex interactions of parameters across the hidden layers . Specifically , under suitable model parameter setting , we prove that ‖W ( k ) ` −W ( 0 ) ` ‖2 ≤ ÕP ( 1√ m ) 3 over all ` ∈ { 1 , . . . , L } . Here , ÕP ( · ) hides the dependencies on the model parameters ; L , ω , and n. This result allows us to adopt the so-called “ Forward Stability ” argument developed by Allen-Zhu et al . [ 2018 ] , and eventually leads to the control of network gradient under ` 2 sense . 1.3 ADDITIONAL RELATED WORKS There has been another line of works trying to characterize the generalizabilities of DNN under noisy observation settings . Specifically , it has been shown that the neural network model can achieve minimax style optimal convergence rates of L2-prediction risk both in regression [ Bauer & Kohler , 2019 ; Liu et al. , 2019 ; Schmidt-Hieber , 2020 ] and classification [ Kim et al. , 2021 ] problems . Nonetheless , a limitation of the aforementioned papers is that they assume an adequate minimizer of the empirical risk can be obtained . In other words , the mathematical proofs of their theorems do not correspond to implementable algortihms . Recently , several papers , which study the generalization properties of neural network with algorithmic guarantees , appear online . Specifically , Kohler & Krzyzak [ 2019 ] showed that the data interpolants obtained through DNN by vanilla GD is inconsistent . This result is consistent with our result , but they consider the overparametrized DNN that is a linear combination of Ω ( n10d 2 ) smaller neural network , and the activation function they consider is sigmoid function , which is smooth and differentiable . Along this line of research , Kuzborskij & Szepesvári [ 2021 ] ( regression ) and Ji et al . [ 2021 ] ( classification ) showed that when training overparametrized shallow neural network , early stopping of vanilla GD enables us to obtain consistent estimators . Notation . We use the following notation for asymptotics : For sufficiently large n , we write f ( n ) = O ( g ( n ) ) , if there exists a constant K > 0 such that f ( n ) ≤ Kg ( n ) , and f ( n ) = Ω ( g ( n ) ) if f ( n ) ≥ K ′g ( n ) for some constant K ′ > 0 . The notation f ( n ) = Θ ( g ( n ) ) means that f ( n ) = O ( g ( n ) ) and 2This was empirically shown to be true in paper Wei et al . [ 2019 ] . See Figure 3 in their paper . We provide a brief mathematical explanation on why this result is hard to be shown in Appendix C. 3Readers can find the proof of this result in Appendix G. f ( n ) = Ω ( g ( n ) ) . Let 〈A , B〉Tr : = Tr ( A > B ) for the two matrices A , B ∈ Rd1×d2 . We adopt the shorthand notation denoting [ n ] : = { 1 , 2 , . . . , n } for n ∈ N. 2 PROBLEM FORMULATION 2.1 NON-PARAMETRIC REGRESSION Let X ⊂ Rd and Y ⊂ R be the measureable feature space and output space . We denote ρ as a joint probability measure on the product space X × Y , and let ρX be the marginal distribution of the feature space X . We assume that the noisy data-set D : = { ( xi , yi ) } ni=1 are generated from the non-parametric regression model yi = f ? ρ ( xi ) + εi , where εi i.i.d.∼ N ( 0 , 12 ) for i = 1 , . . . , n. Let f̂W ( k ) ( · ) be the value of neural network evaluated with the parameters W at the k-th iterations of GD update rule . At k = 0 , we randomly initialize the weight parameters in the model following He initialization [ He et al. , 2015 ] with a slight modification . Then , the L2 prediction risk is defined as the difference between two expected risks ( i.e. , excess risk ) R ( f̂W ( k ) ) : = Eρ∼ ( x , y ) [ ( y − f̂W ( k ) ( x ) ) 2 ] and R ( f ? ρ ) : = Eρ∼ ( x , y ) [ ( y − f ? ρ ( x ) ) 2 ] , where f ? ρ ( x ) : = E [ y|x ] . Then , we can easily show the prediction risk has a following form : R ( f̂k , f ? ρ ) : = R ( f̂W ( k ) ) −R ( f ? ) = Eρx , ε [ ( f̂W ( k ) ( x ) − f ? ρ ( x ) ) 2 ] . ( 1 ) Note that the expectation is taken over the marginal probability measure of feature space , ρx , and the noise of the data , ε . However , the ( 1 ) is still a random quantity due to the randomness of the initialized parameters ( W ( 0 ) ` ) ` =1 , ... , L . 2.2 DEEP NEURAL NETWORK WITH RELU ACTIVATION Following the setting introduced in Allen-Zhu et al . [ 2018 ] , we consider a fully-connected deep neural networks with L hidden layers and m network width . For L ≥ 2 , the output of the network fW ( · ) ∈ R with input data x ∈ X can be formally written as follows : fW ( x ) = √ m · vTσ ( WLσ ( WL−1 · · ·σ ( W1x ) · · · ) ) , ( 2 ) where Sd−1 is a unit sphere in d-dimensional euclidean space , σ ( · ) is an entry-wise activation function , W1 ∈ Rm×d , W2 , . . . , WL ∈ Rm×m denote the weight matrices for hidden layers and v ∈ Rm×1 denote the weight vector for the output layer . Following the existing literature , we will consider ReLU activation function σ ( x ) = max ( x , 0 ) , which is the most commonly used activation function by practitioners . Random Initialization . Each entries of weight matrices in hidden layers are assumed to be generated from ( Wi , j ) ` =1 , ... , L ∼ N ( 0 , 2m ) , and entries of the output layer are drawn from vj ∼ N ( 0 , ωm ) . This initialization scheme helps the forward propagation neither explode nor vanish at the initialization , seeing Allen-Zhu et al . [ 2018 ] ; Zou et al . [ 2018 ; 2020 ] . Note that we initialize the parameters in the last layer with variance ωm , where ω ≤ 1 is a model parameter to be chosen later for technical convenience . Unregularized GD update rule . We solve a following ` 2-loss function with the given dataset D : LS ( W ) = 1 2 n∑ i=1 ( yi − fW ( xi ) ) 2 . ( 3 ) Let W ( 0 ) 1 , . . . , W ( 0 ) L be the initialized weight matrices introduced above , and we consider a following gradient descent update rule : W ( k ) ` = W ( k−1 ) ` − η∇W ` ( LS ( W ( k−1 ) ` ) ) , ` ∈ [ L ] , k ≥ 1 , ( 4 ) where ∇W ` ( LS ( · ) ) is a partial gradient of the loss function LS ( · ) with respect to the ` -th layer parameters W ` , and η > 0 is the learning rate of the gradient descent . ` 2-regularized GD update rule . The estimator is obtained by minimizing a ` 2-regularized function ; ΦD ( W ) : = LS ( WD ) + µ 2 L∑ ` =1 ∥∥∥WD , ` −W ( 0 ) D , ` ∥∥∥2 F . ( 5 ) Naturally , we update the model parameters { WD , ` } ` =1 , ... , L via modified GD update rule : W ( k ) D , ` = ( 1− η2µ ) W ( k−1 ) D , ` − η1∇W ` [ LS ( W ( k−1 ) D ) ] + η2µW ( 0 ) D , ` , ∀ ` ∈ [ L ] , ∀k ≥ 1 . ( 6 ) The notations η1 , η2 are step sizes , and µ > 0 is a tuning parameter on regularization . We adopt the different step sizes for the partial gradient and regularized term for the theoretical conveniences . Furthermore , we add the additional subscript D to the update rule ( 6 ) to denote the variables are under the regularized GD update rule . Recall that the W ( 0 ) D , ` are initialized parameters same with the unregularized case . For simplicity , we fix the output layer , and train L hidden layers for both unregularized and regularized cases . 3 MAIN THEORY First , we describe the neural tangent kernel ( NTK ) matrix of ( 2 ) , which is first proposed by Jacot et al . [ 2018 ] and further studied by Arora et al . [ 2019b ] ; Du et al . [ 2019 ] ; Lee et al . [ 2018 ] ; Yang [ 2019 ] . NTK matrix of DNN is a L-times recursively defined n × n kernel matrix , whose entries are the infinite-width limit of the gram matrix . Let ∇W ` [ fW ( 0 ) ( · ) ] be the gradient of the ReLU DNN ( 2 ) with respect to the weight matrix in ` th hidden layer at random initialization . Note that when ` = 1 , ∇W ` [ fW ( 0 ) ( · ) ] ∈ Rm×d and when ` ∈ { 2 , . . . , L } , ∇W ` [ fW ( 0 ) ( · ) ] ∈ Rm×m . Then , as m→∞ , H ( 0 ) : = ( 1 m L∑ ` =1 〈 ∇W ` [ fW ( 0 ) ( xi ) ] , ∇W ` [ fW ( 0 ) ( xj ) ] 〉 Tr ) n×n → H∞L , ( 7 ) where H∞L : = { Ker ( xi , xj ) } n i , j=1 . Here , Ker ( · , · ) denotes a NTK function of ( 2 ) to be defined as follows : Definition 3.1 . ( NTK function of ( 2 ) ) . For any x , x′ ∈ X and ` ∈ [ L ] , define Φ ( 0 ) ( x , x′ ) = 〈x , x′〉 , Θ ( ` ) ( x , x′ ) = ( Φ ( ` −1 ) ( x , x ) Φ ( ` −1 ) ( x , x′ ) Φ ( ` −1 ) ( x′ , x ) Φ ( ` −1 ) ( x′ , x′ ) ) ∈ R2×2 , Φ ( ` ) ( x , x′ ) = 2 · E ( u , v ) ∼N ( 0 , Θ ( ` ) ) [ σ ( u ) · σ ( v ) ] , and Φ̇ ( ` ) ( x , x′ ) = 2 · E ( u , v ) ∼N ( 0 , Θ ( ` ) ) [ σ̇ ( u ) · σ̇ ( v ) ] , where σ̇ ( u ) = 1 ( u ≥ 0 ) . Then , we can derive the final expression of NTK function of ( 2 ) as follows : Ker ( x , x′ ) = ω 2 · L∑ ` =1 ( Φ ( ` −1 ) ( x , x′ ) · L∏ ` ′= ` Φ̇ ( ` ′ ) ( x , x′ ) ) . ( 8 ) The expression in ( 8 ) is adapted from Cao & Gu [ 2019 ] . As remarked in Cao & Gu [ 2019 ] , a coefficient 2 in Φ ( ` ) and Φ̇ ( ` ) remove the exponential dependence on the network depth L in the NTK function . However , when compared with the NTK formula in Cao & Gu [ 2019 ] , ( 8 ) is different from two aspects : ( i ) An additional factor ω in ( 8 ) ) comes from the difference in initialization settings of the output layer , in which Cao & Gu [ 2019 ] considers vj ∼ N ( 0 , 1m ) , whereas we consider vj ∼ N ( 0 , ωm ) . ( ii ) Φ ( L ) is not added in the final expression of ( 8 ) ) , whereas it is added in the definition provided in Cao & Gu [ 2019 ] . This is because we only train the L hidden layers but fix the output layer , while Cao & Gu [ 2019 ] train the entire layers of the network including the output layer . As already been pointed by several papers Cho & Saul [ 2009 ] ; Jacot et al . [ 2018 ] , it can be proved that the NTK function ( 8 ) is a positive semi-definite kernel function . Furthermore , Cho & Saul [ 2009 ] prove that the expectations in Φ and Φ̇ have closed form solutions , when the covariance matrices have the form ( 1 tt 1 ) with |t| ≤ 1 : E ( u , v ) ∼N ( 0 , Θ ( ` ) ) [ σ ( u ) · σ ( v ) ] = 1 2π ( t · ( π − arccos ( t ) ) + √ 1− t2 ) , E ( u , v ) ∼N ( 0 , Θ ( ` ) ) [ σ̇ ( u ) · σ̇ ( v ) ] = 1 2π ( π − arccos ( t ) ) . ( 9 ) Clearly , ( 8 ) is symmetric and continuous on the product space X ×X , from which it can be implied that Ker ( · , · ) is a Mercer kernel inducing an unique RKHS . Following Ghorbani et al . [ 2020 ] , we define the RKHS induced by ( 8 ) as : Definition 3.2 . ( NTK induced RKHS ) . For some integer p ∈ N , set of points { x̃j } pj=1 ⊂ X , and weight vector α : = { α1 , . . . , αp } ∈ Rp , define a complete vector space of functions , f : X → R , HNTKL : = cl ( { f ( · ) = p∑ j=1 αjKer ( · , x̃j ) } ) , ( 10 ) where cl ( · ) denotes closure . In the remaining of our work , we assume the regression function f ? ρ ( x ) : = E [ y|x ] belongs toHNTKL . 3.1 ASSUMPTIONS . In this subsection , we state the assumptions imposed on the data distribution with some remarks . ( A1 ) ρX is an uniform distribution on Sd−1 : = { x ∈ Rd | ‖x‖2 = 1 } , and noisy observations are assumed to be bounded . ( i.e. , ρx ∼ Unif ( Sd−1 ) , yi = O ( 1 ) , ∀i ∈ [ n ] . ) ( A2 ) Draw n i.i.d . samples { xi , f ? ρ ( xi ) } ni=1 from the joint measure ρ . Then , with probability at least 1− δ , we have λmin ( H∞L ) = λ0 > 0 . Remark 3.3 . • When the feature space is restricted on the unit sphere , the NTK function in ( 8 ) becomes rotationally invariant zonal kernel . This setting allows to adopt the results of spectral decay of ( 8 ) in the basis of spherical harmonic polynomials for measuring the complexity of hypothesis space , HNTKL . See the Appendix 3.2 and references therein . • Assumption ( A2 ) is commonly employed in NTK related literature for proving global convergence of training error and generalization error of both deep and shallow neural network , Du et al . [ 2018 ; 2019 ] ; Arora et al . [ 2019a ] . Note that the ( A2 ) holds as long as no two xi and xj are parallel to each other , which is true for most of the real-world distributions . See the proof of this claim in Du et al . [ 2019 ] . 3.2 MINIMAX RATE FOR RECOVERING f ? ρ ∈ HNTKL The obtainable minimax rate of L2-prediction error is directly related with the complexity of function space of interest . In our setting , the complexity of RKHS HNTKL can be characterized by the eigen-decay rate of the NTK function . Since Ker ( x , x′ ) is defined on the sphere , the decomposition can be given in the basis of spherical harmonics as follows : Ker ( x , x′ ) = ∞∑ k=0 µk N ( d , k ) ∑ j=1 Yk , j ( x ) Yk , j ( x ′ ) , where Yk , j , j = 1 , . . . , N ( d , k ) are spherical harmonic polynomials of degree k and { µk } ∞k=0 are non-negative eigenvalues . Recently , several researchers , both empirically [ Basri et al. , 2020 ] and theoretically [ Chen & Xu , 2020 ; Geifman et al. , 2020 ; Bietti & Bach , 2021 ] , showed that , for large enough harmonic function frequency k , the decay rate of the eigenvalues µk is in the order of Θ ( k−d ) 4 . Given this result and the fact N ( d , k ) = 2k+d−3k ( k+d−3 d−2 ) grows as kd−2 for large k , it can be easily shown λj = Θ ( j− d d−1 ) , when Ker ( x , x′ ) = ∑∞ j=1 λjφj ( x ) φj ( x ′ ) , for eigen-values λ1 ≥ λ2 ≥ · · · ≥ 0 and orthonormal basis { φj } ∞j=1 . Furthermore , it is a well known fact that if the eigenvalues decay at the rate λj = Θ ( j−2ν ) , then the corresponding minimax rate for estimating function in RKHS is O ( n− 2ν 2ν+1 ) , [ Raskutti et al. , 2014 ; Yuan & Zhou , 2016 ; Hu et al. , 2021 ] . By setting 2ν = dd−1 , we can see the minimax rate for recovering f ? ρ ∈ HNTKL isO ( n− d 2d−1 ) . Remark 3.4 . We defer all the technical proofs of the Theorems in subsections 3.3 and 3.4 in the Appendix for conciseness of the paper . At the beginning of each proof , a high-level proof idea is also provided . We also provide numerical experiments which can corroborate our theoretical findings in the Appendix A . 3.3 ANALYSIS OF UNREGULARIZED DNN In this subsection , we provide the results on the training loss of DNN estimator obtained via minimizing unregularized ` 2-loss ( 3 ) and on the corresponding estimator ’ s L2-prediction riskR ( f̂k , f ? ρ ) . Theorem 3.5 . ( Optimization ) For some δ ∈ [ 0 , 1 ] , if we set the width of the network as m log3 ( m ) ≥ Ω ( ω7n8L18 λ80δ 2 ) , and we set the step-size of gradient descent as η = O ( λ0 n2L2m ) . Then , with probability at least 1−δ over the randomness of initialized parametersW ( 0 ) : = { W ( 0 ) ` } L+1 ` =1 with W ( 0 ) L+1 = v , we have for k = 0 , 1 , 2 , . . . , LS ( W ( k ) ) ≤ ( 1− ηmλ0 2 ) k LS ( W ( 0 ) ) . ( 11 ) In other words , the training loss drops to 0 at a linear rate . We acknowledge a series of past works Allen-Zhu et al . [ 2018 ] ; Du et al . [ 2019 ] have similar spirits with those in Theorem 3.5 . However , it is worth noting that their results are not applicable in our problem settings and data assumptions . Specifically , the result of Du et al . [ 2019 ] is based on the smooth and differentiable activation function , whereas the Theorem 3.5 is about the training error of ReLU activation function , which is not differentiable at 0 . Furthermore , the result of Allen-Zhu et al . [ 2018 ] relies on φ-separateness assumption stating that the every pair of feature vectors { xi , xj } n i 6=j is apart from each other by some constant φ > 0 in a Euclidean norm . In our work , the positivity assumption on the minimum eigenvalue of the NTK is imposed ( i.e. , λ0 > 0 ) . Remark 3.6 . Reducing the order of network width is definitely another line of interesting research direction . We are aware of some works in literature , but we chose not to adopt the techniques since this can make the analysis overly complicated . To the best of our knowledge , the paper that most neatly summarizes this line of literature is Zou & Gu [ 2019 ] . See the table in page 3 in their paper . The order of width they obtained is Ω ( n8L12 φ8 ) , where they impose φ-separateness assumption . Remark 3.7 . There has been an attempt to make a connection between the positivity and φseparateness assumptions . Recently , Zou & Gu [ 2019 ] proved the relation λ0 = Ω ( φn−2 ) 5 in a shallow-neural net setting . See Proposition 3.6. of their work . However , it is still an open question on whether this relation holds in DNN setting as well . The results in Theorem 3.5 suggest a positive conjecture on this question . Indeed , plugging the relation λ0 = Ω ( φn−2 ) in ( 11 ) and in the η = O ( λ0 n2L2m ) yield the discount factor ( 1− Ω ( ηmφ n2 ) ) k and step-size η = O ( φ n4L2m ) , which are exactly the same orders as presented in Allen-Zhu et al . [ 2018 ] . See Theorem 1 of their ArXiv version paper for the clear comparison . We leave the proof of this conjecture as a future work . 4In shallow neural network with ReLU activation without bias terms , it is shown that µk satisfy µ0 , µ1 > 0 , µk = 0 if k = 2j + 1 with j ≥ 1 , and otherwise µk = Θ ( k−d ) . See Bietti & Mairal [ 2019 ] . However , in ReLU DNN , it is shown that these parity constraints can be removed even without bias terms and µk achieves Θ ( k−d ) decay rate for large enough k. Readers can refer Bietti & Bach [ 2021 ] for this result . 5We conjecture that this is not the tightest lower bound on λ0 . Recently , Bartlett et al . [ 2021 ] proves that λ0 & d/n in shallow neural net setting . See Lemma 5.3 in their paper . Theorem 3.8 . ( Generalization ) Let f ? ρ ∈ HNTKL . Fix a failure probability δ ∈ [ 0 , 1 ] . Set the width of the network as m log3 ( m ) ≥ Ω ( ω7n8L18 λ80δ 2 ) , the step-size of gradient descent as η = O ( λ0 n2L2m ) , and the variance parameter ω ≤ O ( ( λ0δ n ) 2/3 ) . Then , if the GD iteration k ≥ Ω ( log ( n ) ηmλ0 ) or k ≤ O ( 1 ηmωL ) , with probability at least 1− δ over the randomness of initialized parametersW ( 0 ) , we have R ( f̂k , f ? ρ ) = Ω ( 1 ) . This theorem states that if the network is trained for too long or too short , the L2-prediction error of f̂W ( k ) is bounded away from 0 by some constant factor . Specifically , the former scenario indicates that the overfitting can be harmful for recovering f ? ρ ∈ HNTKL given the noisy observations . Remark 3.9 . Readers should note that the Theorem 3.8 does not consider if the GD algorithm can achieve low prediction risk R ( f̂k , f ? ρ ) over the range of iterations ( ηmωL ) −1 . k . ( ηmλ0 ) −1 log ( n ) . In the numerical experiment to be followed in Section A , we observe that for some algorithm iterations k∗ , the risk indeed decreases to the same minimum as low as the ` 2- regularized algorithm can achieve , and increases again . This observation implies that the unregularized algorithm can achieve the minimax rate of prediction risk . However , analytically deriving a data-dependent stopping time k∗ in our scenario requires further studies , since we need a sharp characterization of eigen-distribution of NTK matrix of ReLU DNN , denoted as H∞L in this paper . Readers can refer the Theorem 4.2. of Hu et al . [ 2021 ] in shallow-neural network and equation ( 6 ) in Raskutti et al . [ 2014 ] in kernel regression context on how to compute k ? with the given eigenvalues of the associated kernel matrices . Remark 3.10 . From practitioner ’ s point of view , we can simply stop GD before it hits the nearlyzero empirical risk , but monitor the performance on a held-out validation sample , and stop training early when a minimum on the validation sample has been reached . 3.4 ANALYSIS OF ` 2-REGULARIZED DNN In this subsection , we study the training dynamics of ` 2-regularized DNN and the effects of the regularization for obtaining the minimax optimal convergence rate of L2-prediction risk . In the results to be followed , we set the orders of model parameters µ , η1 , η2 in ( 6 ) , and a variance parameter of output layer , ω as follows : µ = Θ ( n d−1 2d−1 ) , η1 = Θ ( 1 m n− 3d−2 2d−1 ) , η2 = Θ ( 1 L n− 3d−2 2d−1 ) , ω = O ( 1 L3/2 n− 5d−2 2d−1 ) . ( 12 ) Theorem 3.11 . ( Optimization ) Suppose we minimize ` 2-regularized objective function ( 5 ) via modified GD ( 6 ) . Set the network width m log3 ( m ) ≥ Ω ( L20n24 δ2 ) and model parameters as in ( 12 ) . Then , with probability at least 1− δ , the mean-squared error follows LS ( W ( k ) D ) /n ≤ ( 1− η2µL ) k · LS ( W ( 0 ) D ) /n+OP ( 1 ) , ( 13 ) for k ≥ 0 . Additionally , after k ≥ Ω ( ( η2µL ) −1 log ( n3/2 ) ) iterations of ( 6 ) , for some constant C > 0 , we have ∥∥∥∥∥uD ( k ) −H∞L ( Cµ · I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ OP ( 1 n ) , ( 14 ) where we denote uD ( k ) : = [ f̂W ( k ) D ( x1 ) , . . . , f̂W ( k ) D ( xn ) ] > . Several comments are in sequel . Theorem 3.11 is , to our knowledge , the first result that rigorously shows the training dynamics of ` 2-regularized ReLU DNN in overparametrized setting . Observe that the first term on the right-hand side of the inequality ( 13 ) converges linearly to 0 , and the second term is some positive constant that is bounded away from 0 . This implies that the MSE of regularized DNN is upper-bounded by some positive constant . Note that we only provide the upper bound , but the results of our numerical experiments indicate that the MSE is lower-bounded by OP ( 1 ) as well . We leave the proof of this conjecture for the future work . The inequality ( 14 ) states that the trained dynamics of the regularized neural network can approximate the optimal solution ( denoted as g ? µ ) of the following kernel ridge regression problem : min f∈HNTK { 1 2 n∑ i=1 ( yi − f ( xi ) ) 2 + Cµ 2 ‖f‖2HNTKL } , ( 15 ) where ‖ · ‖HNTKL denotes a NTK-induced RKHS norm . ( Note that the optimization problem in ( 15 ) is not normalized by sample size n. ) The inequality ( 14 ) states that after approximately ( η2µL ) −1 iterations of ( 6 ) , the error rate becomes OP ( 1 n ) . The approximation error is computed at the training data points under ` 2 norm . This should be compared with the Theorem 5.1 of Hu et al . [ 2021 ] , where they showed that the similar approximation holds “ within ” a certain range of algorithm in shallow neural network setting . In contrast , we show that the approximation holds “ after ” k ≥ Ω ( ( η2µL ) −1 log ( n3/2 ) ) in deep neural network . It should be noted that the difference of results comes from the regularization scheme , where we penalize the ∑L ` =1 ‖W ` −W ( 0 ) ` ‖2F , whereas Hu et al . [ 2021 ] regularized the term ‖W1‖2F . As another important comparison , Hu et al . [ 2019 ] showed the equivalence of a solution of kernel ridge regression associated with NTK and first order Taylor expansion of the regularized neural network dynamics ; note , however , that the uD ( k ) in ( 14 ) is a full neural network dynamics . LetR ( f̂ W ( k ) D , f ? ρ ) be the L2-prediction risk of the regularized estimator f̂W ( k ) D via modified GD ( 6 ) . Next theorem states the result of generalization ability of f̂ W ( k ) D . Theorem 3.12 . ( Generalization ) Let f ? ρ ∈ HNTKL . Suppose the network width mlog3 ( m ) ≥ Ω ( L20n24 δ2 ) and model parameters are set as suggested in ( 12 ) . Then , with probability tending to 1 , we have R ( f̂ W ( k ) D , f ? ρ ) = OP ( n− d 2d−1 ) . The resulting convergence rate is O ( n− d 2d−1 ) with respect to the training sample size n. Note that the rate is always faster than O ( n−1/2 ) and turns out to be the minimax optimal [ Caponnetto & De Vito , 2007 ; Blanchard & Mücke , 2018 ] for recovering f ? ρ ∈ HNTKL in the following sense : lim r→0 lim inf n→∞ inf f̂ sup ρ P [ R ( f̂ , f ? ρ ) > rn− d 2d−1 ] = 1 , ( 16 ) where ρ is a data distribution class satisfying the Assumptions ( A1 ) , ( A2 ) and f ? ρ ∈ HNTKL , and infimum is taken over all estimators D → f̂ . It is worth noting that the minimax rate in ( 16 ) is same with the minimax rate for recovering f ? ρ ∈ HNTK1 . ( i.e. , Hu et al . [ 2021 ] ) This result can be derived from the recent discovery of the equivalence between two function spaces , HNTK1 = HNTKL . See Geifman et al . [ 2020 ] and Chen & Xu [ 2020 ] . Remark 3.13 . A particular choice of µ = Θ ( n d−1 2d−1 ) in ( 12 ) is for obtaining an optimal minimax rate for prediction error in Theorem 3.12 . Specifically , the order of µ determines the L2 distance between the f ? ρ and the kernel regressor g ? µ . That is , ‖f ? ρ − g ? µ‖22 = OP ( µ n ) . With the resultH NTK 1 = HNTKL , the same proof of Lemma D.2 . in Hu et al . [ 2021 ] can be applied for proving this result . 4 CONCLUSION We analyze the convergence rate of L2-prediction error of both the unregularized and the regularized gradient descent for overparameterized DNN with ReLU activation for a regression problem . Under a positivity assumption of NTK , we show that without the adoption of early stopping , the L2-prediction error of the estimated DNN via vanilla GD is bounded away from 0 ( Theorem 3.5 ) , whereas the prediction error of the DNN via ` 2-regularized GD achieves the optimal minimax rate ( Theorem 3.12 ) . The minimax rate O ( n− d 2d−1 ) is a faster than the O ( n−1/2 ) by specifying the complexities of target function and hypothesis space . ACKNOWLEDGMENTS Use unnumbered third level headings for the acknowledgments . All acknowledgments , including those to funding agencies , go at the end of the paper . REFERENCES Zeyuan Allen-Zhu , Yuanzhi Li , and Zhao Song . A convergence theory for deep learning via over- parameterization . arXiv e-prints , pp . arXiv–1811 , 2018 . Zeyuan Allen-Zhu , Yuanzhi Li , and Yingyu Liang . Learning and generalization in overparameterized neural networks , going beyond two layers . Advances in Neural Information Processing Systems , 32:6158–6169 , 2019 . Sanjeev Arora , Simon Du , Wei Hu , Zhiyuan Li , and Ruosong Wang . Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks . In International Conference on Machine Learning , pp . 322–332 . PMLR , 2019a . Sanjeev Arora , Simon S Du , Wei Hu , Zhiyuan Li , Russ R Salakhutdinov , and Ruosong Wang . On exact computation with an infinitely wide neural net . Advances in Neural Information Processing Systems , 32:8141–8150 , 2019b . Peter L. Bartlett , Andrea Montanari , and Alexander Rakhlin . Deep learning : a statistical viewpoint . Acta Numerica , 30:87–201 , 2021. doi : 10.1017/S0962492921000027 . Ronen Basri , Meirav Galun , Amnon Geifman , David Jacobs , Yoni Kasten , and Shira Kritchman . Frequency bias in neural networks for input of non-uniform density . In International Conference on Machine Learning , pp . 685–694 . PMLR , 2020 . Benedikt Bauer and Michael Kohler . On deep learning as a remedy for the curse of dimensionality in nonparametric regression . Annals of Statistics , 47 ( 4 ) :2261–2285 , 2019 . Alberto Bietti and Francis Bach . Deep equals shallow for ReLU networks in kernel regimes . In ICLR 2021-International Conference on Learning Representations , pp . 1–22 , 2021 . Alberto Bietti and Julien Mairal . On the inductive bias of neural tangent kernels . In NeurIPS 2019-Thirty-third Conference on Neural Information Processing Systems , volume 32 , pp . 12873– 12884 , 2019 . Gilles Blanchard and Nicole Mücke . Optimal rates for regularization of statistical inverse learning problems . Foundations of Computational Mathematics , 18 ( 4 ) :971–1013 , 2018 . Yuan Cao and Quanquan Gu . Generalization bounds of stochastic gradient descent for wide and deep neural networks . Advances in Neural Information Processing Systems , 32:10836–10846 , 2019 . Andrea Caponnetto and Ernesto De Vito . Optimal rates for the regularized least-squares algorithm . Foundations of Computational Mathematics , 7 ( 3 ) :331–368 , 2007 . Lin Chen and Sheng Xu . Deep neural tangent kernel and Laplace kernel have the same RKHS . In International Conference on Learning Representations , 2020 . Lénaı̈c Chizat and Francis Bach . On the global convergence of gradient descent for overparameterized models using optimal transport . In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pp . 3040–3050 , 2018 . Youngmin Cho and Lawrence Saul . Kernel methods for deep learning . Advances in Neural Information Processing Systems , 22:342–350 , 2009 . Simon Du , Jason Lee , Haochuan Li , Liwei Wang , and Xiyu Zhai . Gradient descent finds global minima of deep neural networks . In International Conference on Machine Learning , pp . 1675– 1685 . PMLR , 2019 . Simon S Du , Xiyu Zhai , Barnabas Poczos , and Aarti Singh . Gradient descent provably optimizes over-parameterized neural networks . In International Conference on Learning Representations , 2018 . Amnon Geifman , Abhay Yadav , Yoni Kasten , Meirav Galun , David Jacobs , and Ronen Basri . On the similarity between the Laplace and neural tangent kernels . arXiv preprint arXiv:2007.01580 , 2020 . Behrooz Ghorbani , Song Mei , Theodor Misiakiewicz , and Andrea Montanari . When do neural networks outperform kernel methods ? arXiv preprint arXiv:2006.13409 , 2020 . Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Delving deep into rectifiers : Surpassing human-level performance on imagenet classification . In Proceedings of the IEEE international conference on computer vision , pp . 1026–1034 , 2015 . Tianyang Hu , Wenjia Wang , Cong Lin , and Guang Cheng . Regularization matters : A nonparametric perspective on overparametrized neural network . In International Conference on Artificial Intelligence and Statistics , pp . 829–837 . PMLR , 2021 . Wei Hu , Zhiyuan Li , and Dingli Yu . Simple and effective regularization methods for training on noisily labeled data with generalization guarantee . In International Conference on Learning Representations , 2019 . Arthur Jacot , Clément Hongler , and Franck Gabriel . Neural tangent kernel : Convergence and generalization in neural networks . In NeurIPS , 2018 . Ziwei Ji , Justin D Li , and Matus Telgarsky . Early-stopped neural networks are consistent . arXiv preprint arXiv:2106.05932 , 2021 . Yongdai Kim , Ilsang Ohn , and Dongha Kim . Fast convergence rates of deep neural networks for classification . Neural Networks , 138:179–197 , 2021 . Michael Kohler and Adam Krzyzak . Over-parametrized deep neural networks do not generalize well . arXiv preprint arXiv:1912.03925 , 2019 . Ilja Kuzborskij and Csaba Szepesvári . Nonparametric regression with shallow overparameterized neural networks trained by gd with early stopping . In Conference on Learning Theory , pp . 2853– 2890 . PMLR , 2021 . Jaehoon Lee , Yasaman Bahri , Roman Novak , Samuel S Schoenholz , Jeffrey Pennington , and Jascha Sohl-Dickstein . Deep neural networks as gaussian processes . In International Conference on Learning Representations , 2018 . Yuanzhi Li and Yingyu Liang . Learning overparameterized neural networks via stochastic gradient descent on structured data . In NeurIPS , 2018 . Ruiqi Liu , Ben Boukai , and Zuofeng Shang . Optimal nonparametric inference via deep neural network . arXiv preprint arXiv:1902.01687 , 2019 . Atsushi Nitanda and Taiji Suzuki . Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime . In International Conference on Learning Representations , 2020 . Prajit Ramachandran , Barret Zoph , and Quoc V Le . Searching for activation functions . arXiv preprint arXiv:1710.05941 , 2017 . Garvesh Raskutti , Martin J Wainwright , and Bin Yu . Early stopping and non-parametric regression : an optimal data-dependent stopping rule . The Journal of Machine Learning Research , 15 ( 1 ) : 335–366 , 2014 . Johannes Schmidt-Hieber . Nonparametric regression using deep neural networks with ReLU activation function . Annals of Statistics , 48 ( 4 ) :1875–1897 , 2020 . Richard S Varga . Geršgorin-type theorems for partitioned matrices . In Geršgorin and His Circles , pp . 155–187 . Springer , 2004 . Roman Vershynin . High-dimensional probability : An introduction with applications in data science , volume 47 . Cambridge university press , 2018 . Colin Wei , Jason Lee , Qiang Liu , and Tengyu Ma . Regularization matters : Generalization and optimization of neural nets vs their induced kernel . 2019 . Greg Yang . Scaling limits of wide neural networks with weight sharing : Gaussian process behavior , gradient independence , and neural tangent kernel derivation . arXiv preprint arXiv:1902.04760 , 2019 . Ming Yuan and Ding-Xuan Zhou . Minimax optimal rates of estimation in high dimensional additive models . Annals of Statistics , 44 ( 6 ) :2564–2593 , 2016 . Difan Zou and Quanquan Gu . An improved analysis of training over-parameterized deep neural networks . Advances in neural information processing systems , 2019 . Difan Zou , Yuan Cao , Dongruo Zhou , and Quanquan Gu . Stochastic gradient descent optimizes over-parameterized deep ReLU networks . arxiv e-prints , art . arXiv preprint arXiv:1811.08888 , 2018 . Difan Zou , Yuan Cao , Dongruo Zhou , and Quanquan Gu . Gradient descent optimizes overparameterized deep ReLU networks . Machine Learning , 109 ( 3 ) :467–492 , 2020 . A NUMERICAL ILLUSTRATIONS In this section , we use synthetic data to corroborate our theoretical findings . We use the He initialization [ He et al. , 2015 ] and employ ( ` 2-regularized ) GD as introduced in subsection 2.2 . For the experiments , we run 1000 epochs of GD and use a fixed step size , setting η1 = η2 = 0.001 . We uniformly generate n feature data xitrain from Sd−1 with d = 2 and generate yi from f ? ρ ( xtraini ) with εi ∼ N ( 0 , 1 ) . To create a function f ? ρ ∈ HNTKL , we use the definition in ( 10 ) with α ∈ Unif ( Sp−1 ) and with p fixed points { x̃j } pj=1 ⊂ Unif ( Sd−1 ) , where p is simply set as 1 . Note that Ker ( · , · ) in ( 10 ) can be calculated via the formulas ( 8 ) and ( 9 ) with specified network depth L. We consider a scenario where we have a network with depth L = 8 and width m = 2000 . The variance parameter of the output layer ( ω ) is set as 1 for unregularized and 0.001 for regularized cases . In Fig 1 . ( a ) , we record the training errors of regularized networks over the GD epochs k ≤ 1000 , where we have n ∈ { 100 , 300 , 500 , 1000 , 5000 } training samples . This aims to verify the inequality ( 13 ) that the MSE of regularized network is bounded away from 0 by some constant . In Fig 1 . ( b ) , the prediction risks of both unregularized and regularized networks are displayed . We approximate the risk with 1500 ∑500 j=1 ( f̂k ( x test j ) − f ? ρ ( xtestj ) ) 2 with a new test data set { xtestj , f ? ρ ( xtestj ) } 500j=1 over k ≤ 1000 for both unregularized and regularized cases . In both cases , they reach the same minimal risks , but the risk of unregularized network increase after it hits the minimal point , whereas the risk of regularized network stays stable . Theorem 3.6 tells us that for the iteration less than the order O ( 1 ηmωL ) , the prediction error is bounded away from 0 . In the experiment for unregularized case , we set η = 0.01 , m = 2000 , L = 8 , and ω = 1 . Plugging in these parameters in the bound says that the minimum can be achieved within a very few iterations . Note that the optimal risk is non-zero as long as we have finite sample sizes n , but converges to 0 at the rate O ( n− d 2d−1 ) . In Fig 1 . ( c ) , we verify that the more training sample sizes we have , the closer the risks of the regularized networks get to 0 . The risk is evaluated at the sample sizes n = { 100 , 300 , 500 , 1000 , 5000 } . We have to acknowledge that there is a discrepancy between our experiment setting and theory . Specifically , due to the limited computing power , we could not run the experiment under the regime of width m log3 ( m ) ≥ Ω ( ω7n8L18 λ80δ 2 ) . But the prediction risk behaves similarly as expected by our theorem , which can be a partial evidence that the statement in Theorem 3.6 still holds in the narrower width of the network . B PRELIMINARY NOTATIONS Before presenting the formal proofs of Lemmas and main results , we introduce several notations used frequently throughout the proofs . First , we denote x ` , i the output of the ` th hidden layer with the input data xi after applying entry-wise ReLU activation function . x ` , i = σ ( W ` σ ( W ` −1 · · ·σ ( W1xi ) · · · ) ) . Denote fW ( k ) ( x ) a value of neural network ( 2 ) evaluated at the collection of network parameters W ( k ) : = { W ( k ) ` } ` =1 , ... , L and W ( k ) ` denotes the ` th hidden layer parameter updated by kth GD iterations . Partial gradient of fW ( k ) ( x ) . We employ the following matrix product notation which was used in several other papers [ Zou et al. , 2018 ; Cao & Gu , 2019 ] : ` 2∏ r= ` 1 Ar : = { A ` 2A ` 2−1 · · ·A ` 1 if ` 1 ≤ ` 2 , I otherwise . ( 17 ) Then , the partial gradient of fW ( k ) ( x ) with respect to W ( k ) ` for 1 ≤ ` ≤ L has a following form : for i ∈ { 1 , . . . , n } , ∇W ` [ fW ( k ) ( xi ) ] = √ m · [ x ( k ) ` −1 , iv T ( L∏ r= ` +1 Σ ( k ) r , i W ( k ) r ) Σ ( k ) ` , i ] > , ` ∈ [ L ] , where Σ ( k ) ` , i : = Diag ( 1 ( 〈w ( k ) ` ,1 , x ( k ) ` −1 , i〉 ≥ 0 ) , . . . ,1 ( 〈w ( k ) ` , m , x ( k ) ` −1 , i〉 ≥ 0 ) ) ∈ Rm×m and w ( k ) ` , j denotes jth column of the matrix W ( k ) ` . Gram matrix H ( k ) . Each entries of empirical gram matrix evaluated at the kth GD update are defined as follows : Hi , j ( k ) = 1 m L∑ ` =1 〈 ∇W ` [ fW ( k ) ( xi ) ] , ∇W ` [ fW ( k ) ( xj ) ] 〉 Tr . Note that H ( 0 ) → H∞L as m → ∞ which is proved in Jacot et al . [ 2018 ] ; Yang [ 2019 ] ; Lee et al . [ 2018 ] ; Arora et al . [ 2019b ] . Perturbation region of weight matrices . Consider a collection of weight matrices W̃ = { W̃ ` } ` =1 , ... , L such that W̃ ∈ B ( W ( 0 ) , τ ) : = { W̃ ` : ‖W̃ ` −W ( 0 ) ` ‖2 ≤ τ , ∀ ` ∈ [ L ] } . ( 18 ) For all i ∈ { 1 , . . . , n } and ` = 1 , . . . , L , we denote x ` , i and x̃ ` , i as the outputs of the ` -th layer of the neural network with weight matrices W ( 0 ) and W̃ , and Σ ` , i and Σ̃ ` , i are diagonal matrices with ( Σ ` , i ) jj = 1 ( 〈w ( 0 ) ` , j , x ` −1 , i〉 ≥ 0 ) and ( Σ̃ ` , i ) jj = 1 ( 〈w̃ ` , j , x̃ ` −1 , i〉 ≥ 0 ) , respectively . C WHY IS IT HARD TO PROVE ‖W ( k ) D , ` −W ( 0 ) D , ` ‖2 ≤ O ( 1 ) ? In this subsection , we provide a heuristic argument on why it is hard to prove ‖W ( k ) D , ` −W ( 0 ) D , ` ‖2 ≤ O ( 1 ) , where W ( k ) D , ` is the model parameter of ` th layer in kth iteration of algorithm . Here , we regularize solely on the model parameter , instead on the relative to the initialization . In this case , we can write the update rule as follows : W ( k ) D , ` = ( 1− η2µ ) W ( k−1 ) D , ` − η1∇W ` [ LS ( W ( k−1 ) D ) ] , ∀1 ≤ ` ≤ L and ∀k ≥ 1 . ( 19 ) By recursively applying above equation ( 4.3 ) , we can write W ( k ) D , ` with respect to W ( 0 ) D , ` as follows : W ( k ) D , ` = ( 1− η2µ ) kW ( 0 ) D , ` − η1 k−1∑ ` =0 ( 1− η2µ ) ` ∇W ` [ LS ( W ( k− ` −1 ) D ) ] . Then , we can control the bound as follows : ‖W ( k ) D , ` −W ( 0 ) D , ` ‖2 ≤ ( 1− ( 1− η2µ ) k ) ∥∥∥W ( 0 ) D , ` ∥∥∥ 2 + η1 η2µ max ` =0 , ... , k−1 ∥∥∥∇W ` [ LS ( W ( k− ` −1 ) D ) ] ∥∥∥ 2 . We know under the initialization setting in our paper , ‖W ( k ) D , ` ‖2 ≤ O ( 1 ) with high-probability ( see Vershynin [ 2018 ] ) , and as long as we can prove the ` 2-norm of gradient is bounded , then we can conclude ‖W ( k ) D , ` −W ( 0 ) D , ` ‖2 ≤ O ( 1 ) . However , we are not aware of works in which they control the size of ‖∇W ` [ LS ( W ( k− ` −1 ) D ) ] ‖2 where the non-convex interactions between model parameters across the hidden layers are allowed . To the best of our knowledge , we know the work Allen-Zhu et al . [ 2019 ] deals with the three layer case under this setting . But we need further investigations on whether the techniques employed in their paper can be generalized to arbitrary L-hidden layer setting . D USEFUL LEMMAS A simple fact . Suppose vj i.i.d∼ N ( 0 , ωm ) for j ∈ [ m ] . Then , with probability at least 1 − exp [ −Ω ( m ) ] , ‖v‖22 ≤ O ( ω ) . Proof . Since ∥∥v2j∥∥Ψ1 ≤ O ( ωm ) for j ∈ [ m ] , where ‖ · ‖Ψ1 denotes a sub-exponential norm , Bernstein ’ s inequality for i.i.d . centered sub-exponential random variables can be employed : For any t ≥ 0 , P ( ∣∣∣∣∣∣ m∑ j=1 ( v2j − ω m ) ∣∣∣∣∣∣ ≥ t ) ≤ 2 exp ( − cmin ( t2∑m j=1 ∥∥v2j∥∥2Ψ1 , t maxj ∥∥v2j∥∥Ψ1 ) ) , ( 20 ) where c > 0 is an absolute constant . Note that we used the fact centering does not hurt the subexponentiality of random variable . Choosing t = O ( ω ) concludes the proof . Lemma 4.1 ( Lemma 7.1 . Allen-Zhu et al . [ 2018 ] ) . With probability at least 1 − O ( nL ) · exp [ −Ω ( m/L ) ] , 3/4 ≤ ‖x ( 0 ) ` , i ‖2 ≤ 5/4 for all i ∈ { 1 , . . . , n } and ` ∈ { 1 , . . . , L } . Lemma 4.2 ( Lemma B.1 . Cao & Gu [ 2019 ] ) . If τ ≤ O ( L−9/2 [ log ( m ) ] −3 ) , then with probability at least 1 − O ( nL ) · exp [ −Ω ( mτ2/3L ) ] , 1/2 ≤ ‖x̃ ` , i‖2 ≤ 3/2 for all W̃ ∈ B ( W ( 0 ) , τ ) , i ∈ { 1 , . . . , n } and ` ∈ { 1 , . . . , L } . Lemma 4.3 ( Allen-Zhu et al . [ 2018 ] ) . Uniformly over i ∈ { 1 , . . . , n } and 1 ≤ ` 1 ≤ ` 2 ≤ L , the following results hold : 1 . ( Lemma.7.3 , Allen-Zhu et al . [ 2018 ] ) Suppose m ≥ Ω ( nL log ( nL ) ) , then with probability at least , 1−O ( nL2 ) · exp [ −Ω ( mτ2/3L ) ] , ∥∥∥∥∥ ` 2∏ r= ` 1 Σ ( 0 ) r , iW ( 0 ) r ∥∥∥∥∥ 2 ≤ O ( √ L ) . 2 . ( Lemma.7.4 , Allen-Zhu et al . [ 2018 ] ) Suppose m ≥ Ω ( nL log ( nL ) ) , then with probability at least , 1−O ( nL ) · exp [ −Ω ( m/L ) ] , ∥∥∥∥∥v > ( L∏ r= ` 1 Σ ( 0 ) r , iW ( 0 ) r ) ∥∥∥∥∥ 2 ≤ O ( √ w ) . 3 . ( Lemma.8.2 , Allen-Zhu et al . [ 2018 ] ) Suppose τ ≤ O ( L−9/2 [ log ( m ) ] −3 ) . For all W̃ ∈ B ( W ( 0 ) , τ ) , with probability at least , 1−O ( nL2 ) · exp [ −Ω ( mτ2/3L ) ] , ∥∥∥x̃ ` 1 , i − x ( 0 ) ` 1 , i∥∥∥2 ≤ O ( τL5/2√log ( m ) ) . 4 . ( Corollary.8.4 , Allen-Zhu et al . [ 2018 ] ) Suppose τ ≤ O ( L−9/2 [ log ( m ) ] −3 ) , then with probability at least , 1−O ( nL2 ) · exp [ −Ω ( mτ2/3L ) ] , ∥∥∥Σ̃ ` 1 , i −Σ ( 0 ) ` 1 , i∥∥∥0 ≤ O ( mτ2/3L ) . 5 . ( Lemma.8.7 , Allen-Zhu et al . [ 2018 ] ) For all ` ∈ [ L ] , let Σ′′ ` , i ∈ [ −3 , 3 ] m×m be the diagonal matrices with at most s = O ( mτ2/3L ) non-zero entries . For all W̃ ∈ B ( W ( 0 ) , τ ) , where τ = O ( 1 L1.5 ) , with probability at least 1−O ( nL ) · exp [ −Ω ( s log ( m ) ) ] , ∥∥∥∥∥vT ( L∏ r= ` 1+1 ( Σ′′r , i + Σ ( 0 ) r , i ) W̃r , i ) ( Σ′′ ` 1 , i + Σ ( 0 ) r , i ) − vT ( L∏ r= ` 1+1 Σ ( 0 ) r , iW ( 0 ) r , i ) Σ ( 0 ) ` 1 , i ∥∥∥∥∥ 2 ≤ O ( τ1/3L2 √ ω log ( m ) ) . Lemma 4.4 ( Lemma B.3 . Cao & Gu [ 2019 ] ) . There exists an absolute constant κ such that , with probability at least 1 − O ( nL2 ) · exp [ −Ω ( mτ2/3L ) ] , i ∈ 1 , . . . , n and ` ∈ 1 , . . . , L and for all W̃ ∈ B ( W ( 0 ) , τ ) , with τ ≤ κL−6 [ log ( m ) ] −3 , it holds uniformly that∥∥∇W ` [ fW̃ ( xi ) ] ∥∥2 ≤ O ( √ωm ) . Lemma 4.5 . Suppose W̃ ∈ B ( W ( 0 ) , τ ) and τ ≤ O ( L−9/2 [ log ( m ) ] −3 ) . For all u ∈ Rm with a cardinality ‖u‖0 ≤ s , for any 1 ≤ ` ≤ L and i ∈ { 1 , . . . , n } , with probability at least 1−O ( nL ) · exp ( − Ω ( s log ( m ) ) ) −O ( nL ) · exp ( − Ω ( mτ2/3L ) ) , ∣∣∣∣∣v > ( L∏ r= ` Σ̃r , iW̃r , i ) u ∣∣∣∣∣ ≤ √ ωs log ( m ) m · O ( ‖u‖2 ) . Proof . Recall Lemma 4.2 . For any fixed vector u ∈ Rm , with probability at least 1 − O ( nL ) · exp [ −Ω ( mτ2/3L ) ] for τ ≤ O ( L−9/2 [ log ( m ) ] −3 ) , for any 1 ≤ ` ≤ L and i ∈ { 1 , . . . , n } , we have the event T , ∥∥∥∥∥ ( L∏ r= ` Σ̃r , iW̃r , i ) u ∥∥∥∥∥ 2 ≤ 3 ‖u‖2 . ( 21 ) Conditioned on this event happens , it is easy to see the random variable v > ( ∏L r=a Σ̃r , iW̃r , i ) u ∼ SG ( 9ω m ‖u‖ 2 2 ) . Based on this observation , we have the probability , P ( ∣∣∣∣∣v > ( L∏ r= ` Σ̃r , iW̃r , i ) u ∣∣∣∣∣ ≥ √ ωs log ( m ) m · O ( ‖u‖2 ) ) ≤ P ( ∣∣∣∣∣v > ( L∏ r= ` Σ̃r , iW̃r , i ) u ∣∣∣∣∣ ≥ √ ωs log ( m ) m · O ( ‖u‖2 ) ∣∣∣ T ) + P ( T c ) ≤ O ( nL ) · exp ( − Ω ( s log ( m ) ) ) +O ( nL ) · exp ( − Ω ( mτ2/3L ) ) , where in the last inequality , union bounds over the indices ` and i , and over the vector u ∈ Rm with ‖u‖0 ≤ s are taken . Lemma 4.6 . Suppose τ ≤ 1 CL9/2 [ log ( m ) ] 3 for some constant C > 0 . Then , for all i ∈ [ n ] and ` ∈ [ L ] , with probability at least 1−O ( nL ) · exp [ −Ω ( mτ2/3L ) ] , we have ∥∥∇W ` [ fW ( k ) ( xi ) ] −∇W ` [ fW ( 0 ) ( xi ) ] ∥∥2 ≤ O ( τ1/3L2√ωm log ( m ) ) . Proof . By using the results from Lemma 4.3 , we can control the term : ‖∇W ` [ fW ( k ) ( xi ) ] −∇W ` [ fW ( 0 ) ( xi ) ] ‖2 = √ m · ∥∥∥∥∥x ( k ) ` −1vT ( L∏ r= ` +1 Σ ( k ) r W ( k ) r ) Σ ( k ) ` − x ( 0 ) ` −1v T ( L∏ r= ` +1 Σ ( 0 ) r W ( 0 ) r ) Σ ( 0 ) ` ∥∥∥∥∥ 2 ≤ √ m · ‖x ( k ) ` −1 − x ( 0 ) ` −1 ‖2︸ ︷︷ ︸ ≤O ( τL5/2 √ log ( m ) ) · ∥∥∥∥∥vT ( L∏ r= ` +1 Σ ( k ) r W ( k ) r ) Σ ( k ) ` ∥∥∥∥∥ 2︸ ︷︷ ︸ ≤O ( √ ω ) + √ m · ∥∥∥x ( 0 ) ` −1∥∥∥ 2︸ ︷︷ ︸ ≤O ( 1 ) · ∥∥∥∥∥vT ( L∏ r= ` +1 Σ ( k ) r W ( k ) r ) Σ ( k ) ` − v T ( L∏ r= ` +1 Σ ( 0 ) r W ( 0 ) r ) Σ ( 0 ) ` ∥∥∥∥∥ 2︸ ︷︷ ︸ ≤O ( τ1/3L2 √ ω log ( m ) ) ≤ O ( τ1/3L2 √ ωm log ( m ) ) , where , in the last inequality , we used the condition on τ ≤ 1 CL9/2 [ log ( m ) ] 3 < 1 . Remark 4.7 . Note that the results in Lemmas 6.3 ( second and fifth items ) , 6.4 , 6.5 , 6.6 are in the setting of vj ∼ N ( 0 , ωm ) for j ∈ [ m ] . For the notational convenience , in following Lemmas we denote fW ( k ) ( xi ) as ui ( k ) and let u ( k ) : = [ u1 ( k ) , . . . , un ( k ) ] > for k ≥ 0 . Lemma 4.8 . For some δ ∈ [ 0 , 1 ] , if m ≥ Ω ( L log ( nL/δ ) ) , then with probability at least 1 − δ , ‖u ( k ) ‖2 ≤ O ( √nω δ ) for any k ≥ 0 . Proof . Recall the Lemma 4.2 stating that ∥∥∥x ( k ) L , i∥∥∥ 2 = O ( 1 ) for any input data xi for i ∈ [ n ] . Also recall that vj ∼ N ( 0 , ωm ) for j ∈ [ m ] , xL , i ∈ R m and ui ( k ) = √ mv > xL , i ∼ N ( 0 , O ( ω ) ) . Then , we have a following via simple Markov inequality : for any t ≥ 0 , P ( ‖u ( k ) ‖2 ≥ t ) ≤ E [ ‖u ( k ) ‖2 ] t ≤ √ E [ ‖u ( k ) ‖22 ] t ≤ O ( √ nω ) t . Lemma 4.9 . For some δ ∈ [ 0 , 1 ] , if m ≥ Ω ( L log ( nL/δ ) ) , then with probability at least 1− δ , we have ‖u ( 0 ) − y‖2 ≤ O ( √ n δ ) . Proof . By Markov ’ s inequality , for any t ≥ 0 , P ( ‖u ( 0 ) − y‖2 ≥ t ) ≤ Eε , W ( 0 ) , v [ ‖u ( 0 ) − y‖22 ] t2 . ( 22 ) Note that the expectation in the nominator of ( 22 ) is taken over the random noise ε and initialized parameter W ( 0 ) , v. We can expand the nominator as follows : Eε , W ( 0 ) , v [ ‖u ( 0 ) − y‖22 ] = EW ( 0 ) , v‖u ( 0 ) ‖22 + Eε‖y‖22 − 2Eε , W ( 0 ) , v [ y > u ( 0 ) ] . ( 23 ) For the convenience of notation , let y∗ : = [ f ? ρ ( x1 ) , . . . , f ? ρ ( xn ) ] > and ε : = [ ε1 , . . . , εn ] > . Recall that we have y = y∗ + ε , and ‖y∗‖22 = O ( n ) . Also note that by Lemma 4.1 , with probability at least 1−O ( nL ) · exp [ −Ω ( m/L ) ] , for any i = 1 , . . . , n , ‖x ( 0 ) L , i‖22 = O ( 1 ) . Then , we have a random variable ui ( 0 ) = √ mv > xL , i ∼ N ( 0 , O ( ω ) ) . Now , we are ready to derive the orders of three terms on the RHS of ( 23 ) . EW ( 0 ) , v‖u ( 0 ) ‖22 = O ( n ) , Eε‖y‖22 = Eε [ ‖y∗‖22 + ‖ε‖22 − 2y > ε ] = O ( n ) , Eε , W ( 0 ) , v [ y > u ( 0 ) ] = Eε , W ( 0 ) , v [ ( y∗ + ε ) > u ( 0 ) ] = 0 . Combining the above three equalities , we conclude the proof . Lemma 4.10 . Suppose τ = O ( n√ω√ mδλ0 ) . For some δ ∈ [ 0 , 1 ] such that δ ≥ O ( nL ) · exp [ −Ω ( mτ2/3L ) ] , then with probability at least 1− δ , we have ‖H ( k ) −H ( 0 ) ‖2 ≤ O ( ω7/6n4/3L3 6 √ log3 ( m ) mδλ20 ) . Proof . By the definition of gram matrix Hi , j ( k ) for any k ≥ 0 , we have |Hi , j ( k ) −Hi , j ( 0 ) | = ∣∣∣∣∣ 1m L∑ ` =1 〈 ∇W ` [ fW ( k ) ( xi ) ] , ∇W ` [ fW ( k ) ( xj ) ] 〉 Tr − 〈 ∇W ` [ fW ( 0 ) ( xi ) ] , ∇W ` [ fW ( 0 ) ( xj ) ] 〉 Tr ∣∣∣∣∣ ≤ 1 m L∑ ` =1 { ∣∣∣∣〈∇W ` [ fW ( k ) ( xi ) ] , ∇W ` [ fW ( k ) ( xj ) ] −∇W ` [ fW ( 0 ) ( xj ) ] 〉 Tr ∣∣∣∣ + ∣∣∣∣〈∇W ` [ fW ( 0 ) ( xj ) ] , ∇W ` [ fW ( k ) ( xi ) ] −∇W ` [ fW ( 0 ) ( xi ) ] 〉 Tr ∣∣∣∣ } ≤ 1 m L∑ ` =1 { ∥∥∇W ` [ fW ( k ) ( xi ) ] ∥∥2︸ ︷︷ ︸ ≤O ( √ ωm ) · ∥∥∇W ` [ fW ( k ) ( xj ) ] −∇W ` [ fW ( 0 ) ( xj ) ] ∥∥2︸ ︷︷ ︸ ≤O ( τ1/3L2 √ ωm log ( m ) ) + ∥∥∇W ` [ fW ( 0 ) ( xj ) ] ∥∥2︸ ︷︷ ︸ ≤O ( √ ωm ) · ∥∥∇W ` [ fW ( k ) ( xi ) ] −∇W ` [ fW ( 0 ) ( xi ) ] ∥∥2︸ ︷︷ ︸ ≤O ( τ1/3L2 √ ωm log ( m ) ) } ≤ O ( ω7/6n1/3L3 6 √ log3 ( m ) mδλ20 ) . In the second inequality , Lemmas 4.4 and 4.6 are used , and in the last inequality , τ = O ( n√ω√ mδλ0 ) is plugged in . With this , using the fact that Frobenius norm of a matrix is bigger than the operator norm , we bound the term ‖H ( k ) −H ( 0 ) ‖2 as follows : ‖H ( k ) −H ( 0 ) ‖2 ≤ ‖H ( k ) −H ( 0 ) ‖F ≤ O ( ω7/6n4/3L3 6 √ log3 ( m ) mδλ20 ) . Lemma 4.11 . For some δ ∈ [ 0 , 1 ] , with probability at least 1− δ , ‖H∞L −H ( 0 ) ‖2 ≤ O ( ωnL5/2 4 √ log ( nL/δ ) m ) Proof . For some δ ′ ∈ [ 0 , 1 ] , set ε = L3/2 4 √ log ( L/δ ′ ) m from Theorem 3.1. of Arora et al . [ 2019b ] . For any fixed i , j ∈ [ n ] , we have P [ ∣∣H∞i , j −Hi , j ( 0 ) ∣∣ ≤ O ( ωL5/2 4√ log ( L/δ′ ) m ) ] ≥ 1− δ′ . After applying the union bound over all i , j ∈ [ n ] , setting δ = δ ′ n2 , and using the fact that Frobenius norm of a matrix is bigger than the operator norm , we conclude the proof . For two positive semi-definite matrices A and B , if we write A B , then it means A − B is positive semi-definite matrix . Similarly , if we write A B , then it means A−B is positive definite matrix . With these notations , we introduce a following Lemma . Lemma 4.12 ( Lemma D.6 . Hu et al . [ 2021 ] ) . For two positive semi-definite matrices A and B , 1 . Suppose A is non-singular , then A B ⇐⇒ λmax ( BA−1 ) ≤ 1 and A B ⇐⇒ λmax ( BA −1 ) < 1 , where λmax ( · ) denotes the maximum eigenvalue of the input matrix . 2 . Suppose A , B and Q are positive definite matrices , A and B are exchangeable , then A B =⇒ AQA BQB . E PROOF OF THEOREM 3.5 For the convenience of notation , denote ui ( k ) = fW ( k ) ( xi ) and let u ( k ) = [ u1 ( k ) , u2 ( k ) , . . . , un ( k ) ] > . In order to achieve linear convergence rate of the training error , ‖u ( k ) − y‖22 , we decompose the term as follows : ‖u ( k + 1 ) − y‖22 = ∥∥u ( k ) − y + ( u ( k + 1 ) − u ( k ) ) ∥∥2 2 = ‖u ( k ) − y‖22 − 2 ( u ( k ) − y ) > ( u ( k + 1 ) − u ( k ) ) + ‖u ( k + 1 ) − u ( k ) ‖22 . Equipped with this decomposition , the proof consists of the following steps : 1 . Similarly with Du et al . [ 2019 ] , a term ( u ( k + 1 ) − u ( k ) ) is decomposed into two terms , where we denote them as I ( k ) 1 and I ( k ) 2 , respectively . We note that the first term I ( k ) 1 is related with a gram matrix H ( k ) and a second term I ( k ) 2 can be controlled small enough in ` 2 sense with proper choices of the step size and the width of network . 2 . A term ‖u ( k + 1 ) − u ( k ) ‖22 needs to be controlled small enough to ensure 2 ( u ( k ) − y ) > ( u ( k + 1 ) − u ( k ) ) > ‖u ( k + 1 ) − u ( k ) ‖22 so that the loss decreases . 3 . It is shown that the distance between the gram matrix H ( k ) and the NTK matrix H∞L is close enough in terms of operator norm . 4 . Lastly , we inductively show that the weights generated from gradient descent stay within a perturbation region B ( W ( 0 ) , τ ) , irrespective with the number of iterations of algorithm , We start the proof by analyzing the term u ( k + 1 ) − u ( k ) . Step 1 . Control on u ( k + 1 ) − u ( k ) . Recall ( Σ ( k ) ` , i ) jj = 1 ( 〈w ( k ) ` , j , x ( k ) ` −1 , i〉 ≥ 0 ) and we introduce a diagonal matrix Σ̃ ( k ) ` , i , whose jth entry is defined as follows : ( Σ̃ ( k ) ` , i ) jj = ( Σ ( k+1 ) ` , i −Σ ( k ) ` , i ) jj · 〈w ( k+1 ) ` , j , x ( k+1 ) ` −1 , i 〉 〈w ( k+1 ) ` , j , x ( k+1 ) ` −1 , i 〉 − 〈w ( k ) ` , j , x ( k ) ` −1 , i〉 . With this notation , the difference x ( k+1 ) L , i − x ( k ) L , i can be rewritten via the recursive applications of Σ̃ ( k ) ` , i : x ( k+1 ) L , i − x ( k ) L , i = ( Σ ( k ) L , i + Σ̃ ( k ) L , i ) ( W ( k+1 ) L x ( k+1 ) L−1 , i −W ( k ) L x ( k ) L−1 , i ) = ( Σ ( k ) L , i + Σ̃ ( k ) L , i ) W ( k+1 ) L ( x ( k+1 ) L−1 , i − x ( k ) L−1 , i ) + ( Σ ( k ) L , i + Σ̃ ( k ) L , i ) ( W ( k+1 ) L −W ( k ) L ) x ( k ) L−1 , i = L∑ ` =1 ( L∏ r= ` +1 ( Σ ( k ) r , i + Σ̃ ( k ) r , i ) W ( k+1 ) r ) ( Σ ( k ) ` , i + Σ̃ ( k ) ` , i ) ( W ( k+1 ) ` −W ( k ) ` ) x ( k ) ` −1 , i ( 24 ) Then , we introduce following notations : D ( k ) ` , i = ( L∏ r= ` +1 Σ ( k ) r , i W ( k ) r ) Σ ( k ) ` , i , D̃ ( k ) ` , i = ( L∏ r= ` +1 ( Σ ( k ) r , i + Σ̃ ( k ) r , i ) W ( k+1 ) r ) ( Σ ( k ) ` , i + Σ̃ ( k ) ` , i ) . Now , we can write ui ( k + 1 ) − ui ( k ) by noting that ui ( k ) = √ m · vTx ( k ) L , i : ui ( k + 1 ) − ui ( k ) = √ m · vT ( x ( k+1 ) L , i − x ( k ) L , i ) = √ m · vT L∑ ` =1 D̃ ( k ) ` , i ( W ( k+1 ) ` −W ( k ) ` ) x ( k ) ` −1 , i ( 25 ) = −η √ m · vT L∑ ` =1 D ( k ) ` , i ∇W ` [ LS ( W ( k ) ) ] x ( k ) ` −1 , i︸ ︷︷ ︸ I ( k ) 1 , i −η √ m · vT L∑ ` =1 ( D̃ ( k ) ` , i −D ( k ) ` , i ) ∇W ` [ LS ( W ( k ) ) ] x ( k ) ` −1 , i︸ ︷︷ ︸ I ( k ) 2 , i Here , I ( k ) 1 , i can be rewritten as follows : I ( k ) 1 , i = −η √ m · vT L∑ ` =1 D ( k ) ` , i n∑ j=1 ( uj ( k ) − yj ) ∇W ` [ fW ( k ) ( xj ) ] x ( k ) ` −1 , i = −η · n∑ j=1 ( uj ( k ) − yj ) · ( √ m L∑ ` =1 vTD ( k ) ` , i ∇W ` [ fW ( k ) ( xj ) ] x ( k ) ` −1 , i ) = −mη · n∑ j=1 ( uj ( k ) − yj ) · 1 m L∑ ` =1 〈 ∇W ` [ fW ( k ) ( xi ) ] , ∇W ` [ fW ( k ) ( xj ) ] 〉 Tr = −mη · n∑ j=1 ( uj ( k ) − yj ) ·Hi , j ( k ) . For I ( k ) 2 , i , we need a more careful control . First , we pay our attention on bounding the term ‖v > ( D̃ ( k ) ` , i −D ( k ) ` , i ) ‖2 as follows : By triangle inequality , we have∥∥∥∥v > ( D̃ ( k ) ` , i −D ( k ) ` , i ) ∥∥∥∥ 2 ≤ ∥∥∥∥v > ( D ( k ) ` , i −D ( 0 ) ` , i ) ∥∥∥∥ 2 + ∥∥∥∥v > ( D̃ ( k ) ` , i −D ( 0 ) ` , i ) ∥∥∥∥ 2 . ( 26 ) We control the first term of the right-hand side ( R.H.S ) in ( 26 ) . By the fourth item of the Lemma 4.3 , we know ‖Σ ( k ) r , i −Σ ( 0 ) r , i ‖0 ≤ O ( mτ2/3L ) and | ( Σ ( k ) r , i −Σ ( 0 ) r , i ) j , j | ≤ 1 for j ∈ [ m ] . Then , we can plug Σ′′r , i = Σ ( k ) r , i −Σ ( 0 ) r , i in the inequality of the fifth item of Lemma 4.3 . So , the first term of the R.H.S in ( 26 ) can be bounded by O ( τ1/3L2 √ ω log ( m ) ) . The second term of the R.H.S in ( 26 ) can be similarly controlled as the first term . Observe that | ( Σ ( k ) r , i +Σ̃ ( k ) r , i ) jj | ≤ 1 , then we have | ( Σ ( k ) r , i +Σ̃ ( k ) r , i −Σ ( 0 ) r , i ) j , j | ≤ 2 for all j ∈ [ m ] . Note that by the definition of Σ̃ ( k ) r , i , we have ‖Σ̃ ( k ) r , i ‖0 = ‖Σ ( k+1 ) r , i −Σ ( k ) r , i ‖0 ≤ ‖Σ ( k+1 ) r , i −Σ ( 0 ) r , i ‖0+‖Σ ( k ) r , i −Σ ( 0 ) r , i ‖0 ≤ O ( mτ2/3L ) . Thus , by the triangle inequality , we have ‖Σ ( k ) r , i + Σ̃ ( k ) r , i − Σ ( 0 ) r , i ‖0 ≤ O ( mτ2/3L ) . These observations enable us to plug Σ′′r , i = Σ ( k ) r , i + Σ̃ ( k ) r , i −Σ ( 0 ) r , i in the inequality of the fifth item of Lemma 4.3 , and give the bound on the second term as O ( τ1/3L2 √ ω log ( m ) ) . We have ‖v > ( D̃ ( k ) ` , i − D ( k ) ` , i ) ‖2 ≤ O ( τ1/3L2 √ ω log ( m ) ) . Now , we control the ` 2-norm of the I ( k ) 2 as follows : ∥∥∥I ( k ) 2 ∥∥∥ 2 ≤ n∑ i=1 ∣∣∣I ( k ) 2 , i ∣∣∣ ≤ η √ m · n∑ i=1 [ L∑ ` =1 ∥∥∥∥v > ( D̃ ( k ) ` , i −D ( k ) ` , i ) ∥∥∥∥ 2︸ ︷︷ ︸ ≤O ( L2τ1/3 √ ω log ( m ) ) · ∥∥∥∇W ` [ LS ( W ( k ) ) ] ∥∥∥ 2 · ∥∥∥x ( k ) ` −1 , i∥∥∥ 2︸ ︷︷ ︸ ≤O ( 1 ) : Lemma 4.2 ] ≤ O ( ηnL2τ1/3 √ ωm log ( m ) ) L∑ ` =1 ∥∥∥∇W ` [ LS ( W ( k ) ) ] ∥∥∥ 2 ≤ O ( ηnL5/2τ1/3 √ ωm log ( m ) ) √√√√ L∑ ` =1 ∥∥∇W ` [ LS ( W ( k ) ) ] ∥∥2F ≤ O ( ηnL5/2τ1/3 √ ωm log ( m ) ) √√√√ n∑ j=1 ( uj ( k ) − yj ) 2 L∑ ` =1 ∥∥∇W ` [ fW ( k ) ( xj ) ] ∥∥2F ≤ O ( ηnL3τ1/3ωm √ log ( m ) ) ‖u ( k ) − y‖2 . ( 27 ) Step 2 . Control on ‖u ( k + 1 ) −u ( k ) ‖22 . Recall that by ( 25 ) , x ( k+1 ) L , i −x ( k ) L , i can be written as follows : x ( k+1 ) L , i − x ( k ) L , i = L∑ ` =1 D̃ ( k ) ` , i ( W ( k+1 ) ` −W ( k ) ` ) x ( k ) ` −1 , i = −η · L∑ ` =1 D̃ ( k ) ` , i ∇W ` [ LS ( W ( k ) ) ] x ( k ) ` −1 , i . It is worth noting that , ∥∥∥∇W ` [ LS ( W ( k ) ) ] ∥∥∥2 2 = ∥∥∥∥∥∥ n∑ j=1 ( uj ( k ) − yj ) ∇W ` [ fW ( k ) ( xj ) ] ∥∥∥∥∥∥ 2 2 ≤ n∑ j=1 ( uj ( k ) − yj ) 2 n∑ j=1 ∥∥∇W ` [ fW ( k ) ( xj ) ] ∥∥22 ≤ O ( nmω ) ‖u ( k ) − y‖22 . ( 28 ) Also , observe that | ( Σ ( k ) r , i + Σ̃ ( k ) r , i ) jj | ≤ 1 for all j ∈ [ m ] , so by Lemma A.3 of Zou et al . [ 2020 ] , we know ‖D̃ ( k ) ` , i ‖2 ≤ O ( √ L ) . Combining all the facts , we can conclude : ‖u ( k + 1 ) − u ( k ) ‖22 = m · n∑ i=1 ( vTx ( k+1 ) L , i − v Tx ( k ) L , i ) 2 ≤ m · ‖v‖22 n∑ i=1 ∥∥∥x ( k+1 ) L , i − x ( k ) L , i∥∥∥2 2 ≤ η2m · ‖v‖22 n∑ i=1 [ L∑ ` =1 ∥∥∥D̃ ( k ) ` , i ∥∥∥2 2 · ∥∥∥∇W ` [ LS ( W ( k ) ) ] ∥∥∥2 2 · ∥∥∥x ( k ) ` −1 , i∥∥∥2 2 ] ≤ O ( η2n2L2m2ω2 ) ‖u ( k ) − y‖22 ≤ O ( η2n2L2m2 ) ‖u ( k ) − y‖22 , ( 29 ) where in the third inequality , we additionally used the fact ‖v‖22 = O ( ω ) with probability at least 1− exp ( −Ω ( m ) ) , and the inequality ( 28 ) . In the last inequality , we used the assumption ω ≤ 1 . Step 3. λmin ( H ( k ) ) ≥ λ02 with sufficiently large m. Denote ρ ( A ) as a sprectral radius of a matrix A . Then , we have ‖H ( k ) −H∞L ‖2 ≥ ρ ( H ( k ) −H∞L ) ≥ −λmin ( H ( k ) −H∞L ) ≥ λmin ( H∞L ) − λmin ( H ( k ) ) ≥ λ0 − λmin ( H ( k ) ) , ( 30 ) where , in the second inequality , we used a triangle inequality , λmin ( H ( k ) −H∞L ) + λmin ( H∞L ) ≤ λmin ( H ( k ) ) . By Lemmas 4.10 and 4.11 , setting m ≥ Ω ( ω7n8L18 log 3 ( m ) λ80δ ) and Õ ( λ 4/3 0 δ 1/3 n4/3L4 ) ≤ ω ≤ 1 , we have ‖H ( k ) −H∞L ‖2 ≤ ‖H ( k ) −H ( 0 ) ‖2 + ‖H ( 0 ) −H ∞ L ‖2 ≤ O ( ω7/6n4/3L3 6 √ log3 ( m ) mδλ20 ) +O ( ωn2L5/2 4 √ log ( nL/δ ) m ) ≤ O ( ω7/6n4/3L3 6 √ log3 ( m ) mδλ20 ) ≤ λ0 2 . ( 31 ) Thus , combining ( 30 ) and ( 31 ) yields that λmin ( H ( k ) ) ≥ λ02 . Step 4 . Concluding the proof . Recall that I ( k ) 1 = −mη · H ( k ) ( u ( k ) − y ) . Then observe that ( u ( k ) − y ) > I ( k ) 1 = −ηm · ( u ( k ) − y ) > H ( k ) ( u ( k ) − y ) ≤ −ηm · λmin ( H ( k ) ) ‖u ( k ) − y‖22 ≤ −ηm · λ0 2 ‖u ( k ) − y‖22 . ( 32 ) We set the step size η , radius of perturbation region τ and network width m as follows , η = Ω ( λ0 n2L2m ) , τ = O ( n √ ω√ mδλ0 ) , m ≥ Ω ( ω7n8L18 log3 ( m ) λ80δ ) . With the above settings , we can control the ‖u ( k + 1 ) − y‖22 by combining ( 27 ) , ( 29 ) and ( 32 ) as follows , ‖u ( k + 1 ) − y‖22 = ∥∥u ( k ) − y + ( u ( k + 1 ) − u ( k ) ) ∥∥2 2 = ‖u ( k ) − y‖22 − 2ηm · ( u ( k ) − y ) > H ( k ) ( u ( k ) − y ) − ( u ( k ) − y ) > I ( k ) 2 + ‖u ( k + 1 ) − u ( k ) ‖ 2 2 ≤ ( 1− ηmλ0 +O ( ηnL3τ1/3mω √ log ( m ) ) +O ( η2n2L2m2 ) ) ‖u ( k ) − y‖22 ≤ ( 1− ηmλ0 2 ) ‖u ( k ) − y‖22 . So far , we have shown from Step 1 to Step 4 that given the radius of perturbation region τ has the order O ( n√ω√ mδλ0 ) , then we can show the training error drops linearly to 0 with the discount factor ( 1 − ηmλ02 ) along with the proper choices of η and m. It remains us to prove the iterates W ( k ) ` for all ` ∈ [ L ] generated by GD algorithm indeed stay in the perturbation region B ( W ( 0 ) , τ ) over k ≥ 0 with τ = O ( n√ω√ mδλ0 ) . Step 5 . The order of the radius of perturbation region . We employ the induction process for the proof . The induction hypothesis is : ∀s ∈ [ k + 1 ] , ∥∥∥W ( s ) ` −W ( 0 ) ` ∥∥∥ 2 ≤ η · O ( √ nmω ) s−1∑ t=0 ( 1− ηmλ0 2 ) t 2 O ( √ n δ ) ≤ O ( n √ ω√ mδλ0 ) . ( 33 ) First , it is easy to see it holds for s = 0 . Now , suppose it holds for s = 0 , . . . , k , we consider s = k + 1.∥∥∥W ( k+1 ) ` −W ( k ) ` ∥∥∥ 2 = ∥∥∥∇W ` [ LS ( W ( k ) ) ] ∥∥∥ 2 = η · ∥∥∥∥∥∥ n∑ j=1 ( uj ( k ) − yj ) ∇W ` [ fW ( k ) ( xj ) ] ∥∥∥∥∥∥ 2 ≤ η · √√√√ n∑ j=1 ∥∥∇W ` [ fW ( k ) ( xj ) ] ∥∥22 √√√√ n∑ j=1 ( uj ( k ) − yj ) 2 ≤ η · O ( √ nmω ) √ 2LS ( W ( k ) ) ≤ η · O ( √ nmω ) ( 1− ηmλ0 2 ) k 2 O ( √ n δ ) , ( 34 ) where in the second inequality , we used Lemmas 4.4 . Note that since it is assumed that W ( k ) ` ∈ B ( W ( 0 ) , τ ) , the Lemma is applicable with m ≥ Ω ( ω7n8L18 log 3 ( m ) λ80δ ) . Simi- larly , since it is assumed that the induction hypothesis holds for s = 0 , . . . , k , we can see ‖u ( k ) − y‖22 ≤ ( 1 − ηmλ02 ) k‖u ( 0 ) − y‖22 . This inequality is plugged in the last inequality with Lemma 4.9 . By combining the inequalities ( 33 ) for s ∈ [ k ] and ( 34 ) , and triangle inequality , we conclude the proof : ∥∥∥W ( k+1 ) ` −W ( 0 ) ` ∥∥∥ 2 ≤ η · O ( √ nmω ) k∑ t=0 ( 1− ηmλ0 2 ) t 2 O ( √ n δ ) ≤ O ( n √ ω√ mδλ0 ) . Proposition 5.1 . For some δ ∈ [ 0 , 1 ] , if we set the width of the network as m ≥ Ω ( ω7n8L18 log 3 ( m ) λ80δ 2 ) , and we set the step-size of gradient descent as η = O ( λ0 n2L2m ) . Then , with probability at least 1 − δ over the randomness of initialized parameters W ( 0 ) , we have for k = 0 , 1 , 2 , . . . , u ( k ) − y = ( I− ηmH∞L ) k ( u ( 0 ) − y ) + ξ ( k ) , where ‖ξ ( k ) ‖2 = k ( 1− ηmλ0 2 ) k−1 O ( ηm · ω7/6n4/3L3 6 √ log3 ( m ) mλ20δ ) ‖y − u ( 0 ) ‖2 . Proof . Define ui ( k ) : = fW ( k ) ( xi ) , then we have u ( k + 1 ) − u ( k ) = −ηm ·H ( k ) ( u ( k ) − y ) + I ( k ) 1 = −ηm ·H∞L ( u ( k ) − y ) − ηm · ( H ( k ) −H∞L ) ( u ( k ) − y ) + I ( k ) 1 = −ηm ·H∞L ( u ( k ) − y ) + e ( k ) . By recursively applying the above equality , we can easily derive a following for any k ≥ 0 , u ( k ) − y = ( I− ηmH∞L ) k ( u ( 0 ) − y ) + k−1∑ t=0 ( I− ηmH∞L ) t e ( k − 1− t ) ︸ ︷︷ ︸ =ξ ( k ) . ( 35 ) Now , we want to show ξ ( k ) can be controlled in arbitrarily small number . First , e ( k ) needs to be bounded in an ` 2 norm : ‖e ( k ) ‖2 ≤ ηm ‖H ∞ L −H ( k ) ‖2 ‖u ( k ) − y‖2 + ∥∥∥I ( k ) 2 ∥∥∥ 2 ≤ ηm · O ( ω7/6n4/3L3 6 √ log3 ( m ) mλ20δ ) ‖u ( k ) − y‖2 , where , in the second inequality , τ = O ( n √ ω√ mδλ0 ) is plugged in ( 27 ) . Equipped with the bound on ‖e ( k ) ‖2 , we can easily bound the ‖ξ ( k ) ‖2 as follows : ∥∥∥∥∥ k−1∑ t=0 ( I− ηmH∞L ) t e ( k − 1− t ) ∥∥∥∥∥ 2 ≤ k−1∑ t=0 ‖I− ηmH∞L ‖ t 2 ‖e ( k − 1− t ) ‖2 ≤ k−1∑ t=0 ( 1− ηmλ0 ) t O ( ηm · ω7/6n4/3L3 6 √ log3 ( m ) mλ20δ ) ‖u ( k − 1− t ) − y‖2 ≤ k−1∑ t=0 ( 1− ηmλ0 ) t O ( ηm · ω7/6n4/3L3 6 √ log3 ( m ) mλ20δ ) ( 1− ηmλ0 2 ) k−1−t ‖u ( 0 ) − y‖2 = k ( 1− ηmλ0 2 ) k−1 O ( ηm · ω7/6n4/3L3 6 √ log3 ( m ) mλ20δ 2 ) ‖u ( 0 ) − y‖2 . ( 36 ) Note that in the third inequality , we used the result from Theorem 1 . F PROOF OF THEOREM 3.8 We begin the proof by decomposing the error f̂W ( k ) ( x ) − f∗ ( x ) for any fixed x ∈ Unif ( Sd−1 ) into two terms as follows : f̂W ( k ) ( x ) − f∗ ( x ) = ( f̂W ( k ) ( x ) − g∗ ( x ) ) ︸ ︷︷ ︸ ∆1 + ( g∗ ( x ) − f∗ ( x ) ) ︸ ︷︷ ︸ ∆2 . ( 37 ) Here , we denote the solution of kernel regression with kernel H∞L as g ∗ ( x ) , which is a minimum RKHS norm interpolant of the noise-free data set { xi , f ? ρ ( xi ) } ni=1 . To avoid the confusion of the notation , we write Ker ( x , X ) = ( H∞L ( x , x1 ) , . . . , H ∞ L ( x , xn ) ) n i=1 ∈ Rn and let y∗ = [ f ? ρ ( x1 ) , . . . , f ? ρ ( xn ) ] > . Then , we have a following closed form solution g∗ ( x ) as , g∗ ( x ) : = Ker ( x , X ) ( H∞L ) −1 y∗ . With the decomposition ( 37 ) , the proof sketch of Theorem 3.8 is as follows . 1 . Note that for any ` ∈ [ L ] , we have f̂W ( k ) ( x ) = 〈vec ( ∇W ` [ fW ( k ) ( x ) ] ) , vec ( W ( k ) ` ) 〉 . We can write the term vec ( W ( k ) ` ) with respect to vec ( W ( 0 ) ` ) , H∞L and the residual term via recursive applications of GD update rule and the result from proposition 2.1 . Readers can refer ( 38 ) . Using the equality ( 38 ) , we can further decompose ∆1 into three terms . That is , ∆1 = ∆11 + ∆12 + ∆13 . Then , using the boundedness of ` 2-norm of network gradient and the fact that the size of ‖ξ ( k ) ‖2 can be controlled with wide enough network , we can control the size of ‖∆12‖2 and ‖∆13‖2 aribtarily small . 2 . In the term ∆2 , the g ? is an interpolant based on noiseless data . For large enough data points , g ? converges fastly to f ? at the rate OP ( 1√n ) . 3 . Lastly , the ∆11 is the only term that is involved with random error ε , and we show that ‖∆11‖2 is bounded away from 0 for small and large GD iteration index k. Step 1 . Control on ∆1 . For n data points ( x1 , . . . , xn ) and for the kth updated parameter W ( k ) , denote : ∇W ` [ fW ( k ) ( X ) ] = [ vec ( ∇W ` [ fW ( k ) ( x1 ) ] ) , · · · , vec ( ∇W ` [ fW ( k ) ( xn ) ] ) ] . Note that when ` = 1 , ∇W ` [ fW ( k ) ( X ) ] ∈ Rmd×n and when ` = 2 , . . . , L , ∇W ` [ fW ( k ) ( X ) ] ∈ Rm2×n . With this notation , we can rewrite the Gradient Descent update rule as vec ( W ( k+1 ) ` ) = vec ( W ( k ) ` ) − η∇W ` [ fW ( k ) ( X ) ] ( u ( k ) − y ) , k ≥ 0 . Applying Proposition 3.8 , we can get : vec ( W ( k ) ` ) − vec ( W ( 0 ) ` ) = k−1∑ j=0 ( vec ( W ( j+1 ) ` ) − vec ( W ( j ) ` ) ) = −η · k−1∑ j=0 ∇W ` [ fW ( j ) ( X ) ] ( u ( j ) − y ) = η · k−1∑ j=0 ∇W ` [ fW ( j ) ( X ) ] ( I− ηmH∞L ) j ( y − u ( 0 ) ) − η · k−1∑ j=0 ∇W ` [ fW ( k ) ( X ) ] ξ ( j ) = η · k−1∑ j=0 ∇W ` [ fW ( 0 ) ( X ) ] ( I− ηmH∞L ) j ( y − u ( 0 ) ) − η · k−1∑ j=0 ∇W ` [ fW ( k ) ( X ) ] ξ ( j ) + η · k−1∑ j=0 ( [ ∇W ` [ fW ( j ) ( X ) ] −∇W ` [ fW ( 0 ) ( X ) ] ] ( I− ηmH∞L ) j ( y − u ( 0 ) ) ) = η · k−1∑ j=0 ∇W ` [ fW ( 0 ) ( X ) ] ( I− ηmH∞L ) j ( y − u ( 0 ) ) + ξ ′ ( k ) . ( 38 ) First , we control ` 2-norm of the first term of ξ ′ ( k ) as follows : Note that ‖∇W ` [ fW ( j ) ( X ) ] ‖F ≤ O ( √ nmω ) by Lemma 4.4 for 0 ≤ j ≤ k − 1 . Then , we have∥∥∥∥∥∥η · k−1∑ j=0 ∇W ` [ fW ( j ) ( X ) ] ξ ( j ) ∥∥∥∥∥∥ 2 ≤ k−1∑ j=0 O ( η √ nmω ) O ( j ( 1− ηmλ0 2 ) j−1 ) O ( ηm · ω7/6n4/3L3 6 √ log3 ( m ) mλ20δ ) ‖y − u ( 0 ) ‖2 ≤ O ( n11/6L3ω5/3 m2/3λ 7/3 0 δ 1/6 √ log ( m ) ) ‖y − u ( 0 ) ‖2 . ( 39 ) In the second inequality , ∑∞ j=1 j ( 1− ηmλ02 ) j = O ( 1 η2m2λ20 ) is used . Then , we control ` 2-norm of the second term of ξ ′ ( k ) as follows : ∥∥∥∥∥∥η · k−1∑ j=0 [ ∇W ` [ fW ( j ) ( X ) ] −∇W ` [ fW ( 0 ) ( X ) ] ] ( I− ηmH∞L ) j ( y − u ( 0 ) ) ∥∥∥∥∥∥ 2 ≤ k−1∑ j=0 η ‖I− ηmH∞L ‖ j 2 ‖y − u ( 0 ) ‖2 √√√√ n∑ i=1 ∥∥∇W ` [ fW ( j ) ( xi ) ] −∇W ` [ fW ( 0 ) ( xi ) ] ∥∥22 ≤ k−1∑ j=0 η ( 1− ηmλ0 ) jO ( n1/3m1/3L2ω2/3 λ 1/3 0 δ 1/6 √ log ( m ) ) O ( √ n ) ‖y − u ( 0 ) ‖2 ≤ O ( n5/6L2ω2/3 m2/3λ 4/3 0 δ 1/6 √ log ( m ) ) ‖y − u ( 0 ) ‖2 , ( 40 ) where in the second inequality , we used Lemmas 4.6 with τ = O ( n√ω√ mδλ0 ) . Now , we are ready to control ∆1 term . By using the equality ( 38 ) , we can decompose the term ∆1 as follows : Let us denote Gk = ∑k−1 j=0 ηm ( I − ηmH∞L ) j . Note that for any ` ∈ [ L ] , f̂W ( k ) ( x ) = 〈vec ( ∇W ` [ fW ( k ) ( x ) ] ) , vec ( W ( k ) ` ) 〉 and recall that y = y∗ + ε . Then , for any fixed ` ′ ∈ [ L ] , we have : ∆1 = [ 〈 vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) , vec ( W ( k ) ` ′ ) 〉 −Ker ( x , X ) ( H∞L ) −1 y∗ ] + Ker ( x , X ) Gky −Ker ( x , X ) Gky = [ Ker ( x , X ) [ Gk − ( H∞L ) −1 ] y∗ + Ker ( x , X ) Gkε ] ︸ ︷︷ ︸ =∆11 + [ 1 m L∑ ` =1 vec ( ∇W ` [ fW ( k ) ( x ) ] ) > ∇W ` [ fW ( 0 ) ( X ) ] −Ker ( x , X ) ] Gky − 1 m ∑ ` : ` 6= ` ′ vec ( ∇W ` [ fW ( k ) ( x ) ] ) > ∇W ` [ fW ( 0 ) ( X ) ] Gky︸ ︷︷ ︸ =∆12 + [ 〈 vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) , vec ( W ( 0 ) ` ′ ) 〉 + vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) > ξ ′ ( k ) − 1 m vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) > ∇W ` ′ [ fW ( 0 ) ( X ) ] Gku ( 0 ) ] .︸ ︷︷ ︸ =∆13 ( 41 ) Our goal in this step is to control ‖∆12‖2 and ‖∆13‖2 . Then , in the third step , we will show ‖∆11‖2 is the term , which governs the behavior of the prediction risk with respect to algorithm iteration k. First , we bound the ` 2 norm of the first term in ∆12 as : ∥∥∥∥∥ [ 1 m L∑ ` =1 vec ( ∇W ` [ fW ( k ) ( x ) ] ) > ∇W ` [ fW ( 0 ) ( X ) ] −Ker ( x , X ) ] Gky ∥∥∥∥∥ 2 ≤ 1 mL L∑ ` =1 ∥∥vec ( ∇W ` [ fW ( k ) ( x ) ] ) − vec ( ∇W ` [ fW ( 0 ) ( x ) ] ) ∥∥2︸ ︷︷ ︸ ≤O ( τ1/3L2 √ ωm log ( m ) ) : Lemma 4.6 ∥∥∇W ` [ fW ( 0 ) ( X ) ] ∥∥F︸ ︷︷ ︸ ≤O ( √ ωnm ) : Lemma 4.4 ‖Gky‖2 + 1 L √√√√ n∑ i=1 ( 1 m L∑ ` =1 〈 ∇W ` [ fW ( 0 ) ( x ) ] , ∇W ` [ fW ( 0 ) ( xi ) ] 〉 Tr −Ker ( x , xi ) ) 2 ‖Gky‖2 ≤ { O ( n5/6L2ω7/6 m1/6δ1/6λ 1/3 0 √ log ( m ) ) +O ( ωn1/2L3/2 4 √ log ( nL/δ ) m ) } ‖Gk‖2 ‖y‖2 ≤ O ( n5/6L2ω7/6 m1/6δ1/6λ 4/3 0 √ log ( m ) · ‖y‖2 ) +O ( ωn1/2L3/2 λ0 4 √ log ( nL/δ ) m · ‖y‖2 ) , ( 42 ) where , in the second inequality , we plugged τ = O ( n√ω√ mδλ0 ) in the result of Lemma 4.6 and used Lemma 4.11 . In the last inequality , we used ‖Gk‖2 ≤ O ( 1 λ0 ) . Similarly , we can control the ` 2 norm of the second term in ∆12 as follows : ∥∥∥∥∥∥ 1m ∑ ` : ` 6= ` ′ vec ( ∇W ` [ fW ( k ) ( x ) ] ) > ∇W ` [ fW ( 0 ) ( X ) ] Gky ∥∥∥∥∥∥ 2 ≤ 1 m ∑ ` : ` 6= ` ′ ‖vec ( ∇W ` [ fW ( k ) ( x ) ] ) ‖2︸ ︷︷ ︸ ≤O ( √ ωm ) · ‖∇W ` [ fW ( 0 ) ( X ) ] ‖F︸ ︷︷ ︸ ≤O ( √ ωmn ) · ‖Gk‖2︸ ︷︷ ︸ ≤O ( 1 λ0 ) ‖y‖2 ≤ O ( ωL √ n λ0 ) · ‖y‖2 . ( 43 ) We turn our attention to controlling ‖∆13‖2 . The first term in ∆13 ; Recall that∥∥∥vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) ∥∥∥ 2 ≤ O ( √ mω ) by Lemma 4.4 . Then , the random variable vec ( ∇W ` [ fW ( k ) ( x ) ] ) > vec ( W ( 0 ) ` ) is simply a N ( 0 , O ( ω ) ) for 1 ≤ ` ≤ L. A straightforward application of Chernoff bound for normal random variable and taking union bound over the layer 1 ≤ ` ≤ L yield that : with probability at least 1− δ , ∣∣∣vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) > vec ( W ( 0 ) ` ′ ) ∣∣∣ ≤ O ( √ω log ( L δ ) ) . ( 44 ) The ` 2 norm of the third term in ∆13 can be bounded as follows : ∥∥∥∥ 1mvec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) > ∇W ` ′ [ fW ( 0 ) ( X ) ] Gku ( 0 ) ∥∥∥∥ 2 ≤ 1 m ∥∥∥vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) ∥∥∥ 2︸ ︷︷ ︸ ≤O ( √ mω ) ∥∥∥∇W ` ′ [ fW ( 0 ) ( X ) ] ∥∥∥ F︸ ︷︷ ︸ ≤O ( √ ωmn ) ‖Gku ( 0 ) ‖2︸ ︷︷ ︸≤O ( √nωλ0δ ) ≤ O ( nω3/2 λ0δ ) . ( 45 ) In the last inequality , we used the Lemma 4.8 and ‖Gk‖2 ≤ O ( 1 λ0 ) . By combin- ing ( 39 ) , ( 40 ) , ( 44 ) , ( 45 ) with ∥∥∥∇W ` ′ [ fW ( 0 ) ( x ) ] ∥∥∥ F ≤ O ( √ mω ) , we have a following : ‖∆13‖2 ≤ ∥∥∥〈vec ( ∇W ` ′ [ fW ( k ) ( x ) ] ) , vec ( W ( 0 ) ` ′ ) 〉∥∥∥ 2 + ∥∥∥ ( vec ( ∇W ` ′ [ fW ( 0 ) ( x ) ] ) ) > ξ′ ( k ) ∥∥∥ 2 + ∥∥∥∥ 1m ( vec ( ∇W ` ′ [ fW ( 0 ) ( x ) ] ) > ∇W ` ′ [ fW ( 0 ) ( X ) ] Gku ( 0 ) ∥∥∥∥ 2 ≤ O ( √ ω log ( L δ ) ) +O ( n11/6L3ω13/6 ‖y − u ( 0 ) ‖2 m1/6λ 4/3 0 δ 1/6 √ log ( m ) ) +O ( n5/6L2ω7/6 ‖y − u ( 0 ) ‖2 m1/6λ 7/3 0 δ 1/6 √ log ( m ) ) +O ( nω3/2 λ0δ ) = O ( n11/6L3ω13/6 ‖y − u ( 0 ) ‖2 m1/6λ 4/3 0 δ 1/6 √ log ( m ) ) +O ( n5/6L2ω7/6 ‖y − u ( 0 ) ‖2 m1/6λ 7/3 0 δ 1/6 √ log ( m ) ) +O ( nω3/2 λ0δ ) . ( 46 ) Step 2 . Control on ∆2 . First , note that there is a recent finding that the reproducing kernel Hilbert spaces of NTKs with any number of layers ( i.e. , L ≥ 1 ) have the same set of functions , if kernels are defined on Sd−1 . See Chen & Xu [ 2020 ] . Along with this result , we can apply the proof used in Lemma.D.2 . in Hu et al . [ 2021 ] for proving a following : ‖∆2‖2 = OP ( 1√ n ) . ( 47 ) Step 3 . The behavior of L2 risk is characterized by the term ∆11 . Recall the decompositions ( 37 ) and ( 41 ) , then we have : f̂W ( k ) ( x ) − f∗ ( x ) = ∆11 + ( ∆12 + ∆13 + ∆2 ) : = ∆11 + Θ . ( 48 ) Our goal in this step is mainly two-folded : ( i ) Control Eε ‖Θ‖22 arbitrarily small with proper choices of step-size of GD η and width of the network m. ( ii ) Show that how Eε ‖∆11‖22 affect the behavior of prediction risk over the GD iterations k. First , note that we have Eε ‖y‖22 = Eε ‖y ∗ + ε‖22 ≤ 2 ( y∗ ) > y∗ + 2Eε ‖ε‖22 = O ( n ) . ( 49 ) Second , recall Lemma 4.9 and note that over the random initialization , with probability at least 1−δ , Eε ‖y − u ( 0 ) ‖22 ≤ O ( n δ ) . ( 50 ) Now , by combining the bounds ( 42 ) , ( 46 ) and ( 47 ) , we have Eε ‖Θ‖22 ≤ 3Eε ( ‖∆12‖22 + ‖∆13‖ 2 2 + ‖∆2‖ 2 2 ) ≤ Eε [ O ( n5/3L4ω7/3 m1/3λ 8/3 0 δ 1/3 log ( m ) · ‖y‖22 ) +O ( ω2nL3 λ20 √ log ( nL/δ ) m · ‖y‖22 ) +O ( n11/3L6ω13/3 ‖y − u ( 0 ) ‖22 m1/3λ 8/3 0 δ 1/3 log ( m ) ) +O ( n5/3L4ω7/3 ‖y − u ( 0 ) ‖22 m1/3λ 14/3 0 δ 1/3 log ( m ) ) ] +O ( n2ω3 λ20δ 2 ) +O ( 1 n ) ≤ O ( ω2n2L3 λ20 √ log ( nL/δ ) m ) +O ( n14/3L6ω13/3 m1/3λ 8/3 0 δ 4/3 log ( m ) ) +O ( n8/3L4ω7/3 m1/3λ 14/3 0 δ 4/3 log ( m ) ) +O ( n2ω3 λ20δ 2 ) +O ( 1 n ) , ( 51 ) where in the third inequality , we used ( 49 ) and ( 50 ) . Case 1 . When k is large , the L2 risk is bounded away from zero by some constant . Now we control Eε ‖∆11‖22 . Recall the definitions ‖f‖ 2 2 : = ∫ x∈Sd−1 |f ( x ) | 2dx and Gk = ∑k−1 j=0 ηm ( I− ηmH∞L ) j . Let us denote S = y∗y∗ > . Then , we have Eε ‖∆11‖22 = ∫ x∈Sd−1 Ker ( x , X ) [ ( Gk − ( H∞L ) −1 ) y∗y∗ > ( Gk − ( H∞L ) −1 ) +G2k ] Ker ( X , x ) dx = ∫ x∈Sd−1 Ker ( x , X ) ( H∞L ) −1 Mk ( H∞L ) −1Ker ( X , x ) dx where Mk = ( I− ηmH∞L ) k S ( I− ηmH∞L ) k + ( I− ( I− ηmH∞L ) k ) 2 = [ ( I− ηmH∞L ) k − ( S + I ) −1 ] ( S + I ) [ ( I− ηmH∞L ) k − ( S + I ) −1 ] + I − ( S + I ) −1 . For the algorithm iterations k ≥ ( log ( n ) ηmλ0 ) C0 with some constant C0 > 1 , we have ( I − ηmH∞L ) k ( 1− ηmλ0 ) k · I exp ( −ηmλ0k ) · I exp ( −C0 log ( n ) ) = 1 nC0 · I . Since 1 + ‖y‖22 ≤ C1n for some constant C1 , we have λmax ( 1 nC0 · ( S + I ) ) = 1 + ‖y‖22 nC0 ≤ C1 nC0−1 < 1 . ( 52 ) Using the first item of Lemma ( 4.12 ) with the inequality ( 52 ) , we have ( I − ηmH∞L ) k 1 nC0 · I ≺ ( S + I ) −1 . ( 53 ) The above inequality ( 53 ) lead to a following result : ( S + I ) −1 − ( I − ηmH∞L ) k ( S + I ) −1 − 1nC0 · I . ( 54 ) It is obvious that both ( S + I ) −1 − ( I − ηmH∞L ) k and ( S + I ) −1 − 1nC0 · I are positive definite matrices due to ( 54 ) , and it is also easy to see that they are exchangeable . By using the second item of Lemma ( 4.12 ) , we have Mk = [ ( I− ηmH∞L ) k − ( S + I ) −1 ] ( S + I ) [ ( I− ηmH∞L ) k − ( S + I ) −1 ] + I − ( S + I ) −1 [ ( S + I ) −1 − 1 nC0 · I ] ( S + I ) [ ( S + I ) −1 − 1 nC0 · I ] + I − ( S + I ) −1 = 1 n2C0 S + ( 1− 1 nC0 ) 2 · I . Then , we have Eε ‖∆11‖22 1 n2C0 A+ ( 1− 1 nC0 ) 2 B c0B , where c0 ∈ ( 0 , 1 ) is a constant and A = ∫ x∈Sd−1 [ Ker ( x , X ) ( H∞L ) −1 y∗ ] 2 dx , and B = ∫ x∈Sd−1 [ Ker ( x , X ) ( H∞L ) −1 ] 2 dx . ( 55 ) By triangle inequality with the decomposition ( 48 ) and the bound on Eε ‖Θ‖22 in ( 51 ) , we have : Eε ∥∥∥f̂W ( k ) − f∗∥∥∥2 2 = Eε ‖∆11 + Θ‖22 ≥ 1 2 Eε ‖∆11‖2 − Eε ‖Θ‖ 2 2 ≥ c0 2 B −O ( 1 n ) −O ( n2ω3 λ20δ 2 ) − Õ ( 1 m1/3 poly ( ω , n , L , 1 λ0 , 1 δ ) ) . ( 56 ) For the third term in ( 56 ) , we can choose ω ≤ C2 ( λ0δ n ) 2/3 for some constant C2 > 0 such that the term can be bounded by c08 ∥∥∥Ker ( · , X ) ( H∞L ) −1∥∥∥2 2 . Similarly , the width m can be chosen large enough such that the fourth term in ( 56 ) is upper-bounded by c08 ‖Ker ( · , X ) ( H∞L ) −1‖22 . Using the above choices of k , ω , and m , we can further bound ( 56 ) : Eε ∥∥fW ( k ) − f∗∥∥22 ≥ c04 ∥∥∥Ker ( · , X ) ( H∞L ) −1∥∥∥22 −O ( 1 n ) . ( 57 ) Note that Eε‖f̂∞ − g∗‖22 = ‖Ker ( · , X ) ( H∞L ) −1‖22 where g∗ : = 0 and f̂∞ denotes the noise interpolator . Then , by Theorem 4.2. of Hu et al . [ 2021 ] , we know that Eε‖f̂∞−g∗‖22 ≥ c1 for some constant c1 > 0 . Then , we can take n large enough such that the term O ( 1 n ) is upper-bounded by c0c1 8 , and finish the proof . Case 2 . When k is small , the L2 risk is bounded away from zero by some constant . Recall the definition of ∆11 in the decomposition ( 41 ) , ∆11 : = Ker ( x , X ) Gk [ y∗ + ε ] −Ker ( x , X ) H∞L y∗ : = ∆∗11 −Ker ( x , X ) H∞L y∗ . ( 58 ) We denote the eigen-decomposition of the matrix H∞L : = ∑n i=1 λivivi > , then we can easily see a following : Gk = ηm k−1∑ j=0 ( n∑ i=1 ( 1− ηmλi ) jvivi > ) ηm k−1∑ j=0 n∑ i=1 vivi > ηmk · I . By using the above inequality , we have Eε ‖∆∗11‖ 2 2 = ∫ x∈Sd−1 Ker ( x , X ) Gk ( S + I ) GkKer ( X , x ) dx ≤ η2m2k2 ( ∫ x∈Sd−1 [ Ker ( x , X ) y∗ ] 2 dx+ ‖Ker ( · , X ) ‖22 ) = O ( η2m2k2ω2n2L2 ) . Recall the decompositions ( 37 ) and ( 41 ) , then we have : Eε ∥∥∥f̂W ( k ) − f∗∥∥∥2 2 = Eε ‖∆∗11 + Θ−Ker ( · , X ) H∞L y∗‖ 2 2 ≥ 1 2 ‖Ker ( · , X ) H∞L y∗‖ 2 2 − Eε ‖∆ ∗ 11 + Θ‖ 2 2 ≥ 1 2 ‖Ker ( · , X ) H∞L y∗‖ 2 2 − 2Eε ‖∆ ∗ 11‖ 2 2 − 2Eε ‖Θ‖ 2 2 ≥ 1 2 ‖Ker ( · , X ) H∞L y∗‖ 2 2 −O ( η2m2k2ω2n2L2 ) −O ( 1 n ) −O ( n2ω3 λ20δ 2 ) − Õ ( 1 m1/3 poly ( ω , n , L , 1 λ0 , 1 δ ) ) . ( 59 ) For some constant C ′ 1 > 0 , let k ≤ C ′ 1 ( 1 ηmnωL ) such that the second term in the bound ( 59 ) can be bounded by 18‖Ker ( · , X ) ( H∞L ) −1 y∗‖22 . For the fourth term in ( 59 ) , we can choose ω ≤ C ′ 2 ( λ0δ n ) 2/3 for some constant C ′ 2 > 0 such that the term can be bounded by 1 8 ∥∥∥Ker ( · , X ) ( H∞L ) −1y∗∥∥∥2 2 . Sim- ilarly , the width m can be chosen large enough such that the fifth term in ( 59 ) is upper-bounded by 1 8‖Ker ( · , X ) ( H∞L ) −1 y∗‖22 . Using the above choices of k , ω , and m , we can further bound ( 59 ) : Eε ∥∥fW ( k ) − f∗∥∥22 ≥ 14 ∥∥∥Ker ( · , X ) ( H∞L ) −1y∗∥∥∥22 −O ( 1 n ) ≥ C ′ 3 ∥∥f ? ρ∥∥22 −O ( 1n ) . ( 60 ) In the second inequality , we used ( 47 ) with triangle inequality . In the third inequality , we can take n large enough such that the term O ( 1 n ) is upper-bounded by C ′ 3 2 ∥∥f ? ρ∥∥22 . Lastly , by using the assumption that f ? ρ is a square-integrable function , we finish the proof . G PROOF OF THEOREM 3.11-TRAINING ERROR For the convenience of notation , we denote uD , i ( k ) = fW ( k ) D ( xi ) and let uD ( k ) = [ u1 , D ( k ) , . . . , un , D ( k ) ] > . In order to analyze the training error of ` 2-regularized estimator , ‖uD ( k ) − y‖22 , we decompose the term as follows : ‖uD ( k + 1 ) − y‖22 = ‖uD ( k + 1 ) − ( 1− η2µL ) uD ( k ) ‖ 2 2 + ‖ ( 1− η2µL ) uD ( k ) − y‖ 2 2 − 2 ( y − ( 1− η2µL ) uD ( k ) ) > ( uD ( k + 1 ) − ( 1− η2µL ) uD ( k ) ) ( 61 ) Equipped with this decomposition , the proof consists of the following steps : 1 . We decompose the decayed prediction difference uD ( k+ 1 ) − ( 1− η2µL ) uD ( k ) into two terms . We note that the first term is related with a gram matrix HD ( k ) and denote a second term as I ( k ) D . 2 . The term I ( k ) D can be further decomposed into three terms , where we denote them as I ( k ) 2 , D , I ( k ) 3 , D and I ( k ) 5 , D . The crux for controlling the ` 2-norm of the above three terms is to utilize the results from the Appendix A.4 . The applications of Lemmas in the Appendix A.4 is possible , since we can inductively guarantee that ‖W ( k ) D , ` − W ( 0 ) D , ` ‖2 is sufficiently small enough for large enough m. 3 . Given the decomposition ( 61 ) , we further decompose it into four terms as follows : ( 61 ) = ‖ ( 1− η2µL ) uD ( k ) − y‖22︸ ︷︷ ︸ : =T1 + ‖uD ( k + 1 ) − ( 1− η2µL ) uD ( k ) ‖22︸ ︷︷ ︸ : =T2 + 2mη1 ( y − ( 1− η2µL ) uD ( k ) ) > HD ( k ) ( uD ( k ) − y ) ︸ ︷︷ ︸ : =T3 −2 ( y − ( 1− η2µL ) uD ( k ) ) > I ( k ) D︸ ︷︷ ︸ : =T4 . ( 62 ) In this step , we obtain the upper-bound of ‖Ti‖2 for i = 1 , 2 , 3 , 4 obtained in Step 4 . 4 . We combine the upper-bounds of ‖Ti‖2 for i = 1 , 2 , 3 , 4 in step 3 and obtain the bound on ‖uD ( k + 1 ) − y‖22 with respect to ‖uD ( k ) − y‖22 and ‖y‖2 . 5 . Lastly , we inductively show that the weights generated from regularized gradient descent stay within a perturbation region B ( W ( 0 ) , τ ) , irrespective with the number of iterations of algorithm . We start the proof by analyzing the term u ( k + 1 ) − ( 1− η2µL ) u ( k ) . Step 1 . Dynamics of uD ( k + 1 ) − ( 1−η2µL ) ·uD ( k ) . Recall ( Σ ( k ) D , ` , i ) jj = 1 ( 〈w ( k ) D , ` , j , x ( k ) D , ` −1 , i〉 ≥ 0 ) and we introduce a diagonal matrix Σ̃ ( k ) D , ` , i , whose jth entry is defined as follows : ( Σ̃ ( k ) D , ` , i ) jj = ( Σ ( k+1 ) D , ` , i −Σ ( k ) D , ` , i ) jj · 〈w ( k+1 ) D , ` , j , x ( k+1 ) D , ` −1 , i〉 〈w ( k+1 ) D , ` , j , x ( k+1 ) D , ` −1 , i〉 − 〈w ( k ) D , ` , j , x ( k ) D , ` −1 , i〉 . With this notation , the difference x ( k+1 ) D , L , i − x ( k ) D , L , i can be rewritten via the recursive applications of Σ̃ ( k ) D , ` , i : Then , we introduce following notations : D ( k ) D , ` , i = ( L∏ r= ` +1 Σ ( k ) D , r , iW ( k ) D , r ) Σ ( k ) D , ` , i , D̃ ( k ) D , ` , i = ( L∏ r= ` +1 ( Σ ( k ) D , r , i + Σ̃ ( k ) D , r , i ) W ( k+1 ) D , r ) ( Σ ( k ) D , ` , i + Σ̃ ( k ) D , ` , i ) . Now , we can write uD , i ( k + 1 ) − uD , i ( k ) by noting that uD , i ( k ) = √ m · vTx ( k ) D , L , i : uD , i ( k + 1 ) − uD , i ( k ) = √ m · vT ( x ( k+1 ) D , L , i − x ( k ) D , L , i ) = √ m · vT L∑ ` =1 D̃ ( k ) D , ` , i ( W ( k+1 ) D , ` −W ( k ) D , ` ) x ( k ) D , ` −1 , i = √ m · vT L∑ ` =1 D̃ ( k ) D , ` , i ( − η1∇W ` [ LS ( W ( k ) D ) ] − η2µW ( k ) D , ` + η2µW ( 0 ) D , ` ) x ( k ) D , ` −1 , i = −η1 √ m · vT L∑ ` =1 D ( k ) D , ` , i∇W ` [ LS ( W ( k ) D ) ] x ( k ) D , ` −1 , i︸ ︷︷ ︸ I ( k ) 1 , D , i −η1 √ m · vT L∑ ` =1 ( D̃ ( k ) D , ` , i −D ( k ) D , ` , i ) ∇W ` [ LS ( W ( k ) D ) ] x ( k ) D , ` −1 , i︸ ︷︷ ︸ I ( k ) 2 , D , i −η2µ √ m · vT L∑ ` =1 ( D̃ ( k ) D , ` , i −D ( k ) D , ` , i ) ( W ( k ) D , ` −W ( 0 ) D , ` ) x ( k ) D , ` −1 , i︸ ︷︷ ︸ I ( k ) 3 , D , i −η2µ √ m · vT L∑ ` =1 D ( k ) D , ` , iW ( k ) D , ` x ( k ) D , ` −1 , i︸ ︷︷ ︸ I ( k ) 4 , D , i +η2µ √ m · vT L∑ ` =1 D ( k ) D , ` , iW ( 0 ) D , ` x ( k ) D , ` −1 , i︸ ︷︷ ︸ I ( k ) 5 , D , i ( 63 ) where in the second equality , we used the recursive relation ( 24 ) , and in the third equality , modified GD update rule ( 6 ) is applied . Furthermore , I ( k ) 1 , D , i can be rewritten as follows : I ( k ) 1 , D , i = −η1 √ m · vT L∑ ` =1 D ( k ) D , ` , i n∑ j=1 ( uD , j ( k ) − yj ) ∇W ` [ f W ( k ) D ( xj ) ] x ( k ) D , ` −1 , i = −η1 · n∑ j=1 ( uD , j ( k ) − yj ) · ( √ m L∑ ` =1 vTD ( k ) D , ` , i∇W ` [ f W ( k ) D ( xj ) ] x ( k ) D , ` −1 , i ) = −mη1 · n∑ j=1 ( uD , j ( k ) − yj ) · 1 m L∑ ` =1 〈 ∇W ` [ f W ( k ) D ( xi ) ] , ∇W ` [ f W ( k ) D ( xj ) ] 〉 Tr = −mη1 · n∑ j=1 ( uD , j ( k ) − yj ) ·HD , i , j ( k ) . ( 64 ) With I ( k ) 4 , i = ( −η2µL ) · uD , i ( k ) and ( 64 ) , we can rewrite ( 63 ) as follows : uD , i ( k + 1 ) − ( 1− η2µL ) uD , i ( k ) = −mη1 · n∑ j=1 ( uD , j ( k ) − yj ) ·HD , i , j ( k ) + I ( k ) 2 , D , i + I ( k ) 3 , D , i + I ( k ) 5 , D , i . ( 65 ) Step 2 . Control of the size ∥∥∥I ( k ) D ∥∥∥ 2 . Let I ( k ) D = [ I ( k ) 2 , D,1 + I ( k ) 3 , D,1 + I ( k ) 5 , D,1 , . . . , I ( k ) 2 , D , n + I ( k ) 3 , D , n + I ( k ) 5 , D,1 ] > . Now , we control the bound on the ∥∥∥I ( k ) D ∥∥∥2 2 . Recall that in Eq . ( 27 ) , we have ∥∥∥I ( k ) 2 , D∥∥∥ 2 ≤ O ( η1nL 3τ1/3ωm √ log ( m ) ) ‖uD ( k ) − y‖2 . ( 66 ) Similarly , ∥∥∥I ( k ) 3 , D∥∥∥ 2 can be bounded : ∥∥∥I ( k ) 3 , D∥∥∥ 2 ≤ n∑ i=1 ∣∣∣I ( k ) 3 , D , i∣∣∣ ≤ η2µ√m · n∑ i=1 [ L∑ ` =1 ∥∥∥∥v > ( D̃ ( k ) D , ` , i −D ( k ) D , ` , i ) ∥∥∥∥ 2︸ ︷︷ ︸ ≤O ( L2τ1/3 √ ω log ( m ) ) · ∥∥∥W ( k ) D , ` −W ( 0 ) D , ` ∥∥∥ 2︸ ︷︷ ︸ ≤τ · ∥∥∥x ( k ) D , ` −1 , i∥∥∥ 2︸ ︷︷ ︸ ≤O ( 1 ) ] ≤ O ( η2µnL 3τ4/3 √ ωm log ( m ) ) . ( 67 ) Lastly ∥∥∥I ( k ) 5 , D∥∥∥ 2 can be bounded : ∥∥∥I ( k ) 5 , D∥∥∥ 2 ≤ n∑ i=1 ∣∣∣I ( k ) 5 , D , i∣∣∣ ≤ n∑ i=1 ∣∣∣∣∣η2µ√m · vT L∑ ` =1 D ( k ) D , ` , iW ( k ) D , ` x ( k ) D , ` −1 , i ∣∣∣∣∣+ n∑ i=1 ∣∣∣∣∣η2µ√m · vT L∑ ` =1 D ( k ) D , ` , i ( W ( k ) D , ` −W ( 0 ) D , ` ) x ( k ) D , ` −1 , i ∣∣∣∣∣ ≤ η2µL · n∑ i=1 |ui , D ( k ) |+ η2µ √ m · n∑ i=1 [ L∑ ` =1 ‖v‖2︸ ︷︷ ︸ ≤O ( √ ω ) · ∥∥∥D ( k ) D , ` , i∥∥∥ 2︸ ︷︷ ︸ ≤O ( √ L ) · ∥∥∥W ( k ) D , ` −W ( 0 ) D , ` ∥∥∥ 2︸ ︷︷ ︸ ≤τ · ∥∥∥x ( k ) D , ` −1 , i∥∥∥ 2︸ ︷︷ ︸ ≤O ( 1 ) ] ≤ O ( η2µnL √ ω log ( L/δ ) ) +O ( η2µnL 3/2τ √ mω ) , ( 68 ) where in the last inequality , we employed the same logic used in ( 44 ) with the Lemma 4.2 to obtain the upper-bound on the |ui , D ( k ) | . We set the orders of the parameters µ , η1 , η2 , τ , and ω as follows : µ = Θ ( n d−1 2d−1 ) , η1 = Θ ( 1 m n− 3d−2 2d−1 ) , η2 = Θ ( 1 L n− 3d−2 2d−1 ) , τ = O ( L √ ω√ mδ n d 2d−1 ) , ω = O ( 1 L3/2 n− 5d−2 2d−1 ) . ( 69 ) Plugging the choices of parameters ( 69 ) with sufficiently large m in ( 66 ) , ( 67 ) and ( 68 ) yields ∥∥∥I ( k ) D ∥∥∥ 2 ≤ O ( L37/12n− 9d−8 12d−6 √ log ( m ) m1/6δ1/3 ) · ‖uD ( k ) − y‖2 +OP ( 1 n2 ) . ( 70 ) Step 3 . Upper-bound of ‖Ti‖2 on i = 1 , 2 , 3 , 4 . First , we work on getting the upper-bound on λmax ( HD ( k ) ) . By the Gershgorin ’ s circle theorem [ Varga , 2004 ] , we know the maximum eigenvalue of symmetric positive semi-definite matrix is upper-bounded by the maximum absolute column sum of the matrix . Using this fact , we can bound the λmax ( HD ( k ) ) as : λmax ( HD ( k ) ) ≤ max i=1 , ... , n n∑ j=1 |HD , i , j ( k ) | ≤ max i=1 , ... , n n∑ j=1 ∣∣∣∣∣ 1m L∑ ` =1 〈 ∇W ` [ f W ( k ) D ( xi ) ] , ∇W ` [ f W ( k ) D ( xj ) ] 〉 Tr ∣∣∣∣∣ ≤ max i=1 , ... , n n∑ j=1 1 m L∑ ` =1 ∥∥∥∇W ` [ fW ( k ) D ( xi ) ] ∥∥∥F︸ ︷︷ ︸ ≤O ( √ mω ) ∥∥∥∇W ` [ fW ( k ) D ( xj ) ] ∥∥∥F︸ ︷︷ ︸ ≤O ( √ mω ) ≤ O ( nLω ) . ( 71 ) Recall the decomposition ( 62 ) . Our goal is to obtain the upper-bound on Ti for i = 1 , 2 , 3 , 4 . Control on T1 . By using the inequality 2η2µL ( 1−η2µL ) y > ( y−uD ( k ) ) ≤ η2µL ‖y‖22+η2µL ( 1− η2µL ) 2 ‖y − uD ( k ) ‖22 , we have ‖y − ( 1− η2µL ) uD ( k ) ‖22 = ∥∥ ( 1− η2µL ) ( y − uD ( k ) ) + η2µLy∥∥22 = ( 1− η2µL ) 2 ‖y − uD ( k ) ‖22 + η 2 2µ 2L2 ‖y‖22 + 2η2µL ( 1− η2µL ) y > ( y − uD ( k ) ) ≤ ( η2µL+ η22µ2L2 ) ‖y‖ 2 2 + ( 1 + η2µL ) ( 1− η2µL ) 2 ‖y − uD ( k ) ‖22 . ( 72 ) Control on T2 . Recall the equality ( 65 ) . Then , through applications of the Young ’ s inequality ‖a+ b‖22 ≤ 2 ‖a‖ 2 2 + 2 ‖b‖ 2 2 for a , b ∈ Rn , we have ‖uD ( k + 1 ) − ( 1− η2µL ) uD ( k ) ‖22 = ∥∥∥−mη1 ·HD ( k ) ( uD ( k ) − y ) + I ( k ) D ∥∥∥2 2 ≤ 2m2η21λmax ( HD ( k ) ) 2 ‖y − uD ( k ) ‖22 + 2 ∥∥∥I ( k ) D ∥∥∥2 2 . ( 73 ) Similarly with T1 and T2 , we can control T3 and T4 as follows : Control on T3 . Recall HD ( k ) is a Gram matrix by definition . Then , by using the fact λmin ( HD ( k ) ) ≥ 0 and Cauchy-Schwarz inequality , we have 2mη1 ( y − ( 1− η2µL ) uD ( k ) ) > HD ( k ) ( uD ( k ) − y ) = −2mη1 ( 1− η2µL ) ( y − uD ( k ) ) > HD ( k ) ( y − uD ( k ) ) + ( 2mη1η2µL ) · y > HD ( k ) ( uD ( k ) − y ) ≤ ( 2mη1η2µL ) · λmax ( HD ( k ) ) ‖y − uD ( k ) ‖22 + ( 2mη1η2µL ) · ( λmax ( HD ( k ) ) ‖y‖2 ‖y − uD ( k ) ‖2 ) − 2mη1λmin ( HD ( k ) ) ‖y − uD ( k ) ‖22 = ( 4mη1η2µL ) · λmax ( HD ( k ) ) ‖y − uD ( k ) ‖22 + ( 4mη1η2µL ) · λmax ( HD ( k ) ) ‖y‖22 . ( 74 ) Control on T4 . By a simple Cauchy-Schwarz and Young ’ s inequality , we have − 2 ( y − ( 1− η2µL ) uD ( k ) ) > I ( k ) D = −2 ( 1− η2µL ) ( y − uD ( k ) ) > I ( k ) D + 2η2µL · y > ID ( k ) ≤ 2 ( 1− η2µL ) ‖y − uD ( k ) ‖2 ∥∥∥I ( k ) D ∥∥∥ 2 + η2µL ‖y‖22 + η2µL ∥∥∥I ( k ) D ∥∥∥2 2 ( 75 ) Step 4 . Upper-bound of the decomposition on training error ( 62 ) . Before getting the upper bound of the decomposition ( 62 ) , we first work on obtaining the bound of ( 76 ) . Set κ = O ( 1 n2 ) and notice η2µL = O ( 1 n ) by ( 69 ) , then we have 2 ∥∥∥I ( k ) D ∥∥∥2 2 + 2 ( 1− η2µL ) ‖y − uD ( k ) ‖2 ∥∥∥I ( k ) D ∥∥∥ 2 + η2µL ∥∥∥I ( k ) D ∥∥∥2 2 ( 76 ) = ( 2 + η2µL ) ∥∥∥I ( k ) D ∥∥∥2 2 + 2κ ( 1− η2µL ) ‖y − uD ( k ) ‖2 · 1 κ ∥∥∥I ( k ) D ∥∥∥ 2 ≤ ( 2 + η2µL+ 1 κ2 ) ∥∥∥I ( k ) D ∥∥∥2 2 + κ2 ( 1− η2µL ) 2 ‖y − uD ( k ) ‖22 = 1 κ2 · ∥∥∥I ( k ) D ∥∥∥2 2 + κ2 ( 1− η2µL ) 2 ‖y − uD ( k ) ‖22 ≤ { 1 κ2 · O ( L37/6n− 9d−8 6d−3 log ( m ) m1/3δ2/3 ) + κ2 ( 1− η2µL ) 2 } · ‖y − uD ( k ) ‖22 + 1κ2 · OP ( 1 n4 ) ≤ ( η2µL ) 4 ( 1− η2µL ) 2 · ‖y − uD ( k ) ‖22 + η2µL · ‖y‖22 , ( 77 ) where in the second inequality , the Eq . ( 70 ) is used with ( a + b ) 2 ≤ 2a2 + 2b2 for a , b ∈ R , and in the last inequality , we used ‖y‖22 = O ( n ) and the sufficiently large m to control the order of the coefficient terms of ‖y − uD ( k ) ‖22 . Specifically , we choose m ≥ Ω ( L19n20 log 3 ( m ) δ2 ) . Now , by combining the inequalities ( 72 ) , ( 73 ) , ( 74 ) , ( 75 ) , ( 71 ) and ( 77 ) , we obtain the upper-bound on the decomposition ( 62 ) ; ‖uD ( k + 1 ) − y‖22 ≤ ( 2η2µL+ η 2 2µ 2L2 + 4mη1η2µL · λmax ( HD ( k ) ) ) · ‖y‖22 + ( ( 1 + η2µL ) ( 1− η2µL ) 2 + 2m2η21λmax ( HD ( k ) ) 2 + 4mη1η2µL · λmax ( HD ( k ) ) ) · ‖y − uD ( k ) ‖22 + ( 2 ∥∥∥I ( k ) D ∥∥∥2 2 + 2 ( 1− η2µL ) ‖y − uD ( k ) ‖2 ∥∥∥I ( k ) D ∥∥∥ 2 + η2µL ∥∥∥I ( k ) D ∥∥∥2 2 ) ≤ { 3η2µL+ η 2 2µ 2L2 +O ( ωmnη1η2µL 2 ) } · ‖y‖22 + { ( 1 + η2µL+ η 4 2µ 4L4 ) ( 1− η2µL ) 2 +O ( ω2m2n2η21L 2 ) +O ( ωmnη1η2µL 2 ) } · ‖y − uD ( k ) ‖22 : = A · ‖y‖22 + ( 1− B ) · ‖y − uD ( k ) ‖ 2 2 . ( 78 ) With the order choices of µ , η1 and η2 as in ( 69 ) , it is easy to see the leading terms of both A and B are same as η2µL = o ( 1n ) . Then , by recursively applying the inequality ( 78 ) , we can get the upper-bound on the training error . ‖y − uD ( k + 1 ) ‖22 ≤ A · ‖y‖ 2 2 + ( 1− B ) · ‖y − uD ( k ) ‖ 2 2 ≤ A‖y‖22 · ( k∑ j=0 ( 1− B ) j ) + ( 1− B ) k+1 · ‖y − uD ( 0 ) ‖22 ≤ A B · ‖y‖22 + ( 1− B ) k+1 · ‖y − uD ( 0 ) ‖22 ≤ O ( n ) + ( 1− η2µL ) k+1 · ‖y − uD ( 0 ) ‖22 . ( 79 ) In the last inequality , we used AB = o ( 1 ) , B ≥ η2µL and ‖y‖ 2 2 = O ( n ) . Step 5 . The order of the radius of perturbation region . It remains us to prove the radius of perturbation region τ has the order OP ( L √ ω√ m n d 2d−1 ) . First , recall that the ` 2-regularized GD update rule is as : W ( k ) D , ` = ( 1− η2µ ) W ( k−1 ) D , ` − η1∇W ` [ LS ( W ( k−1 ) D ) ] + η2µW ( 0 ) D , ` , ∀1 ≤ ` ≤ L and ∀k ≥ 1 . ( 80 ) Similarly with the proof in the Theorem 3.5 , we employ the induction process for the proof . The induction hypothesis is∥∥∥W ( s ) D , ` −W ( 0 ) D , ` ∥∥∥ 2 ≤ O ( η1n √ mω√ δη2µ ) , ∀s ∈ [ k + 1 ] . ( 81 ) It is easy to see it holds for s = 0 , and suppose it holds for s = 0 , 1 , . . . , k , we consider k+1 . Using the update rule ( 80 ) , we have∥∥∥W ( k+1 ) D , ` −W ( k ) D , ` ∥∥∥ 2 ≤ η2µ ∥∥∥W ( k ) D , ` −W ( 0 ) D , ` ∥∥∥ 2 + η1 ∥∥∥∇W ` [ LS ( W ( k ) D ) ] ∥∥∥ 2 = η2µ ∥∥∥W ( k ) D , ` −W ( 0 ) D , ` ∥∥∥ 2 + η1 ∥∥∥∥∥ n∑ i=1 ( yi − uD , i ( k ) ) ∇W ` [ fWD ( k ) ( xi ) ] ∥∥∥∥∥ 2 ≤ O ( η1n √ mω√ δ ) +O ( η1 √ nmω ) · ‖y − uD ( k ) ‖2 ≤ O ( η1n √ mω√ δ ) +O ( η1 √ nmω ) · { O ( √ n ) + ( 1− η2µL ) k 2O ( √ n δ ) } ≤ O ( η1n √ mω√ δη2µ ) . In the first inequality , we use the induction hypothesis for s = k , and Lemma 4.4 . In the second inequality , since the induction hypothesis holds for s = 0 , 1 , . . . , k , we employ ‖y − uD ( k ) ‖2 ≤ O ( √ n ) + ( 1− η2µL ) k 2 ‖y − uD ( 0 ) ‖2 with the Lemma 4.9 . In the last inequality , we use η2µ < 1 . By triangle inequality , the induction holds for s = k + 1 . Plugging the proper choices of η1 , η2 and µ as suggested in ( 69 ) to O ( η1n √ mω√ δη2µ ) yields ‖W ( k ) D , ` −W ( 0 ) D , ` ‖2 ≤ OP ( L √ ω√ m n d 2d−1 ) . H PROOF OF THEOREM 3.11-KERNEL RIDGE REGRESSOR APPROXIMATION We present a following proof sketch on the approximation of regularized DNN estimator to kernel ridge regressor . 1 . The key idea for proving the second result in Theorem 3.8 is to write the distance between ui , D ( k ) ( where D is to denote the prediction is obtained from regularized GD rule ) and kernel regressor B : = H∞L ( Cµ · I + H∞L ) −1 y in terms of NTK matrix H∞L , which is as follows : uD ( k ) −B = ( ( 1− η2µL ) · I −mη1H∞L ) k ( uD ( 0 ) −B ) + eD ( k ) . Above equality describes how the regularized estimator evolves to fit the kernel regressor as iteration of algorithm goes by . 2 . We can bound the ` 2-norm of residual term eD ( k ) as O ( 1/n ) , and show that the ` 2 norm of the first term on the RHS of equation ( 4.3 ) decays at the rateO ( √ n ( 1− η2µL ) k ) . Here the √ n comes from the bound ‖B‖2 ≤ O ( √ n ) , since we know ‖u ( 0 ) ‖2 hasO ( √ nω ) with small ω ≤ 1 . This yields the claim . Recall the equality ( 65 ) . Then , we have uD ( k + 1 ) − ( 1− η2µL ) uD ( k ) = −mη1 ·HD ( k ) ( uD ( k ) − y ) + I ( k ) 2 , D + I ( k ) 3 , D + I ( k ) 5 , D = −mη1 ·H∞L ( uD ( k ) − y ) −mη1 · ( HD ( k ) −H∞L ) ( uD ( k ) − y ) + I ( k ) 2 , D + I ( k ) 3 , D + I ( k ) 5 , D = −mη1 ·H∞L ( uD ( k ) − y ) + ξD ( k ) . ( 82 ) With τ = O ( L √ ω√ mδ n d 2d−1 ) , similarly with Lemma 4.10 and a direct employment of the result from Lemma 4.11 , we can control the distance from HD ( k ) to H∞L under operator norm as follows : ‖HD ( k ) −H∞L ‖2 ≤ ‖HD ( k ) −H ( 0 ) ‖2 + ‖H ( 0 ) −H ∞ L ‖2 ≤ O ( ω7/6L10/3n 7d−3 6d−3 6 √ log3 ( m ) mδ2 ) +O ( ωL5/2n 4 √ log ( nL/δ ) m ) ≤ O ( L19/12n− 21d−8 12d−6 6 √ log3 ( m ) mδ2 ) +O ( Ln− 18d−6 12d−6 4 √ log ( nL/δ ) m ) ≤ O ( L19/12n− 21d−8 12d−6 6 √ log3 ( m ) mδ2 ) , ( 83 ) where in the third inequality , ω = O ( 1 L3/2 n− 5d−2 2d−1 ) is plugged-in . The last inequality holds with d ≥ 2 with large enough n and the condition on width m ≥ Ω ( L19n20 log 3 ( m ) δ2 ) . Then , the ` 2 norm of ξD ( k ) can be bounded as : ‖ξD ( k ) ‖2 ≤ mη1 · ‖H ∞ L −HD ( k ) ‖2 ‖uD ( k ) − y‖2 + ∥∥∥I ( k ) D ∥∥∥ 2 ≤ O ( L19/12n− 12d−5 6d−3 √ log ( m ) m1/6δ1/3 ) · ‖uD ( k ) − y‖2︸ ︷︷ ︸ ≤O ( √ n/δ ) +OP ( 1 n2 ) ≤ O ( L19/12n− 18d−7 12d−6 √ log ( m ) m1/6δ5/6 ) +OP ( 1 n2 ) = OP ( 1 n2 ) , ( 84 ) where in the second inequality , we used ( 83 ) with η1 = O ( 1 mn − 3d−22d−1 ) to control the first term and employed Eq . ( 70 ) to control the second term . In the last equality , we usedm ≥ Ω ( L19n20 log 3 ( m ) δ2 ) . Now , by setting B : = ( η2µL η1m I + H∞L ) −1 H∞L y , we can easily convert the equality ( 82 ) as follows : for k ≥ 1 , uD ( k ) −B = ( ( 1− η2µL ) · I −mη1H∞L ) ( uD ( k − 1 ) −B ) + ξD ( k − 1 ) . ( 85 ) The recursive applications of the equality ( 85 ) yields uD ( k ) −B = ( ( 1− η2µL ) · I −mη1H∞L ) k ( uD ( 0 ) −B ) + k∑ j=0 ( ( 1− η2µL ) · I −mη1H∞L ) j ξD ( k − j − 1 ) = ( ( 1− η2µL ) · I −mη1H∞L ) k ( uD ( 0 ) −B ) + eD ( k ) . ( 86 ) Now , we bound the ` 2 norm of eD ( k ) in ( 86 ) : ‖eD ( k ) ‖2 = ∥∥∥∥∥∥ k∑ j=0 ( ( 1− η2µL ) · I −mη1H∞L ) j ξD ( k − j − 1 ) ∥∥∥∥∥∥ 2 ≤ k∑ j=0 ∥∥ ( 1− η2µL ) · I −mη1H∞L ∥∥j2 ‖ξD ( k − j − 1 ) ‖2 ≤ k∑ j=0 ( 1− η2µL ) j ‖ξD ( k − j − 1 ) ‖2 = O ( 1n ) , ( 87 ) in the last inequality , we used η2µL = O ( 1 n ) and Eq . ( 84 ) . Now , we control the ` 2-norm of the first term in ( 86 ) as : ∥∥∥∥∥ ( ( 1− η2µL ) · I −mη1H∞L ) k ( uD ( 0 ) −B ) ∥∥∥∥∥ 2 ≤ ( 1− η2µL ) k ‖uD ( 0 ) −B‖2 ≤ O ( √ n ( 1− η2µL ) k ) , ( 88 ) where in the second inequality , we used ‖uD ( 0 ) ‖2 ≤ O ( √ nω/δ ) and the fact that ‖B‖2 ≤ ∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 H∞L ∥∥∥∥∥ 2 · ‖y‖2 ≤ O ( √ n ) . By combining ( 87 ) and ( 88 ) and using a fact ( 1− η2µL ) k ≤ exp ( −η2µLk ) , we conclude that after k ≥ Ω ( ( η2µL ) −1 log ( n3/2 ) ) , the error ‖uD ( k ) −B‖2 decays at the rate O ( 1 n ) . I PROOF OF THEOREM 3.12 We begin the proof by decomposing the error f̂ W ( k ) D ( x ) − f∗ ( x ) for any fixed x ∈ Unif ( Sd−1 ) into two terms as follows : f̂ W ( k ) D ( x ) − f∗ ( x ) = ( f̂ W ( k ) D ( x ) − g∗µ ( x ) ) ︸ ︷︷ ︸ ∆D,1 + ( g∗µ ( x ) − f∗ ( x ) ) ︸ ︷︷ ︸ ∆D,2 . ( 89 ) Here , we devise a solution of kernel ridge regression g∗µ ( x ) in the decomposition ( 89 ) : g∗µ ( x ) : = Ker ( x , X ) ( Cµ · I + H∞L ) −1 y , for some constant C > 0 . Specifically , in the proof to follow , we choose η1 and η2 such that C = η2Lη1m for the theoretical convenience . Our goal is to show that all the terms ‖∆D,1‖ 2 2 , and ‖∆D,2‖22 have the order either equal to or smaller than O ( n− d 2d−1 ) with the proper choices on m , µ , η1 and η2 . Since the high-level proof idea is similar with that of Theorem 3.8 , we omit the step-by-step proof sketch of Theorem 3.12 . The most notable difference between the proof strategies of the two theorems is that the regularized DNN approximate the kernel ridge regressor of noisy data , whereas in Theorem 3.8 , unregularized DNN approximate the interpolant based on noiseless data . Step 1 . Control on ∆D,2 . First , note that there is a recent finding that the reproducing kernel Hilbert spaces induced from NTKs with any number of layers ( i.e. , L ≥ 1 ) have the same set of functions , if kernels are defined on Sd−1 . See Chen & Xu [ 2020 ] . Along with this result , under the choice of model parameters as suggested in ( 69 ) , we can apply exactly the same proof used in Theorem.3.2 in Hu et al . [ 2021 ] for proving a following : ‖∆D,2‖22 : = ∥∥g∗µ − f∗∥∥22 = OP ( n− d2d−1 ) , ∥∥g∗µ∥∥2H = OP ( 1 ) . ( 90 ) Step 2 . Control on ∆D,1 . For n data points ( x1 , . . . , xn ) and for the kth updated parameter W ( k ) D , denote : ∇W ` [ f W ( k ) D ( X ) ] = [ vec ( ∇W ` [ f W ( k ) D ( x1 ) ] ) , · · · , vec ( ∇W ` [ f W ( k ) D ( xn ) ] ) ] . Note that when ` = 1 , ∇W ` [ f W ( k ) D ( X ) ] ∈ Rmd×n and when ` = 2 , . . . , L , ∇W ` [ f W ( k ) D ( X ) ] ∈ Rm2×n . With this notation , we can write the vectorized version of the update rule ( 80 ) as : vec ( W ( k ) D , ` ) = vec ( W ( 0 ) D , ` ) − η1 k−1∑ j=0 ( 1− η2µ ) j∇W ` [ fWD ( k−j−1 ) ( X ) ] ( uD ( k − j − 1 ) − y ) , ∀1 ≤ ` ≤ L and ∀k ≥ 1 . Using the equality , we can get the decomposition : vec ( W ( k ) D , ` ) = vec ( W ( 0 ) D , ` ) ︸ ︷︷ ︸ : =E1 −η1∇W ` [ fWD ( 0 ) ( X ) ] k−1∑ j=0 ( 1− η2µ ) j ( uD ( k − j − 1 ) − y ) ︸ ︷︷ ︸ : =E2 −η1 k−1∑ j=0 ( 1− η2µ ) j [ ∇W ` [ fWD ( k−j−1 ) ( X ) ] −∇W ` [ fWD ( 0 ) ( X ) ] ] ( uD ( k − j − 1 ) − y ) .︸ ︷︷ ︸ : =E3 ( 91 ) Let zD , k ( x ) : = vec ( ∇W ` [ f W ( k ) D ( x ) ] ) , and note that f W ( k ) D ( x ) = 〈zD , k ( x ) , vec ( W ( k ) D , ` ) 〉 . Then , by the definition of ∆D,1 and the decomposition ( 91 ) , we have ∆D,1 = 1 L L∑ ` =1 〈zD , k ( x ) , E1 + E2 + E3〉 −Ker ( x , X ) ( η2µL η1m I + H∞L ) −1 y = 1 L L∑ ` =1 〈zD , k ( x ) , E1〉+ 1 L L∑ ` =1 〈zD , k ( x ) , E3〉 + 1 L L∑ ` =1 〈zD , k ( x ) , E2〉 −Ker ( x , X ) ( η2µL η1m I + H∞L ) −1 y︸ ︷︷ ︸ : =C ( 92 ) First , we focus on controlling the ` 2 bound on the first two terms in ( 92 ) . Observe that the first term can be bounded as : ∣∣∣∣∣ 1L L∑ ` =1 〈zD , k ( x ) , E1〉 ∣∣∣∣∣ 2 ≤ 1 L L∑ ` =1 |〈zD , k ( x ) , E1〉|2 . ( 93 ) Recall that ‖zD , k ( x ) ‖2 ≤ O ( √ mω ) by Lemma 4.4 . Then , the random variable zD , k ( x ) > vec ( W ( 0 ) D , ` ) | zD , k ( x ) is simply a N ( 0 , O ( ω ) ) for 1 ≤ ` ≤ L. A straightforward application of Chernoff bound for normal random variable and taking union bound over the layer 1 ≤ ` ≤ L yield that : with probability at least 1− δ , 1 L L∑ ` =1 ∣∣∣zD , k ( x ) > vec ( W ( 0 ) D , ` ) ∣∣∣2 ≤ O ( ω log ( L δ ) ) . ( 94 ) The ` 2 norm of the second term in ( 92 ) can be similarly bounded as ( 93 ) in addition with the CauchySchwarz inequality : ∣∣∣∣∣ 1L L∑ ` =1 〈zD , k ( x ) , E3〉 ∣∣∣∣∣ 2 ≤ 1 L L∑ ` =1 |〈zD , k ( x ) , E3〉|2 ≤ 1 L L∑ ` =1 ‖zD , k ( x ) ‖22 ‖E3‖ 2 2 . ( 95 ) The ‖E3‖2 is bounded as : ‖E3‖2 = ∥∥∥∥∥∥η1 k−1∑ j=0 ( 1− η2µ ) j [ ∇W ` [ fWD ( k−j−1 ) ( X ) ] −∇W ` [ fWD ( 0 ) ( X ) ] ] ( uD ( k − j − 1 ) − y ) ∥∥∥∥∥∥ 2 ≤ η1 k−1∑ j=0 ( 1− η2µ ) j · ∥∥∇W ` [ fWD ( k−j−1 ) ( X ) ] −∇W ` [ fWD ( 0 ) ( X ) ] ∥∥2 ‖uD ( k − j − 1 ) − y‖2 ≤ η1 k−1∑ j=0 ( 1− η2µ ) j · ∥∥∇W ` [ fWD ( k−j−1 ) ( X ) ] −∇W ` [ fWD ( 0 ) ( X ) ] ∥∥F ‖uD ( k − j − 1 ) − y‖2 = η1 k−1∑ j=0 ( 1− η2µ ) j · √√√√ n∑ i=1 ∥∥∇W ` [ fWD ( k−j−1 ) ( xi ) ] −∇W ` [ fWD ( 0 ) ( xi ) ] ∥∥2F ‖uD ( k − j − 1 ) − y‖2 ≤ η1 k−1∑ j=0 ( 1− η2µ ) j · √√√√2 n∑ i=1 ∥∥∇W ` [ fWD ( k−j−1 ) ( xi ) ] −∇W ` [ fWD ( 0 ) ( xi ) ] ∥∥22 ‖uD ( k − j − 1 ) − y‖2 ≤ η1 η2µ · O ( τ1/3L2 √ ωmn log ( m ) ) · O ( √ n ) ≤ O ( L10/3ω1/6 m2/3δ1/3 n 4d 6d−3 √ log ( m ) ) . ( 96 ) In the first , second and third inequalities , we used a simple fact that for the matrix A ∈ Rd1×d2 with rank r , then ‖A‖2 ≤ ‖A‖F ≤ √ r‖A‖2 . Recall that the rank of the matrix∇W ` [ fWD ( k−j−1 ) ( x ) ] − ∇W ` [ fWD ( 0 ) ( x ) ] is at most 2 . In the second to the last inequality , we use the result of Lemma 4.6 and the ‖uD ( i ) − y‖2 ≤ O ( √ n ) for any i ≥ 1 . In the last inequality , we plug the correct orders as set in ( 69 ) to τ , η1 , η2 and µ . Back to the inequality ( 95 ) , using the ‖zD , k ( x ) ‖2 ≤ O ( √ mω ) and ( 96 ) , we can get 1 L L∑ ` =1 ‖zD , k ( x ) ‖22 ‖E3‖ 2 2 ≤ OP ( L20/3ω4/3 m1/3 n 8d 6d−3 log ( m ) ) . ( 97 ) Before controlling the ` 2 norm of C in ( 92 ) , recall that we set B : = ( η2µL η1m I + H∞L ) −1 H∞L y and the dynamics of uD ( k ) −B can be expressed in terms of H∞L as follows : For any k ≥ 1 , uD ( k ) −B = ( ( 1− η2µL ) · I −mη1H∞L ) k ( uD ( 0 ) −B ) + eD ( k ) , ( 98 ) with ‖eD ( k ) ‖2 ≤ O ( 1 n ) . Using ( 98 ) , we can further decompose the term E2 in ( 91 ) as : E2 : = −η1∇W ` [ fWD ( 0 ) ( X ) ] k−1∑ j=0 ( 1− η2µ ) j ( uD ( k − j − 1 ) − y ) = η1∇W ` [ fWD ( 0 ) ( X ) ] k−1∑ j=0 ( 1− η2µ ) j ( ( 1− η2µL ) · I −mη1H∞L ) k−j−1 B − η1∇W ` [ fWD ( 0 ) ( X ) ] k−1∑ j=0 ( 1− η2µ ) j ( ( 1− η2µL ) · I −mη1H∞L ) k−j−1 uD ( 0 ) − η1∇W ` [ fWD ( 0 ) ( X ) ] k−1∑ j=0 ( 1− η2µ ) jeD ( k − j − 1 ) − η1∇W ` [ fWD ( 0 ) ( X ) ] k−1∑ j=0 ( 1− η2µ ) j ( B− y ) = E2,1 + E2,2 + E2,3 + E2,4 . ( 99 ) Then , we can re-write the error term C in ( 92 ) as : C = 1 L L∑ ` =1 〈zD , k ( x ) , E2,1〉+ 1 L L∑ ` =1 〈zD , k ( x ) , E2,2〉+ 1 L L∑ ` =1 〈zD , k ( x ) , E2,3〉 + { 1 L L∑ ` =1 〈zD , k ( x ) , E2,4〉 −Ker ( x , X ) ( η2µL η1m I + H∞L ) −1 y } ︸ ︷︷ ︸ : =D . ( 100 ) Our goal is to control the ` 2 norm of each summand in the equality ( 100 ) . For the first three terms in ( 100 ) , a simple Cauchy-Schwarz inequality can be applied : for i = 1 , 2 , 3 : ∣∣∣∣∣ 1L L∑ ` =1 〈zD , k ( x ) , E2 , i〉 ∣∣∣∣∣ 2 ≤ 1 L L∑ ` =1 |〈zD , k ( x ) , E2 , i〉|2 ≤ 1 L L∑ ` =1 ‖zD , k ( x ) ‖22 · ‖E2 , i‖ 2 2 . We work on obtaining the bound of ∑L ` =1 ‖E2,1‖ 2 2 . Let Tk be defined as Tk : = k−1∑ j=0 ( 1− η2µ ) j ( ( 1− η2µL ) · I −mη1H∞L ) k−j−1 . Then , we have L∑ ` =1 ‖E2,1‖22 = η 2 1 L∑ ` =1 ( B > T > k ∇W ` [ fWD ( 0 ) ( X ) ] > ∇W ` [ fWD ( 0 ) ( X ) ] TkB ) = mη21B > T > k H ( 0 ) TkB = mη21B > T > k ( H ( 0 ) −H∞L ) TkB +mη21B > T > k H∞L TkB ≤ mη21 ‖H ( 0 ) −H∞L ‖2 ·B > T 2k B +mη21B > T > k H∞L TkB . ( 101 ) To obtain the upper-bound on ( 101 ) , we need to control the terms T > k H∞L Tk and B > T 2k B . Let us denote H∞L = ∑n i=1 λiviv > i be the eigen-decomposition of H ∞ L . Using 1− η2µL ≤ 1− η2µ , note that Tk = k−1∑ j=0 ( 1− η2µ ) j ( 1− η2µL ) k−j−1 ( I − mη1 1− η2µL H∞L ) k−j−1 ( 1− η2µ ) k−1 k−1∑ i=0 ( I − mη1 1− η2µ H∞L ) i = ( 1− η2µ ) k−1 n∑ j=0 ( 1− ( 1− mη11−η2µλj ) k mη1 1−η2µλj ) vjv > j ( 1− η2µ ) k mη1λ0 · I . ( 102 ) A similar logic can be applied to bound T > k H∞L Tk : T > k H∞L Tk ( 1− η2µ ) k−1 n∑ j=0 ( 1− ( 1− mη11−η2µλj ) k mη1 1−η2µλj ) 2 λjvjv > j ( 1− η2µ ) 2k m2η21 · ( H∞L ) −1 . ( 103 ) Recall the definition of the notation B : = H∞L ( η2µL η1m I + H∞L ) −1 y . Then , we can bound the term B > T > k H∞L TkB : B > T > k H∞L TkB ≤ ( 1− η2µ ) 2k m2η21 ·B > ( H∞L ) −1 B = ( 1− η2µ ) 2k m2η21 · y > ( η2µL η1m I + H∞L ) −1 H∞L ( η2µL η1m I + H∞L ) −1 y = O ( ( 1− η2µ ) 2k m2η21 ) , ( 104 ) where in the last equality , we used ∥∥g∗µ∥∥2H = OP ( 1 ) in ( 90 ) . Now we turn our attention to bound the term B > T 2k B , B > T 2k B ≤ ( 1− η2µ ) 2k m2η21λ 2 0 y > ( η2µL η1m I + H∞L ) −1 ( H∞L ) 2 ( η2µL η1m I + H∞L ) −1 y = O ( ( 1− η2µ ) 2k n m2η21λ 2 0 ) , ( 105 ) where we used ‖y‖22 = O ( n ) in the last inequality . Combining the bounds ( 104 ) , ( 105 ) and the result from Lemma 4.11 , we can further bound ( 101 ) and have : L∑ ` =1 ‖E2,1‖22 ≤ O ( ω ( 1− η2µ ) 2k mλ20 n2L5/2 4 √ log ( nL/δ ) m + ( 1− η2µ ) 2k m ) ≤ O ( ( 1− η2µ ) 2k m ) , ( 106 ) where in the second inequality , we used m ≥ Ω ( L19n20 log 3 ( m ) δ2 ) . Similarly , we can bound∑L ` =1 ‖E2,2‖ 2 2 : L∑ ` =1 ‖E2,2‖22 = η 2 1 L∑ ` =1 ( uD ( 0 ) > T > k ∇W ` [ fWD ( 0 ) ( X ) ] > ∇W ` [ fWD ( 0 ) ( X ) ] TkuD ( 0 ) ) = mη21uD ( 0 ) > T > k H ( 0 ) TkuD ( 0 ) = mη21uD ( 0 ) > T > k ( H ( 0 ) −H∞L ) TkuD ( 0 ) +mη21uD ( 0 ) > T > k H∞L TkuD ( 0 ) ≤ mη21 ‖H ( 0 ) −H∞L ‖2 · uD ( 0 ) > T 2k uD ( 0 ) +mη21uD ( 0 ) > T > k H∞L TkuD ( 0 ) ≤ mη21 ( 1− η2µ ) 2k m2η21λ 2 0 O ( ωnL5/2 4 √ log ( nL/δ ) m ) ‖uD ( 0 ) ‖22 +mη21 ( 1− η2µ ) 2k m2η21 uD ( 0 ) > ( H∞L ) −1uD ( 0 ) ≤ O ( ( 1− η2µ ) 2k n2ω2L5/2 mλ20δ 2 4 √ log ( nL/δ ) m + nω ( 1− η2µ ) 2k mλ0δ2 ) = OP ( nω ( 1− η2µ ) 2k mλ0 ) . ( 107 ) Here , in the second inequality , we used the inequalities ( 102 ) and ( 103 ) and Lemma 4.11 . In the third inequality , we used the Lemma 4.8 , ‖u ( 0 ) ‖2 = O ( √nω δ ) with probability at least 1− δ . In the last equality , we used m ≥ Ω ( L19n20 log 3 ( m ) δ2 ) . Next , we bound ∑L ` =1 ‖E2,3‖ 2 2 as : L∑ ` =1 ‖E2,3‖22 = mη 2 1 · ( k−1∑ j=0 ( 1− η2µ ) j ( ek−j−1 ) ) > HD ( 0 ) ( k−1∑ j=0 ( 1− η2µ ) j ( ek−j−1 ) ) ≤ mη 2 1 η22µ 2 · λmax ( HD ( k ) ) · ‖ek−j−1‖22 ≤ mη21 η22µ 2 · O ( ωnL ) · O ( 1 n2 ) = O ( L3 m ω · n− 4d−3 2d−1 ) . ( 108 ) Now , we focus on obtaining the ` 2 norm bound on D in ( 100 ) . Recall the definition of the notation B : = H∞L ( η2µL η1m I + H∞L ) −1 y . A simple calculation yields that B− y = H∞L ( η2µL η1m I + H∞L ) −1 y − y = −η2µL mη1 ( η2µL η1m I + H∞L ) −1 y . Then , we can re-write the expression of the D as : D : = ( η2µL mη1 ) · η1 1 L L∑ ` =1 〈 zD , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 k−1∑ j=0 ( 1− η2µ ) j ( η2µL η1m I + H∞L ) −1 y −Ker ( x , X ) ( η2µL η1m I + H∞L ) −1 y = ( 1 m L∑ ` =1 〈 zD , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 −Ker ( x , X ) ) ( η2µL η1m I + H∞L ) −1 y − ( 1− η2µ ) k 1 m L∑ ` =1 〈 zD , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 ( η2µL η1m I + H∞L ) −1 y = ( 1 m L∑ ` =1 〈 z0 , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 −Ker ( x , X ) ) ( η2µL η1m I + H∞L ) −1 y − ( 1− η2µ ) k 1 m L∑ ` =1 〈 z0 , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 ( η2µL η1m I + H∞L ) −1 y + ( 1− ( 1− η2µ ) k ) ( 1 m L∑ ` =1 〈 zD , k ( x ) − zD,0 ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 ) ( η2µL η1m I + H∞L ) −1 y , ( 109 ) where in the second equality , ∑k−1 j=0 ( 1− η2µ ) j = 1− ( 1−η2µ ) k η2µ is used . The ` 2 norm of first term in the ( 109 ) can be bounded as : ∥∥∥∥∥ ( 1 m L∑ ` =1 〈 zD , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 −Ker ( x , X ) ) ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ∥∥∥∥∥ ( 1 m L∑ ` =1 〈 zD , k ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 −Ker ( x , X ) ) ∥∥∥∥∥ 2 · ∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 = √√√√ n∑ i=1 ( 1 m L∑ ` =1 〈 z0 ( x ) , ∇W ` [ fWD ( 0 ) ( xi ) ] 〉 −Ker ( x , xi ) ) 2 · ∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ O ( ω √ nL5/2 4 √ log ( nL/δ ) m ) · O ( η1m η2µL √ n ) = O ( ωη1mn η2µL L5/2 4 √ log ( nL/δ ) m ) = O ( ωL5/2n d 2d−1 4 √ log ( nL/δ ) m ) , ( 110 ) where , in the second inequality , we used Lemma 4.11 , and also we used∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ √ y > ( η2µL η1m I + H∞L ) −2 y ≤ √ η21m 2 η22µ 2L2 · ‖y‖22 = O ( η1m η2µL √ n ) . ( 111 ) The ` 2 norm of the second term in ( 109 ) can be easily bounded as : ∥∥∥∥∥ ( 1− η2µ ) k 1m L∑ ` =1 〈 z0 ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ∥∥∥∥∥ ( 1− η2µ ) k ( 1 m L∑ ` =1 〈 z0 ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 −Ker ( x , X ) ) ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 + ∥∥∥∥∥ ( 1− η2µ ) kKer ( x , X ) ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ( 1− η2µ ) k ∥∥∥∥∥ 1m L∑ ` =1 〈 z0 ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 −Ker ( x , X ) ∥∥∥∥∥ 2 · ∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 + ( 1− η2µ ) k ∥∥∥∥∥Ker ( x , X ) ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ( 1− η2µ ) k · O ( ω√nL3/2 4√ log ( nL/δ ) m ) · O ( η1mη2µL√n ) +O ( ( 1− η2µ ) k ) ≤ ( 1− η2µ ) k · O ( ωL3/2n d2d−1 4√ log ( nL/δ ) m ) +O ( ( 1− η2µ ) k ) . ( 112 ) Lastly , the ` 2 norm of the third term in ( 109 ) is bounded as : ∥∥∥∥∥ ( 1− ( 1− η2µ ) k ) ( 1 m L∑ ` =1 〈 zD , k ( x ) − zD,0 ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉 ) ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ( 1− ( 1− η2µ ) k ) · ∥∥∥∥∥ 1m L∑ ` =1 〈 zD , k ( x ) − zD,0 ( x ) , ∇W ` [ fWD ( 0 ) ( X ) ] 〉∥∥∥∥∥ 2 · ∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ( 1− ( 1− η2µ ) k ) · ( 1 m L∑ ` =1 ‖zD , k ( x ) − zD,0 ( x ) ‖F ∥∥∇W ` [ fWD ( 0 ) ( X ) ] ∥∥F ) · ∥∥∥∥∥ ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 ≤ ( 1− ( 1− η2µ ) k ) · ( L m O ( τ1/3L2 √ ωm log ( m ) ) · O ( √ ωmn ) ) · O ( η1m η2µL √ n ) ≤ ( 1− ( 1− η2µ ) k ) · O ( ω7/6L10/3n 4d6d−3 √log ( m ) m1/6δ1/3 ) ≤ O ( ω7/6L10/3n 4d 6d−3 √ log ( m ) m1/6δ1/3 ) , ( 113 ) where in the fourth inequality , τ = OP ( L √ ω√ m n d 2d−1 ) is plugged in . Combining the inequali- ties ( 110 ) , ( 112 ) and ( 113 ) , we get the bound on ‖D‖2 in ( 109 ) : ‖D‖22 ≤ O ( ω2L5n 2d 2d−1 √ log ( nL/δ ) m ) + ( 1− η2µ ) 2kO ( ω2L3n 2d2d−1√ log ( nL/δ ) m ) +O ( ( 1− η2µ ) 2k ) +O ( ω7/3L20/3n 8d 6d−3 log ( m ) m1/3δ2/3 ) ≤ O ( ω7/3L20/3n 8d 6d−3 log ( m ) m1/3δ2/3 ) +O ( ( 1− η2µ ) 2k ) . ( 114 ) Step 3 . Combining all pieces . Recall ‖zD , k ( x ) ‖2 ≤ O ( √ mω ) . With this fact , combining the bounds ( 94 ) , ( 97 ) , ( 106 ) , ( 107 ) , ( 108 ) and ( 114 ) , we can bound the ‖∆D,1‖22 via the decomposition ( 92 ) as follows : ‖∆D,1‖22 ≤ 1 L L∑ ` =1 ∣∣∣zD , k ( x ) > vec ( W ( 0 ) D , ` ) ∣∣∣2 + 1L L∑ ` =1 ‖zD , k ( x ) ‖22 ‖E3‖ 2 2 + 1 L L∑ ` =1 ‖zD , k ( x ) ‖22 ‖E2,1‖ 2 2 + 1 L L∑ ` =1 ‖zD , k ( x ) ‖22 ‖E2,2‖ 2 2 + 1 L L∑ ` =1 ‖zD , k ( x ) ‖22 ‖E2.3‖ 2 2 + ∥∥∥∥∥ 1L L∑ ` =1 〈zD , k ( x ) , E2,4〉 −Ker ( x , X ) ( η2µL η1m I + H∞L ) −1 y ∥∥∥∥∥ 2 2 ≤ O ( ω log ( L δ ) ) +OP ( L20/3ω4/3 m1/3 n 8d 6d−3 log ( m ) ) +OP ( ω ( 1− η2µ ) 2k L ) +OP ( nω2 ( 1− η2µ ) 2k Lλ0 ) +OP ( L2 m ω · n− 4d−3 2d−1 ) +O ( ω7/3L20/3n 8d 6d−3 log ( m ) m1/3δ2/3 ) +OP ( ( 1− η2µ ) 2k ) ≤ OP ( n− d 2d−1 ) .

Summary: In this paper, the authors analyze the convergence rate of both the unregularized and the L2 regularized gradient descent for a regression problem. Under a positivity assumption of NTK,  this paper shows that without early stopping, the vanilla GD may fail. This can be solved by using L2 regularization and could achieve a better convergence rate.