Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks

1 INTRODUCTION . A surprising but well established empirical fact is that neural networks optimized by gradient descent can find solutions to the empirical risk minimization ( ERM ) problem that generalize . This is surprising from an optimization point-of-view because the ERM problem induced by neural networks is nonconvex ( Sontag & Sussmann , 1989 ; 1991 ) and can even be NP-Complete in certain cases ( Blum & Rivest , 1993 ) . Perhaps even more surprising is that the discovered solution can generalize even when the network is able to fit arbitrary labels ( Zhang et al. , 2017 ) , rendering traditional complexity measures such as Rademacher complexity inadequate . How does deep learning succeed in the face of pathological behavior by the standards of classical optimization and statistical learning theory ? Towards addressing generalization , a modern line of thought that has emerged is that gradient descent performs implicit regularization , limiting the solutions one encounters in practice to a favorable subset of the model ’ s full capacity ( see , e.g. , Neyshabur et al. , 2015 ; 2017 ; Gunasekar et al. , 2017 ; Wu et al. , 2017 ) . An empirical observation is that neural networks optimized by gradient descent tend to fit the low frequencies of the target function first , and only pick up the higher frequencies later in training ( Rahaman et al. , 2019 ; Ronen et al. , 2019 ; Basri et al. , 2020 ; Xu et al. , 2019 ) . A closely related theme is gradient descent ’ s bias towards smoothness for regression problems ( Williams et al. , 2019 ; Jin & Montúfar , 2021 ) . For classification problems , in suitable settings gradient descent provably selects max-margin solutions ( Soudry et al. , 2018 ; Ji & Telgarsky , 2019 ) . Gradient descent is not impartial , thus understanding its bias is an important program in modern deep learning . Generalization concerns aside , the fact that gradient descent can succeed in a nonconvex optimization landscape warrants attention on its own . A brilliant insight made by Jacot et al . ( 2018 ) is that in function space the neural network follows a kernel gradient descent with respect to the “ Neural Tangent Kernel ” ( NTK ) . This kernel captures how the parameterization biases the trajectory in function space , an abstraction that allows one to largely ignore parameter space and its complications . This is a profitable point-of-view , but there is a caveat . The NTK still depends on the evolution of the network parameters throughout time , and thus is in general time-dependent and complicated to analyze . However , under appropriate scaling of the parameters in the infinite-width limit it remains constant ( Jacot et al. , 2018 ) . Once the NTK matrix has small enough deviations to remain strictly positive definite throughout training , the optimization dynamics start to become comparable to that of a linear model ( Lee et al. , 2019 ) . For wide networks ( quadratic or higher polynomial dependence on the number of training data samples n and other parameters ) this property holds and this has been used by a variety of works to prove global convergence guarantees for the optimization ( Du et al. , 2019b ; Oymak & Soltanolkotabi , 2020 ; Du et al. , 2019a ; Allen-Zhu et al. , 2019a ; b ; Zou et al. , 2020 ; Zou & Gu , 2019 ; Song & Yang , 2020 ; Dukler et al. , 2020 ) 1 and to characterize the solution throughout time ( Arora et al. , 2019 ; Basri et al. , 2020 ) . The NTK has been so heavily exploited in this setting that it has become synonymous with polynomially wide networks where the NTK is strictly positive definite throughout training . This begs the question , to what extent is the NTK informative outside this regime ? While the NTK has hitherto been associated with the heavily overparameterized regime , we demonstrate that refined analysis is possible in the underparameterized setting . Our theorems primarily concern a one-hidden layer network , however unlike many NTK results appearing in the literature our network has biases and both layers are trained . In fact , the machinery we build is strong enough to extend some existing results in the overparameterized regime appearing in the literature to the case of training both layers . 1.1 RELATED WORK . There has been a deluge of works on the Neural Tangent Kernel since it was introduced by Jacot et al . ( 2018 ) , and thus we do our best to provide a partial list . Global convergence guarantees for the optimization , and to a lesser extent generalization , for networks polynomially wide in the number of training samples n and other parameters has been addressed in several works ( Du et al. , 2019b ; Oymak & Soltanolkotabi , 2020 ; Du et al. , 2019a ; Allen-Zhu et al. , 2019a ; b ; Zou et al. , 2020 ; Zou & Gu , 2019 ; Song & Yang , 2020 ; Arora et al. , 2019 ) . To our knowledge , for the regression problem with arbitrary labels , quadratic overparameterization m & n2 is state-of-the art ( Oymak & Soltanolkotabi , 2020 ; Song & Yang , 2020 ; Nguyen & Mondelli , 2020 ) . E et al . ( 2020 ) gave a fairly comprehensive study of optimization and generalization of shallow networks trained under the standard parameterization . Under the standard parameterization , changes in the outer layer weights are more significant , whereas under the NTK parameterization both layers have roughly equal effect . Since we study the NTK parameterization in this work , we view the analysis as complementary . Our work is perhaps most closely connected with Arora et al . ( 2019 ) . In Theorem 4.1 in that work they showed that for a shallow network in the polynomially overparameterized regime m & n7 , the training error along eigendirections of the NTK matrix decay linearly at rates that correspond to their eigenvalues . Our main Theorem 3.5 can be viewed as an analogous statement for the actual risk ( not the empirical risk ) in the underparameterized regime : eigenfunctions of the NTK integral operator TK∞ are approximately learned linearly at rates that correspond to their eigenvalues . In contrast with Arora et al . ( 2019 ) , we have that the requirements on width m and number of samples n required to learn eigenfunctions with large eigenvalues are smaller compared to those with small eigenvalues . Surprisingly the machinery we build is also strong enough to prove in our setting the direct analog of Theorem 4.1 . Note that Arora et al . ( 2019 ) train the hidden layer of a ReLU network via gradient descent , whereas we are training both layers with biases for a network with smooth activations via gradient flow . Due to the different settings , the results are not directly comparable . This important detail notwithstanding , our overparameterization requirement ignoring logarithmic factors is smaller by a factor of n 2 dδ4 where n is the number of input samples , d is the input dimension , and δ is the failure probability . Basri et al . ( 2020 ) extended Theorem 4.1 in Arora et al . ( 2019 ) to deep ReLU networks without bias where the first and last layer are fixed , with a higher overparameterization requirement than the original ( Arora et al. , 2019 ) . Since the first and last layers are fixed this can not be specialized to get a guarantee for training both layers of a shallow network even with ReLU activations . Although it was not our focus , the tools to prove Theorem 3.5 are enough to prove analogs of Theorem 4 and Corollary 2 in the work of Su & Yang ( 2019 ) . Theorem 4 and Corollary 2 of Su & Yang ( 2019 ) are empirical risk guarantees that show that for target functions that participate mostly in the top eigendirections of the NTK integral operator TK∞ , moderate overparameterization is possible . Again in this work they train the hidden layer of a ReLU network via gradient descent , 1Not all these works explicitly use that the NTK is positive definite . However , they all operate in the regime where the weights do not vary much and thus are typically associated with the NTK regime . whereas we are training both layers with biases for a network with smooth activations via gradient flow . Again due to the different settings , we emphasize the results are not directly comparable . In our results the bounds and requirements are comparable to Su & Yang ( 2019 ) , with neither appearing better . Nevertheless we think it is important to demonstrate that these results hold for training both layers with biases , and we hope our “ Damped Deviations ” approach will simplify the interpretation of the aforementioned works . Cao et al . ( 2020 , Theorem 4.2 ) provide an analogous statement to our Theorem 3.5 if you replace our quantities with their empirical counterparts . While our statement concerns the projections of the test residual onto the eigenfunctions of an operator associated with the Neural Tangent Kernel , their statement concerns the inner products of the empirical residual with those eigenfunctions . Their work was a crucial step towards explaining the spectral bias from gradient descent , however we view the difference between tracking the empirical quantities versus the actual quantities to be highly nontrivial . Another difference is they consider a ReLU network whereas we consider smooth activations ; also they consider gradient descent versus we consider gradient flow . Due to the different settings we would like to emphasize that the scalings of the different parameters are not directly comparable , nevertheless the networks they consider are significantly wider . They require at least m ≥ Õ ( max { σ−14k , −6 } ) , where σk is a cutoff eigenvalue and is the error tolerance . By contrast in our work , to have the projection onto the top k eigenvectors be bounded by epsilon in L2 norm requires m = Ω̃ ( σ−4k −2 ) . Another detail is their network has no bias whereas ours does . 1.2 OUR CONTRIBUTIONS . The key idea for our work is the concept of “ Damped Deviatons ” , the fact that for the squared error deviations of the NTK are softened by a damping factor , with large eigendirections being damped the most . This enables the following results . • In Theorem 3.5 we characterize the bias of the neural network to learn the eigenfunctions of the integral operator TK∞ associated with the Neural Tangent Kernel ( NTK ) at rates proportional to the corresponding eigenvalues . • In Theorem 3.7 we show that in the overparameterized setting the training error along different directions can be sharply characterized , showing that Theorem 4.1 in Arora et al . ( 2019 ) holds for smooth activations when training both layers with a smaller overparameterization requirement . • In Theorem 3.8 and Corollary 3.9 we show that moderate overparameterization is sufficient for solving the ERM problem when the target function has a compact representation in terms of eigenfunctions of TK∞ . This extends the results in Su & Yang ( 2019 ) to the setting of training both layers with smooth activations . 2 GRADIENT DYNAMICS AND DAMPED DEVIATIONS . 2.1 NOTATIONS . We will use ‖•‖2 and 〈• , •〉2 to denote the L2 norm and inner product respectively ( for vectors or for functions depending on context ) . For a symmetric matrix A ∈ Rk×k , λi ( A ) denotes its ith largest eigenvalue , i.e . λ1 ( A ) ≥ λ2 ( A ) ≥ · · · ≥ λk ( A ) . For a matrix A , ‖A‖op : = sup‖x‖2≤1 ‖Ax‖2 is the operator norm induced by the Euclidean norm . We will let 〈• , •〉Rn denote the standard inner product on Rn normalized by 1n , namely 〈x , y〉Rn = 1 n 〈x , y〉2 = 1 n ∑n i=1 xiyi . We will let ‖x‖Rn = √ 〈x , x〉Rn be the associated norm . This normalized inner product has the convenient property that if v ∈ Rn such that vi = O ( 1 ) for each i then ‖v‖Rn = O ( 1 ) , where by contrast ‖v‖2 = O ( √ n ) . This is convenient as we will often consider what happens when n → ∞ . ‖•‖∞ will denote the supremum norm with associated space L∞ . We will use the standard big O and Ω notation with Õ and Ω̃ hiding logarithmic terms .

Summary: This work studies the implicit bias of optimizing underparametrized neural networks with MSE loss function and establishes upper-bounds on the training error. One of the takeaways of the paper is that under-parametrized neural networks first learn the most important eigenfunctions associated with the NTK integral operator. This is what the work posits as the implicit bias of gradient-descent in this setting. A key tool developed in the paper is the concept of damped deviations, where in the evolution of MSe w.r.t to one kernel is related to that of another kernel and the damped "distance" between these kernels. As a byproduct, the paper uses this tool to extend certain results in the literature in the over-parametrized setting.