Towards Understanding the Spectral Bias of Deep Learning

1 INTRODUCTION . Over-parameterized neural networks have achieved great success in many applications such as computer vision ( He et al. , 2016 ) , natural language processing ( Collobert and Weston , 2008 ) and speech recognition ( Hinton et al. , 2012 ) . It has been shown that over-parameterized neural networks can fit complicated target function or even randomly labeled data ( Zhang et al. , 2016 ) and still exhibit good generalization performance when trained with real labels . Intuitively , this is at odds with the traditional notion of generalization ability such as model complexity . In order to understand neural network training , a line of work ( Soudry et al. , 2017 ; Gunasekar et al. , 2018b ; a ) has made efforts in the perspective of “ implicit bias ” , which states that training algorithms for deep learning implicitly pose an inductive bias onto the training process and lead to a solution with low complexity measured by certain norms in the parameter space of the neural network . Among many attempts to establish implicit bias , Rahaman et al . ( 2018 ) pointed out an intriguing phenomenon called spectral bias , which says that during training , neural networks tend to learn the components of lower complexity faster . The concept of spectral bias is appealing because this may intuitively explain why over-parameterized neural networks can achieve a good generalization performance without overfitting . During training , the networks fit the low complexity components first and thus lie in the concept class of low complexity . Arguments like this may lead to rigorous guarantee for generalization . Great efforts have been made in seek of explanations about the spectral bias . Rahaman et al . ( 2018 ) evaluated the Fourier spectrum of ReLU networks and empirically showed that the lower frequencies are learned first ; also lower frequencies are more robust to random perturbation . Andoni et al . ( 2014 ) showed that for a sufficiently wide two-layer network , gradient descent with respect to the second layer can learn any low degree bounded polynomial . Xu ( 2018 ) provided Fourier analysis to twolayer networks and showed similar empirical results on one-dimensional functions and real data . Nakkiran et al . ( 2019 ) used information theoretical approach to show that networks obtained by stochastic gradient descent can be explained by a linear classifier during early training . All these studies provide certain explanations about why neural networks exhibit spectral bias in real tasks . But explanations in the theoretical aspect , if any , are to some extent restricted . For example , the popular Fourier analysis is usually done in the one-dimensional setting , and thus lacks generality . Meanwhile , a recent line of work ( Jacot et al. , 2018 ; Du et al. , 2018b ; Li and Liang , 2018 ; Chizat and Bach , 2018 ) has shed light on new approaches to analyze neural networks . In particular , they show that under certain over-parameterized condition , the neural network trained by gradient descent behaves similar to the kernel regression predictor using the neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) . Du et al . ( 2018b ) showed that the convergence is provably guaranteed under certain over-parameterization conditions determined by the smallest eigenvalue of NTK . Arora et al . ( 2019b ) further gave a finer characterization of error convergence based on the eigenvalues of NTK ’ s Gram matrix . Su and Yang ( 2019 ) improved the convergence guarantee in terms of the k-th largest eigenvalue for certain target functions . Inspired by these works mentioned above , we can present a theoretical explanation for spectral bias . Under NTK regime , we establish a precise characterization for the training process of neural networks . More specifically , we theoretically prove that over-parameterized neural networks ’ training process can be controlled by the eigenvalues of the integrating operator defined by the NTK . Under the specific case of uniform distribution on unit sphere , we give an exact calculation for these eigenvalues and show that the lower frequencies have larger eigenvalues , which thus leads to faster convergence . We also conduct experiments to corroborate the theory we establish . Our contributions are highlighted as follows : 1 . We prove a generic theorem for arbitrary data distributions , which states that under certain sample complexity and over-parameterization conditions , the error term ’ s convergence along different directions actually relies on the corresponding eigenvalues . This theorem gives a more precise control on the regression residual than Su and Yang ( 2019 ) , where the authors focused on the case when the labeling function is close to the subspace spanned by the first few eigenfunctions . 2 . We present a more general result about the spectra of the neural tangent kernel . In particular , we show that the order of eigenvalues appears as µk “ Ωpmaxtk´d´1 , d´k ` 1uq . Our result is better than the bound Ωpk´d´1q derived in Bietti and Mairal ( 2019 ) when d `` k , which is clearly a more practical setting . 3 . We establish a rigorous explanation for the spectral bias based on the aforementioned theoretical results without any specific assumptions on the target function . We show that the error terms from different frequencies are provably controlled by the eigenvalues of the NTK , and the lowerfrequency components can be learned with less training examples and narrower networks with a faster convergence rate . As far as we know , this is the first attempt to give a comprehensive theory justifying the existence of spectral bias . 1.1 ADDITIONAL RELATED WORK . Recently , there is a rich literature about the property of neural tangent kernel . Jacot et al . ( 2018 ) first showed that during training , the network function follows a descent along the kernel gradient with respect to the Neural Tangent Kernel ( NTK ) under infinity width setting . Li and Liang ( 2018 ) and Du et al . ( 2018b ) implicitly built connection between Neural Tangent Kernel and gradient descent by showing that GD can provably optimize sufficiently wide two-layer neural networks . In Du et al . ( 2018b ) , it is shown that gradient descent can achieve zero training loss at a linear convergence rate for training two-layer ReLU network with square loss . Allen-Zhu et al . ( 2018 ) ; Du et al . ( 2018a ) ; Zou et al . ( 2018 ) ; Cao and Gu ( 2019b ) ; Arora et al . ( 2019a ) ; Zou and Gu ( 2019 ) ; Cao and Gu ( 2019a ) further studied the optimization and generalization of deep neural networks . These papers are all in the so-called neural tangent kernel regime , and their requirements on the network width depend either implicitly or explicitly on the smallest eigenvalue of the kernel Gram matrix . Later , Su and Yang ( 2019 ) showed that this smallest eigenvalue actually scales in the number of samples n and will eventually converge to 0 . In order to obtain constant convergence rate , Su and Yang ( 2019 ) assumed that the target function f˚ can be approximated by the first few eigenfunctions of the integrating operator Lκfpsq : “ ş Sd κpx , sqfpsqdτpsq where κp¨ , ¨q is the NTK function and τpsq is the input distribution , and proved linear convergence rate up to the this approximation error . A few theoretical results have been established towards understanding the spectra of neural tangent kernels . Bach ( 2017 ) studied two-layer ReLU networks by relating it to kernel methods , and proposed a harmonic decomposition for the functions in the reproducing kernel Hilbert space which we utilize in our proof . Based on the technique in Bach ( 2017 ) , Bietti and Mairal ( 2019 ) studied the eigenvalue decay of integrating operator Lκfpxq defined by NTK on unit sphere by using spheri- cal harmonics . Vempala and Wilmes ( 2018 ) calculated the eigenvalues of NTK corresponding to two-layer neural networks with sigmoid activation function . Basri et al . ( 2019 ) established similar results as Bietti and Mairal ( 2019 ) , but considered the case of training the first layer parameters of a two-layer networks with bias terms . Yang and Salman ( 2019 ) studied the the eigenvalues of integral operator with respect to the NTK on Boolean cube by Fourier analysis . The rest of the paper is organized as follows . We state the notation , problem setup and other preliminaries in Section 2 and present our main results in Section 3 . In Section 4 , we present experimental results to support our theory . Proofs of our main results can be found in the appendix . 2 PRELIMINARIES . In this section we introduce the basic problem setup including the neural network structure and the training algorithm , as well as some background on the neural tangent kernel proposed recently in Jacot et al . ( 2018 ) and the corresponding integral operator . 2.1 NOTATION . We use lower case , lower case bold face , and upper case bold face letters to denote scalars , vectors and matrices respectively . For a vector v “ pv1 , . . . , vdqT P Rd and a number 1 ď p ă 8 , we denote its p´norm by } v } p “ p řd i “ 1 |vi|pq1 { p. We also define infinity norm by } v } 8 “ maxi |vi| . For a matrix A “ pAi , jqmˆn , we use } A } 0 to denote the number of non-zero entries of A , and use } A } F “ p řd i , j “ 1A 2 i , jq1 { 2 to denote its Frobenius norm . Let } A } p “ max } v } pď1 } Av } p for p ě 1 , and } A } max “ maxi , j |Ai , j | . For two matrices A , B P Rmˆn , we define xA , By “ TrpAJBq . We use A ľ B if A´B is positive semi-definite . In addition , we define the asymptotic notations Op¨q , rOp¨q , Ωp¨q and rΩp¨q as follows . Suppose that an and bn be two sequences . We write an “ Opbnq if lim supnÑ8 |an { bn| ă 8 , and an “ Ωpbnq if lim infnÑ8 |an { bn| ą 0 . We use rOp¨q and rΩp¨q to hide the logarithmic factors in Op¨q and Ωp¨q . 2.2 PROBLEM SETUP . Here we introduce the basic problem setup . We consider two-layer fully connected neural networks of the form fWpxq “ ? m ¨W2σpW1xq , where W1 P Rmˆpd ` 1q , W2 P R1ˆm1 are the first and second layer weight matrices respectively , and σp¨q “ maxt0 , ¨u is the entry-wise ReLU activation function . The network is trained according to the square loss on n training examples S “ tpxi , yiq : i P rnsu : LSpWq “ 1 n ÿ pxi , yiqPS pyi ´ θfWpxiqq2 , where θ is a small coefficient to control the effect of initialization , and the data inputs txiuni “ 1 is assumed to follow some unknown distribution τ on the unit sphere Sd P Rd ` 1 . Without loss of generality , we also assume that |yi| ď 1 . We first randomly initialize the parameters of the network , and then apply gradient descent to optimize both layers . We present our detailed neural network training algorithm in Algorithm 1 . Algorithm 1 GD for DNNs starting at Gaussian initialization Input : Number of iterations T , step size η . Generate each entry of Wp0q1 and W p0q 2 from Np0 , 2 { mq and Np0 , 1 { mq respectively . for t “ 0 , 1 , . . . , T ´ 1 do Update Wpt ` 1q “ Wptq ´ η ¨∇WLSpWptqq . end for Output : WpT q . 1Here the dimension of input is d ` 1 since throughout this paper we assume that all training data lie in the d-dimensional unit sphere Sd P Rd ` 1 . The initialization scheme for Wp0q given in Algorithm 1 is known as He initialization ( He et al. , 2015 ) . This scheme generates each entry of the weight matrices from a Gaussian distribution with mean zero . The variances of the Gaussian distributions in initialization are chosen following the principle that the initialization does not change the magnitudes of inputs in each layer . The second layer parameter is not associated with the ReLU activation function , thus it is initialized with variance 1 { m instead of 2 { m .

Summary: This paper studies the training of overparametrized neural networks by gradient descent. More precisely, the authors consider the neural tangent regime (NTK regime). That is, the weights are chosen sufficiently large and the neural network is sufficiently overparametrized. It has been observed that in this scenario, the neural network behaves approximately like a linear function of its weights.