A Fine-Grained Spectral Perspective on Neural Networks

1 INTRODUCTION . Understanding the behavior of neural networks and why they generalize has been a central pursuit of the theoretical deep learning community . Recently , Valle-Pérez et al . ( 2018 ) observed that neural networks have a certain “ simplicity bias ” and proposed this as a solution to the generalization question . One of the ways with which they argued that this bias exists is the following experiment : they drew a large sample of boolean functions by randomly initializing neural networks and thresholding the output . They observed that there is a bias toward some `` simple '' functions which get sampled disproportionately more often . However , their experiments were only done for relu networks . Can one expect this “ simplicity bias ” to hold universally , for any architecture ? A priori , this seems difficult , as the nonlinear nature seems to present an obstacle in reasoning about the distribution of random networks . However , this question turns out to be more easily treated if we allow the width to go to infinity . A long line of works starting with Neal ( 1995 ) and extended recently by Lee et al . ( 2018 ) ; Novak et al . ( 2018 ) ; Yang ( 2019 ) have shown that randomly initialized , infinite-width networks are distributed as Gaussian processes . These Gaussian processes also describe finite width random networks well ( Valle-Pérez et al. , 2018 ) . We will refer to the corresponding kernels as the Conjugate Kernels ( CK ) , following the terminology of Daniely et al . ( 2016 ) . Given the CK K , the simplicity bias of a wide neural network can be read off quickly from the spectrum of K : If the largest eigenvalue of K accounts for most of trK , then a typical random network looks like a function from the top eigenspace of K. In this paper , we will use this spectral perspective to probe not only the simplicity bias , but more generally , questions regarding how hyperparameters affect the generalization of neural networks . Via the usual connection between Gaussian processes and linear models with features , the CK can be thought of as the kernel matrix associated to training only the last layer of a wide randomly initialized network . It is a remarkable recent advance ( Jacot et al. , 2018 ; Allen-Zhu et al. , 2018a ; c ; Du et al. , 2018 ) that , under a certain regime , a wide neural network of any depth evolves like a linear model even when training all parameters . The associated kernel is call the Neural Tangent Kernel , which is typically different from CK . While its theory was initially derived in the infinite width setting , Lee et al . ( 2019 ) confirmed with extensive experiment that this limit is predictive of finite width neural networks as well . Thus , just as the CK reveals information about what a network looks like at initialization , NTK reveals information about what a network looks like after training . As such , if we can understand how hyperparameters change the NTK , we can also hope to understand how they affect the performance of the corresponding finite-width network . Our Contributions In this paper , in addition to showing that the simplicity bias is not universal , we will attempt a first step at understanding the effects of the hyperparameters on generalization from a spectral perspective . At the foundation is a spectral theory of the CK and the NTK on the boolean cube . In Section 3 , we show that these kernels , as integral operators on functions over the boolean cube , are diagonalized by the natural Fourier basis , echoing similar results for over the sphere ( Smola et al. , 2001 ) . We also partially diagonalize the kernels over standard Gaussian , and show that , as expected , the kernels over the different distributions ( boolean cube , sphere , standard Gaussian ) behave very similarly in high dimensions . However , the spectrum is much easier to compute over the boolean cube : while the sphere and Gaussian eigenvalues would require integration against a kind of polynomials known as the Gegenbauer polynomials , the boolean ones only require calculating a linear combination of a small number of terms . For this reason , in the rest of the paper we focus on analyzing the eigenvalues over the boolean cube . Just as the usual Fourier basis over R has a notion of frequency that can be interpreted as a measure of complexity , so does the boolean Fourier basis ( this is just the degree ; see Section 3.1 ) . While not perfect , we adopt this natural notion of complexity in this work ; a “ simple ” function is then one that is well approximated by “ low frequencies. ” This spectral perspective immediately yields that the simplicity bias is not universal ( Section 4 ) . In particular , while it seems to hold more or less for relu networks , for sigmoidal networks , the simplicity bias can be made arbitrarily weak by changing the weight variance and the depth . In the extreme case , the random function obtained from sampling a deep erf network with large weights is distributed like a “ white noise. ” However , there is a very weak sense in which the simplicity bias does hold : the eigenvalues of more “ complex ” eigenspaces can not be bigger than those of less “ complex ” eigenspaces ( Thm 4.1 ) . Next , we examine how hyperparameters affect the performance of neural networks through the lens of NTK and its spectrum . To do so , we first need to understand the simpler question of how a kernel affects the accuracy of the function learned by kernel regression . A coarse-grained theory , concerned with big-O asymptotics , exists from classical kernel literature ( Yao et al. , 2007 ; Raskutti et al. , 2013 ; Wei et al . ; Lin and Rosasco ; Schölkopf and Smola , 2002 ) . However , the fine-grained details , required for discerning the effect of hyperparameters , have been much less studied . We make a first attempt at a heuristic , fractional variance ( i.e . what fraction of the trace of the kernel does an eigenspace contribute ) , for understanding how a minute change in kernel effects a change in performance . Intuitively , if an eigenspace has very large fractional variance , so that it accounts for most of the trace , then a ground truth function from this eigenspace should be very easy to learn . Using this heuristic , we make two predictions about neural networks , motivated by observations in the spectra of NTK and CK , and verify them with extensive experiments . • Deeper networks learn more complex features , but excess depth can be detrimental as well . Spectrally , depth can increase fractional variance of an eigenspace , but past an optimal depth , it will also decrease it . ( Section 5 ) Thus , deeper is not always better . • Training all layers is better than training just the last layer when it comes to more complex features , but the opposite is true for simpler features . Spectrally , fractional variances of more “ complex ” eigenspaces for the NTK are larger than the correponding quantities of the CK . ( Section 6 ) Finally , we use our spectral theory to predict the maximal nondiverging learning rate ( “ max learning rate ” ) of SGD ( Section 7 ) . In general , we will not only verify our theory with experiments on the theoretically interesting distributions , i.e . uniform measures over the boolean cube and the sphere , or the standard Gaussian , but also confirm these findings on real data like MNIST and CIFAR10 1 . 1The code for computing the eigenvalues and for reproducing the plots of this paper is available at github . com/jxVmnLgedVwv6mNcGCBy/NNspectra , which will be open sourced upon publication . For space concerns , we review relevant literature along the flow of the main text , and relegate a more complete discussion of the related research landscape in Appendix A . 2 KERNELS ASSOCIATED TO NEURAL NETWORKS . As mentioned in the introduction , we now know several kernels associated to infinite width , randomly initialized neural networks . The most prominent of these are the neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) and the conjugate kernel ( CK ) ( Daniely et al. , 2016 ) , which is also called the NNGP kernel ( Lee et al. , 2018 ) . We briefly review them below . First we introduce the following notation that we will repeatedly use . Definition 2.1 . For φ : R→ R , write Vφ for the function that takes a PSD ( positive semidefinite ) kernel function to a PSD kernel of the same domain by the formula Vφ ( K ) ( x , x ′ ) = E f∼N ( 0 , K ) φ ( f ( x ) ) φ ( f ( x′ ) ) . Conjugate Kernel Neural networks are commonly thought of as learning a high-quality embedding of inputs to the latent space represented by the network ’ s last hidden layer , and then using its final linear layer to read out a classification given the embedding . The conjugate kernel is just the kernel associated to the embedding induced by a random initialization of the neural network . Consider an MLP with widths { nl } l , weight matrices { W l ∈ Rn l×nl−1 } l , and biases { bl ∈ Rn l } l , l = 1 , . . . , L. For simplicity of exposition , in this paper , we will only consider scalar output nL = 1 . Suppose it is parametrized by the NTK parametrization , i.e . its computation is given recursively as h1 ( x ) = σw√ n0 W 1x+ σbb 1 and hl ( x ) = σw√ nl−1 W lφ ( hl−1 ( x ) ) + σbb l ( MLP ) with some hyperparameters σw , σb that are fixed throughout training2 . At initialization time , suppose W lαβ , b l α ∼ N ( 0 , 1 ) for each α ∈ [ nl ] , β ∈ [ nl−1 ] . It can be shown that , for each α ∈ [ nl ] , hlα is a Gaussian process with zero mean and kernel function Σl in the limit as all hidden layers become infinitely wide ( nl →∞ , l = 1 , . . . , L− 1 ) , where Σl is defined inductively on l as Σ1 ( x , x′ ) def = σ2w ( n 0 ) −1〈x , x′〉+ σ2b , Σl def = σ2wVφ ( Σ l−1 ) + σ2b ( CK ) The kernel ΣL corresponding the the last layer L is the network ’ s conjugate kernel , and the associated Gaussian process limit is the reason for its alternative name Neural Network-Gaussian process kernel . In short , if we were to train a linear model with features given by the embedding x 7→ hL−1 ( x ) when the network parameters are randomly sampled as above , then the CK is the kernel of this linear model . See Daniely et al . ( 2016 ) ; Lee et al . ( 2018 ) and Appendix F for more details . Neural Tangent Kernel On the other hand , the NTK corresponds to training the entire model instead of just the last layer . Intuitively , if we let θ be the entire set of parameters { W l } l ∪ { bl } l of Eq . ( MLP ) , then for θ close to its initialized value θ0 , we expect hL ( x ; θ ) − hL ( x ; θ0 ) ≈ 〈∇θhL ( x ; θ0 ) , θ − θ0〉 via a naive first-order Taylor expansion . In other words , hL ( x ; θ ) − hL ( x ; θ0 ) behaves like a linear model with feature of x given by the gradient taken w.r.t . the initial network , ∇θhL ( x ; θ0 ) , and the weights of this linear model are the deviation θ− θ0 of θ from its initial value . It turns out that , in the limit as all hidden layer widths tend to infinity , this intuition is correct ( Jacot et al. , 2018 ; Lee et al. , 2018 ; Yang , 2019 ) , and the following inductive formula computes the corresponding infinite-width kernel of this linear model : Θ1 def = Σ1 , Θl ( x , x′ ) def = Σl ( x , x′ ) + σ2wΘ l−1 ( x , x′ ) Vφ′ ( Σ l−1 ) ( x , x′ ) . ( NTK ) Computing CK and NTK While in general , computing Vφ and Vφ′ requires evaluating a multivariate Gaussian expectation , in specific cases , such as when φ = relu or erf , there exists explicit , efficient formulas that only require pointwise evaluation of some simple functions ( see Facts F.1 and F.2 ) . This allows us to evaluate CK and NTK on a set X of inputs in only time O ( |X |2L ) . 2SGD with learning rate α in this parametrization is roughly equivalent to SGD with learning rate α/width in the standard parametrization with Glorot initialization ; see Lee et al . ( 2018 ) What Do the Spectra of CK and NTK Tell Us ? In summary , the CK governs the distribution of a randomly initialized neural network and also the properties of training only the last layer of a network , while the NTK governs the dynamics of training ( all parameters of ) a neural network . A study of their spectra thus informs us of the “ implicit prior ” of a randomly initialized neural network as well as the “ implicit bias ” of GD in the context of training neural networks . In regards to the implicit prior at initialization , we know from Lee et al . ( 2018 ) that a randomly initialized network as in Eq . ( MLP ) is distributed as a Gaussian process N ( 0 , K ) , where K is the corresponding CK , in the infinite-width limit . If we have the eigendecomposition K = ∑ i≥1 λiui ⊗ ui ( 1 ) with eigenvalues λi in decreasing order and corresponding eigenfunctions ui , then each sample from this GP can be obtained as ∑ i≥1 √ λiωiui , ωi ∼ N ( 0 , 1 ) . If , for example , λ1 ∑ i≥2 λi , then a typical sample function is just a very small perturbation of u1 . We will see that for relu , this is indeed the case ( Section 4 ) , and this explains the “ simplicity bias ” in relu networks found by Valle-Pérez et al . ( 2018 ) . Training the last layer of a randomly initialized network via full batch gradient descent for an infinite amount of time corresponds to Gaussian process inference with kernel K ( Lee et al. , 2018 ; 2019 ) . A similar intuition holds for NTK : training all parameters of the network ( Eq . ( MLP ) ) for an infinite amount of time yields the mean prediction of the GPN ( 0 , NTK ) in expectation ; see Lee et al . ( 2019 ) and Appendix F.4 for more discussion . Thus , the more the GP prior ( governed by the CK or the NTK ) is consistent with the ground truth function f∗ , the more we expect the Gaussian process inference and GD training to generalize well . We can measure this consistency in the “ alignment ” between the eigenvalues λi and the squared coefficients a2i of f ∗ ’ s expansion in the { ui } i basis . The former can be interpreted as the expected magnitude ( squared ) of the ui-component of a sample f ∼ N ( 0 , K ) , and the latter can be interpreted as the actual magnitude squared of such component of f∗ . In this paper , we will investigate an even cleaner setting where f∗ = ui is an eigenfunction . Thus we would hope to use a kernel whose ith eigenvalue λi is as large as possible . Neural Kernels From the forms of the equation Eqs . ( CK ) and ( NTK ) and the fact that Vφ ( K ) ( x , x ′ ) only depends on K ( x , x ) , K ( x , x′ ) , and K ( x′ , x′ ) , we see that CK or NTK of MLPs takes the form K ( x , y ) = Φ ( 〈x , y〉 ‖x‖‖y‖ , ‖x‖2 d , ‖y‖2 d ) ( 2 ) for some function Φ : R3 → R. We will refer to this kind of kernel as Neural Kernel in this paper . Kernels as Integral Operators We will consider input spaces of various forms X ⊆ Rd equipped with some probability measure . Then a kernel function K acts as an integral operator on functions f ∈ L2 ( X ) by Kf ( x ) = ( Kf ) ( x ) = E y∼X K ( x , y ) f ( y ) . We will use the “ juxtaposition syntax ” Kf to denote this application of the integral operator . 3 Under certain assumptions , it then makes sense to speak of the eigenvalues and eigenfunctions of the integral operator K. While we will appeal to an intuitive understanding of eigenvalues and eigenfunctions in the main text below , we include a more formal discussion of Hilbert-Schmidt operators and their spectral theory in Appendix G for completeness . In the next section , we investigate the eigendecomposition of neural kernels as integral operators over different distributions . 3In cases when X is finite , K can be also thought of as a big matrix and f as a vector — but do not confuse Kf with their multiplication ! If we use · to denote matrix multiplication , then the operator application Kf is the same as the matrix multiplication K ·D · f where D is the diagonal matrix encoding the probability values of each point in X .

Summary: Aiming to resolve the question whether and why deep networks are biased towards simple functions, this paper gives a spectral analysis on neural networks' conjugate kernel(CK) and neural tangent kernel(NTK) on boolean cube. The eigenfunctions are identified and the eigenvalues are shown computable in polynomial time. Another main contribution of this paper is showing that the simplicity bias exists at least in a weak sense.