Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

1 INTRODUCTION . Understanding the training process of Deep Neural Networks ( DNNs ) is a fundamental problem in the area of deep learning . We find a common behavior of the gradient-based training process of DNNs , that is , a Frequency Principle ( F-Principle ) : DNNs often fit target functions from low to high frequencies during the training process . In another word , at the early stage of training , the low-frequencies are fitted and as iteration steps of training increase , the high-frequencies are fitted . For example , when a DNN is trained to fit y = sin ( x ) + sin ( 2x ) , its output would be close to sin ( x ) at early stage and as training goes on , its output would be close to sin ( x ) + sin ( 2x ) . F-Principle was observed empirically in synthetic low-dimensional data with MSE loss during DNN training ( Xu et al. , 2018 ; Rahaman et al. , 2018 ) . However , in deep learning , empirical phenomena could vary from one network structure to another , from one dataset to another and could exhibit significant difference between synthetic data and highdimensional real data . Therefore , the universality of the F-Principle remains an important problem for further study . Especially for high-dimensional real problems , because the computational cost of high-dimensional Fourier transform is prohibitive in practice , it is of great challenge to demonstrate the F-Principle . On the other hand , the mechanism underlying the F-Principle and its implication to the application of DNNs , e.g. , design of DNN-based PDE solver , as well as their generalization ability are also important open problems to be addressed . In this work , we design two methods , i.e. , projection and filtering methods , to show that the FPrinciple exists in the training process of DNNs for high-dimensional benchmarks , i.e. , MNIST ( LeCun , 1998 ) , CIFAR10 ( Krizhevsky et al. , 2010 ) . The settings we have considered are i ) different DNN architectures , e.g. , fully-connected network , convolutional neural network ( CNN ) , and VGG16 ( Simonyan & Zisserman , 2014 ) ; ii ) different activation functions , e.g. , tanh and rectified linear unit ( ReLU ) ; iii ) different loss functions , e.g. , cross entropy , mean squared error ( MSE ) , and loss energy functional in variational problems . These results demonstrate the universality of the F-Principle . To facilitate the designs and applications of DNN-based schemes , we characterize a stark difference between DNNs and conventional numerical schemes on various scientific computing problems , where most of the conventional methods ( e.g. , Jacobi method ) exhibit the opposite convergence behavior — faster convergence for higher frequencies . This difference implies that DNN can be adopted to accelerate the convergence of low frequencies for computational problems . We also intuitively explain with theories under an idealized setting how the smoothness/regularity of commonly used activation functions contributes to the F-Principle . Note that this mechanism is rigorously demonstrated for DNNs of general settings in a subsequent work ( Luo et al. , 2019 ) . Finally , we discuss that the F-Principle provides an understanding of good generalization of DNNs in many real datasets ( Zhang et al. , 2016 ) and poor generalization in learning the parity function ( Shalev-Shwartz et al. , 2017 ; Nye & Saxe , 2018 ) , that is , the F-Principle which implies that DNNs prefer low frequencies , is consistent with the property of low frequencies dominance in many real datasets , e.g. , MNIST/CIFAR10 , but is different from the parity function whose spectrum concentrates on high frequencies . Compared with previous studies , our main contributions are as follows : 1 . By designing both the projection and filtering methods , we consistently demonstrate the F-Principle for MNIST/CIFAR10 over various architectures such as VGG16 and various loss functions . 2 . For the application of solving differential equations , we show that ( i ) conventional numerical schemes learn higher frequencies faster whereas DNNs learn lower frequencies faster by the FPrinciple , ( ii ) convergence of low frequencies can be greatly accelerated with DNN-based schemes . 3 . We present theories under an idealized setting to illustrate how smoothness/regularity of activation function contributes to the F-Principle . 4 . We discuss in detail the implication of the F-Principle to the generalization of DNNs that DNNs are implicitly biased towards a low frequency function and provide an explanation of good and poor generalization of DNNs for low and high frequency dominant target functions , respectively . 2 FREQUENCY PRINCIPLE . The concept of “ frequency ” is central to the understanding of F-Principle . In this paper , the “ frequency ” means response frequency NOT image ( or input ) frequency as explained in the following . Image ( or input ) frequency ( NOT used in the paper ) : Frequency of 2-d function I : R2 → R representing the intensity of an image over pixels at different locations . This frequency corresponds to the rate of change of intensity across neighbouring pixels . For example , an image of constant intensity possesses only the zero frequency , i.e. , the lowest frequency , while a sharp edge contributes to high frequencies of the image . Response frequency ( used in the paper ) : Frequency of a general Input-Output mapping f . For example , consider a simplified classification problem of partial MNIST data using only the data with label 0 and 1 , f ( x1 , x2 , · · · , x784 ) : R784 → { 0 , 1 } mapping 784-d space of pixel values to 1-d space , where xj is the intensity of the j-th pixel . Denote the mapping ’ s Fourier transform as f̂ ( k1 , k2 , · · · , k784 ) . The frequency in the coordinate kj measures the rate of change of f ( x1 , x2 , · · · , x784 ) with respect to xj , i.e. , the intensity of the j-th pixel . If f possesses significant high frequencies for large kj , then a small change of xj in the image might induce a large change of the output ( e.g. , adversarial example ) . For a dataset with multiple classes , we can similarly define frequency for each output dimension . For real data , the response frequency is rigorously defined via the standard nonuniform discrete Fourier transform ( NUDFT ) , see Appendix A . Frequency Principle : DNNs often fit target functions from low to high ( response ) frequencies during the training process . An illustration of F-Principle using a function of 1-d input is in Appendix B . The F-Principle is rigorously defined through the frequency defined by the Fourier transform ( Appendix A , Bracewell & Bracewell ( 1986 ) ) and the converging speed defined by the relative error . By using high-dimensional real datasets , we then experimentally demonstrate F-Principle at the levels of both individual frequencies ( projection method ) and coarse-grained frequencies ( filtering method ) . 3 F-PRINCIPLE IN MNIST/CIFAR10 THROUGH PROJECTION METHOD . Real datasets are very different from synthetic data used in previous studies . In order to utilize the F-Principle to understand and better use DNNs in real datasets , it is important to verify whether the F-Principle also holds in high-dimensional real datasets . In the following experiments , we examine the F-Principle in a training dataset of { ( xi , yi ) } n−1i=0 where n is the size of dataset . xi ∈ Rd is a vector representing the image and yi ∈ { 0 , 1 } 10 is the output ( a one-hot vector indicating the label for the dataset of image classification ) . d is the dimension of the input ( d = 784 for MNIST and d = 32× 32× 3 for CIFAR10 ) . Since the high dimensional discrete Fourier transform ( DFT ) requires prohibitively high computational cost , in this section , we only consider one direction in the Fourier space through a projection method for each examination . 3.1 EXAMINATION METHOD : PROJECTION . For a dataset { ( xi , yi ) } n−1i=0 we consider one entry of 10-d output , denoted by yi ∈ R. The high dimensional discrete non-uniform Fourier transform of { ( xi , yi ) } n−1i=0 is ŷk = 1n ∑n−1 i=0 yi exp ( −i2πk · xi ) . The number of all possible k grows exponentially on dimension d. For illustration , in each examination , we consider a direction of k in the Fourier space , i.e. , k = kp1 , p1 is a chosen and fixed unit vector , hence |k| = k. Then we have ŷk = 1n ∑n−1 i=0 yi exp ( −i2π ( p1 · xj ) k ) , which is essentially the 1-d Fourier transform of { ( xp1 , i , yi ) } n−1i=0 , where xp1 , i = p1 · xi is the projection of xi on the direction p1 ( Bracewell & Bracewell , 1986 ) . For each training dataset , p1 is chosen as the first principle component of the input space . To examine the convergence behavior of different frequency components during the training , we compute the relative difference between the DNN output and the target function for selected important frequencies k ’ s at each recording step , that is , ∆F ( k ) = |ĥk− ŷk|/|ŷk| , where ŷk and ĥk are 1-d Fourier transforms of { yi } n−1i=0 and the corresponding DNN output { hi } n−1i=0 , respectively , along p1 . Note that each response frequency component , ĥk , of DNN output evolves as the training goes . 3.2 MNIST/CIFAR10 . In the following , we show empirically that the F-Principle is exhibited in the selected direction during the training process of DNNs when applied to MNIST/CIFAR10 with cross-entropy loss . The network for MNIST is a fully-connected tanh DNN ( 784-400-200-10 ) and for CIFAR10 is two ReLU convolutional layers followed by a fully-connected DNN ( 800-400-400-400-10 ) . All experimental details of this paper can be found in Appendix C. We consider one of the 10-d outputs in each case using non-uniform Fourier transform . As shown in Fig . 1 ( a ) and 1 ( c ) , low frequencies dominate in both real datasets . During the training , the evolution of relative errors of certain selected frequencies ( marked by black squares in Fig . 1 ( a ) and 1 ( c ) ) is shown in Fig . 1 ( b ) and 1 ( d ) . One can easily observe that DNNs capture low frequencies first and gradually capture higher frequencies . Clearly , this behavior is consistent with the F-Principle . For other components of the output vector and other directions of p , similar phenomena are also observed . 4 F-PRINCIPLE IN MNIST/CIFAR10 THROUGH FILTERING METHOD . The projection method in the previous section enables us to visualize the F-Principle in one direction for each examination at the level of individual frequency components . However , demonstration by this method alone is insufficient because it is impossible to verify the F-Principle at all potentially informative directions for high-dimensional data . To compensate the projection method , in this section , we consider a coarse-grained filtering method which is able to unravel whether , in the radially averaged sense , low frequencies converge faster than high frequencies .

Summary: The paper studies the training process of NNs through the lens of Fourier analysis. The authors argue that during the training process, NNs will first learn low frequencies part of the function first and then the high frequency part. To verify this claim empirically, the author propose two methods: 1. examine the convergence of different frequencies in a pre-selected direction in the frequency space during training; 2. examine the convergence rate of the 2-norm of low v.s. high frequencies during training.  Through the experimental results of these two methods, the authors conclude that NNs learn the low  frequency components before the high frequency components. The authors also discuss a potential application of this observation to solving high dimensional PDEs: coupling DNNs training (good at learning low frequency components) with the Jacobi method (good at learning high frequency components). Finally, the authors also provide some theoretical intuition (Thm 1., 2.) why low frequency components are learned faster and an explanation why NNs could generalize well on images but perform poorly on tasks like learning parity functions.