Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

1 INTRODUCTION . Understanding the training process of Deep Neural Networks ( DNNs ) is a fundamental problem in the area of deep learning . We find a common behavior of the gradient-based training process of DNNs , that is , a Frequency Principle ( F-Principle ) : DNNs often fit target functions from low to high frequencies during the training process . In another word , at the early stage of training , the low-frequencies are fitted and as iteration steps of training increase , the high-frequencies are fitted . For example , when a DNN is trained to fit y = sin ( x ) + sin ( 2x ) , its output would be close to sin ( x ) at early stage and as training goes on , its output would be close to sin ( x ) + sin ( 2x ) . F-Principle was observed empirically in synthetic low-dimensional data with MSE loss during DNN training ( Xu et al. , 2018 ; Rahaman et al. , 2018 ) . However , in deep learning , empirical phenomena could vary from one network structure to another , from one dataset to another and could exhibit significant difference between synthetic data and highdimensional real data . Therefore , the universality of the F-Principle remains an important problem for further study . Especially for high-dimensional real problems , because the computational cost of high-dimensional Fourier transform is prohibitive in practice , it is of great challenge to demonstrate the F-Principle . On the other hand , the mechanism underlying the F-Principle and its implication to the application of DNNs , e.g. , design of DNN-based PDE solver , as well as their generalization ability are also important open problems to be addressed . In this work , we design two methods , i.e. , projection and filtering methods , to show that the FPrinciple exists in the training process of DNNs for high-dimensional benchmarks , i.e. , MNIST ( LeCun , 1998 ) , CIFAR10 ( Krizhevsky et al. , 2010 ) . The settings we have considered are i ) different DNN architectures , e.g. , fully-connected network , convolutional neural network ( CNN ) , and VGG16 ( Simonyan & Zisserman , 2014 ) ; ii ) different activation functions , e.g. , tanh and rectified linear unit ( ReLU ) ; iii ) different loss functions , e.g. , cross entropy , mean squared error ( MSE ) , and loss energy functional in variational problems . These results demonstrate the universality of the F-Principle . To facilitate the designs and applications of DNN-based schemes , we characterize a stark difference between DNNs and conventional numerical schemes on various scientific computing problems , where most of the conventional methods ( e.g. , Jacobi method ) exhibit the opposite convergence behavior — faster convergence for higher frequencies . This difference implies that DNN can be adopted to accelerate the convergence of low frequencies for computational problems . We also intuitively explain with theories under an idealized setting how the smoothness/regularity of commonly used activation functions contributes to the F-Principle . Note that this mechanism is rigorously demonstrated for DNNs of general settings in a subsequent work ( Luo et al. , 2019 ) . Finally , we discuss that the F-Principle provides an understanding of good generalization of DNNs in many real datasets ( Zhang et al. , 2016 ) and poor generalization in learning the parity function ( Shalev-Shwartz et al. , 2017 ; Nye & Saxe , 2018 ) , that is , the F-Principle which implies that DNNs prefer low frequencies , is consistent with the property of low frequencies dominance in many real datasets , e.g. , MNIST/CIFAR10 , but is different from the parity function whose spectrum concentrates on high frequencies . Compared with previous studies , our main contributions are as follows : 1 . By designing both the projection and filtering methods , we consistently demonstrate the F-Principle for MNIST/CIFAR10 over various architectures such as VGG16 and various loss functions . 2 . For the application of solving differential equations , we show that ( i ) conventional numerical schemes learn higher frequencies faster whereas DNNs learn lower frequencies faster by the FPrinciple , ( ii ) convergence of low frequencies can be greatly accelerated with DNN-based schemes . 3 . We present theories under an idealized setting to illustrate how smoothness/regularity of activation function contributes to the F-Principle . 4 . We discuss in detail the implication of the F-Principle to the generalization of DNNs that DNNs are implicitly biased towards a low frequency function and provide an explanation of good and poor generalization of DNNs for low and high frequency dominant target functions , respectively . 2 FREQUENCY PRINCIPLE . The concept of “ frequency ” is central to the understanding of F-Principle . In this paper , the “ frequency ” means response frequency NOT image ( or input ) frequency as explained in the following . Image ( or input ) frequency ( NOT used in the paper ) : Frequency of 2-d function I : R2 → R representing the intensity of an image over pixels at different locations . This frequency corresponds to the rate of change of intensity across neighbouring pixels . For example , an image of constant intensity possesses only the zero frequency , i.e. , the lowest frequency , while a sharp edge contributes to high frequencies of the image . Response frequency ( used in the paper ) : Frequency of a general Input-Output mapping f . For example , consider a simplified classification problem of partial MNIST data using only the data with label 0 and 1 , f ( x1 , x2 , · · · , x784 ) : R784 → { 0 , 1 } mapping 784-d space of pixel values to 1-d space , where xj is the intensity of the j-th pixel . Denote the mapping ’ s Fourier transform as f̂ ( k1 , k2 , · · · , k784 ) . The frequency in the coordinate kj measures the rate of change of f ( x1 , x2 , · · · , x784 ) with respect to xj , i.e. , the intensity of the j-th pixel . If f possesses significant high frequencies for large kj , then a small change of xj in the image might induce a large change of the output ( e.g. , adversarial example ) . For a dataset with multiple classes , we can similarly define frequency for each output dimension . For real data , the response frequency is rigorously defined via the standard nonuniform discrete Fourier transform ( NUDFT ) , see Appendix A . Frequency Principle : DNNs often fit target functions from low to high ( response ) frequencies during the training process . An illustration of F-Principle using a function of 1-d input is in Appendix B . The F-Principle is rigorously defined through the frequency defined by the Fourier transform ( Appendix A , Bracewell & Bracewell ( 1986 ) ) and the converging speed defined by the relative error . By using high-dimensional real datasets , we then experimentally demonstrate F-Principle at the levels of both individual frequencies ( projection method ) and coarse-grained frequencies ( filtering method ) . 3 F-PRINCIPLE IN MNIST/CIFAR10 THROUGH PROJECTION METHOD . Real datasets are very different from synthetic data used in previous studies . In order to utilize the F-Principle to understand and better use DNNs in real datasets , it is important to verify whether the F-Principle also holds in high-dimensional real datasets . In the following experiments , we examine the F-Principle in a training dataset of { ( xi , yi ) } n−1i=0 where n is the size of dataset . xi ∈ Rd is a vector representing the image and yi ∈ { 0 , 1 } 10 is the output ( a one-hot vector indicating the label for the dataset of image classification ) . d is the dimension of the input ( d = 784 for MNIST and d = 32× 32× 3 for CIFAR10 ) . Since the high dimensional discrete Fourier transform ( DFT ) requires prohibitively high computational cost , in this section , we only consider one direction in the Fourier space through a projection method for each examination . 3.1 EXAMINATION METHOD : PROJECTION . For a dataset { ( xi , yi ) } n−1i=0 we consider one entry of 10-d output , denoted by yi ∈ R. The high dimensional discrete non-uniform Fourier transform of { ( xi , yi ) } n−1i=0 is ŷk = 1n ∑n−1 i=0 yi exp ( −i2πk · xi ) . The number of all possible k grows exponentially on dimension d. For illustration , in each examination , we consider a direction of k in the Fourier space , i.e. , k = kp1 , p1 is a chosen and fixed unit vector , hence |k| = k. Then we have ŷk = 1n ∑n−1 i=0 yi exp ( −i2π ( p1 · xj ) k ) , which is essentially the 1-d Fourier transform of { ( xp1 , i , yi ) } n−1i=0 , where xp1 , i = p1 · xi is the projection of xi on the direction p1 ( Bracewell & Bracewell , 1986 ) . For each training dataset , p1 is chosen as the first principle component of the input space . To examine the convergence behavior of different frequency components during the training , we compute the relative difference between the DNN output and the target function for selected important frequencies k ’ s at each recording step , that is , ∆F ( k ) = |ĥk− ŷk|/|ŷk| , where ŷk and ĥk are 1-d Fourier transforms of { yi } n−1i=0 and the corresponding DNN output { hi } n−1i=0 , respectively , along p1 . Note that each response frequency component , ĥk , of DNN output evolves as the training goes . 3.2 MNIST/CIFAR10 . In the following , we show empirically that the F-Principle is exhibited in the selected direction during the training process of DNNs when applied to MNIST/CIFAR10 with cross-entropy loss . The network for MNIST is a fully-connected tanh DNN ( 784-400-200-10 ) and for CIFAR10 is two ReLU convolutional layers followed by a fully-connected DNN ( 800-400-400-400-10 ) . All experimental details of this paper can be found in Appendix C. We consider one of the 10-d outputs in each case using non-uniform Fourier transform . As shown in Fig . 1 ( a ) and 1 ( c ) , low frequencies dominate in both real datasets . During the training , the evolution of relative errors of certain selected frequencies ( marked by black squares in Fig . 1 ( a ) and 1 ( c ) ) is shown in Fig . 1 ( b ) and 1 ( d ) . One can easily observe that DNNs capture low frequencies first and gradually capture higher frequencies . Clearly , this behavior is consistent with the F-Principle . For other components of the output vector and other directions of p , similar phenomena are also observed . 4 F-PRINCIPLE IN MNIST/CIFAR10 THROUGH FILTERING METHOD . The projection method in the previous section enables us to visualize the F-Principle in one direction for each examination at the level of individual frequency components . However , demonstration by this method alone is insufficient because it is impossible to verify the F-Principle at all potentially informative directions for high-dimensional data . To compensate the projection method , in this section , we consider a coarse-grained filtering method which is able to unravel whether , in the radially averaged sense , low frequencies converge faster than high frequencies .

Summary: This paper proposes to analyze the loss of neural networks in the Fourier domain. Since this is computationally expensive for larger-dimensional datasets, the analysis instead first projects the data onto the principal component of the data, and then using a Gaussian kernel estimation (which has nice properties in the Fourier domain). The analysis finds that DNNs tend to learn low-frequency components before high-frequency ones.