Spectral Bias in Practice: the Role of Function Frequency in Generalization

1 INTRODUCTION . Two fundamental questions in machine learning are why overparameterized models generalize and how to make them more robust to distribution shift . Resolving both of these questions requires understanding how complex our models should be . For instance , it is thought that overparameterized models generalize well because there are implicit regularizers that constrain the complexity of the learned functions . However , the precise nature of these implicit regularizers , and their importance in practice , remains unclear . As for how to achieve robustness , no consensus has emerged regarding function complexity . On the one hand , Cranko et al . ( 2018 ) and Leino et al . ( 2021 ) argue that Lipschitz smoothness of the learned function offers a guarantee of robustness ; on the other , Shah et al . ( 2020 ) and Madry et al . ( 2019 ) argue that simplicity can be harmful and a robust model must actually be more complex than its non-robust counterpart . One window into function complexity is spectral bias – the tendency of neural networks to learn low frequency ( simple and smooth ) functions early in training , gradually increasing the frequency ( complexity ) of the learned function as training proceeds . Foundational work in this area has shown theoretical evidence of spectral bias by analyzing convergence rates of neural networks towards functions of different frequencies and showing that they converge to low frequency functions at a faster rate ( Basri et al. , 2019 ; Rahaman et al. , 2019 ) . In practice , spectral bias is difficult to measure : the most direct method involves taking a Fourier transform with respect to the input , which is expensive to compute due to the high dimensionality of images . Early experimental work focused on low-dimensional synthetic data ( Basri et al. , 2019 ; 2020 ) or used proxy measurements of spectral bias by inserting label noise of various frequencies during training ( Rahaman et al. , 2019 ) . However , the label noise method is limited to binary classification/regression and involves modifying the training data , making it difficult to disentangle how other changes to the training data ( such as data augmentation ) affect the spectral content of the learned function . Thus , it remains an open question as to whether modern neural networks exhibit spectral bias and what role it plays in generalization . In this work , we investigate model complexity through the lens of spectral bias , by introducing experimental methods to study the frequency decomposition of the functions learned by modern image classification networks . We find that neither simplicity nor complexity is purely beneficial or detrimental to performance . Contributions . We extend the label noise procedure of Rahaman et al . ( 2019 ) to enable measuring spectral bias in multi-class classification and apply this technique to understand the function frequencies present in high-accuracy models on CIFAR-10 ( Krizhevsky et al. ) . We also introduce a second method for measuring the smoothness of a learned function via linear interpolation between test examples . This offers a proxy measurement of spectral bias without the need to modify training labels , allowing us to probe the effects of the training data on the learned function frequencies . Using these experimental methods , we find that increasing the model size decreases spectral bias ; larger models learn high frequencies faster . However , we observe that more accurate models are not always higher frequency . Several forms of explicit and implicit regularization , including weight decay , increasing the dataset size , and applying Mixup ( Zhang et al. , 2018 ) data augmentation , increase spectral bias and produce a lower-frequency learned function . Our experiments show that these common training interventions affect the frequency content of the learned function in different ways , helping us to understand their mechanism and offering a tool to aid design of improved training methods in the future . This perspective allows us to shed light on the mechanism behind self-distillation , in which a student model is trained to fit the predictions of a teacher model . Using our linear interpolation methodology , we observe that self-distillation produces a student model whose learned function is smoother than that of its teacher . This suggests that the teacher model acts as a sort of low-pass filter on the target function , perhaps making it more accessible to the spectrally-biased student . Finally , we explore the relationship between image frequency and function frequency . By further extending the label noise methodology of Rahaman et al . ( 2019 ) to study spectral bias in directions of interest through the input space , we find that models are most sensitive to the low image frequencies common in natural images . 2 RELATED WORK . Implicit bias . A common belief is that some form of implicit bias imposed by the training procedure may account for the generalization ability of overparameterized neural networks , and accordingly , much research has been directed towards understanding these implicit biases ( Soudry et al. , 2018 ; Zhang et al. , 2017 ; Keskar et al. , 2017 ; Gunasekar et al. , 2019 ; 2020 ; Hardt et al. , 2016 ; Hoffer et al. , 2018 ; Belkin et al. , 2018 ; Neyshabur et al. , 2018 ; 2015 ; 2017 ; Mania et al. , 2019 ; Gunasekar et al. , 2019 ; Nakkiran et al. , 2019 ) . One of the most well-known forms of implicit bias comes from the optimization algorithm itself , which typically guides networks to solutions that generalize well ( Soudry et al. , 2018 ; Keskar et al. , 2017 ; Wilson et al. , 2017 ) . Spectral bias . Spectral bias is a form of implicit bias , with much recent attention including theoretical and experimental approaches ( Oymak et al. , 2019 ; Xu et al. , 2019 ; Rahaman et al. , 2019 ; Basri et al. , 2019 ; Zhang et al. , 2021 ) ; we describe some of these prior results here . Basri et al . ( 2019 ) studied spectral bias using a linear model of SGD training dynamics to show that models learn low frequency ( simple ) functions early in training and then gradually learn higher frequencies as training proceeds , assuming training data that is distributed uniformly on the hypersphere . Basri et al . ( 2020 ) extended the analysis to consider nonuniformly spaced training data , and found that learning is faster where samples are denser ; if sampling is nonuniform , then during training , the learned function will be higher frequency in regions with denser samples . Rahaman et al . ( 2019 ) used a more direct analysis to show the same spectral bias toward low frequency functions and also suggested that when training data lie on a manifold ( as natural images are believed to ) , increasing the complexity of the manifold makes it easier for models to fit high frequency functions . Additionally , Rahaman et al . ( 2019 ) posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters , connecting back to the idea of flat optimization minima ( Neyshabur et al. , 2017 ) or smooth loss landscape Mehmeti-Göpel et al . ( 2021 ) . They proposed experimental methods to study spectral bias in image classification , but focused on a binary subset of the relatively simple MNIST dataset ( Deng , 2012 ) , with mean square error loss . Our label smoothing experiments are a direct extension of Rahaman et al . ( 2019 ) to multiclass classification with the more common cross-entropy loss . Our linear interpolation experiments are somewhat similar to the recently-proposed experimental methods in Zhang et al . ( 2021 ) , except that we sample along paths between images rather than in regions surrounding each image ; this allows for a more global measurement of spectral bias . Model sensitivity to image frequency . While we focus on function frequency in this work , prior research aimed to understand model sensitivity to image frequencies . Jo & Bengio ( 2017 ) found that CNNs are sensitive to Fourier statistics of the training data , even those irrelevant to human viewers . Ortiz-Jimenez et al . ( 2020b ) studied the image frequency bias induced by using a convolutional architecture , and Ortiz-Jimenez et al . ( 2020a ) argued that models are sensitive primarily to discriminative Fourier directions in the training data . They posit that adversarial training induces large margins in orthogonal directions , thereby improving robustness . Yin et al . ( 2020 ) introduced a procedure to measure the sensitivity of trained models to Fourier image perturbations , and showed that adversarial training and Gaussian data augmentation increase robustness to high frequency perturbations but increase sensitivity to low frequency perturbations . Our work adds to this line of research a study of the interplay between two notions of frequency : the function frequency involved in spectral bias , and the image frequencies present in the data . 3 METHODOLOGY . The goal of our work is to measure the complexity of neural network functions through the lens of frequency . In low dimensions , measuring the frequency decomposition of a function is straightforward and tractable : evaluate the function at dense , uniform sampling positions and compute its discrete Fourier transform . However , the function of interest in image classification is unavoidably highdimensional , mapping images ( with thousands of pixel values ) to object classes . It would be intractable even to collect sufficient samples of this function to compute a discrete Fourier transform , let alone compute the transform itself . Instead , we employ two complementary approaches to measure informative proxies of this frequency decomposition . 3.1 LABEL SMOOTHING . The core idea for measuring function frequency introduced by Rahaman et al . ( 2019 ) is to construct a sinusoid over the space of images , and to use that sinusoid as a form of label noise during training . Let D = { ( Xi , yi ) } ntraini=1 be the training examples and Dvalid = { ( Xj , yj ) } nval j=1 be the validation examples , with Xi an image and yi a one-hot class encoding , where ntrain is the number of training examples , nval is the number of validation examples , d is the side length of an image ( assumed to be square for simplicity ) , c is the number of color channels , and M is the number of classes , so Xi ∈ Rd×d×c and yi ∈ RM . To extend this procedure to the multi-class setting , we add noise of various frequencies to an M - dimensional label vector via label smoothing ( Szegedy et al. , 2015 ) . Let S : Rd×d×c → [ 0 , 1 ] be a target function that maps an input image Xi to a scalar value between 0 and 1 . We apply label smoothing to each label yi , mapping it to ȳi = yi ( 1−S ( Xi ) ) + 1M S ( Xi ) to retain a valid probability distribution . We then train from scratch using the original examples Xi and their smoothed labels ȳi . Finally , we evaluate on the validation images Xj , comparing to both their original one-hot labels yj and smoothed labels ȳj = yj ( 1− S ( Xj ) ) + 1M S ( Xj ) . Our experiments use the CIFAR-10 dataset ( Krizhevsky et al . ) , where ntrain = 50000 , nval = 10000 , d = 32 , c = 3 , and M = 10 ; however , our experimental setup is sufficiently general as to transfer easily to any image classification task . This method is introduced in Figure 1 . As training proceeds , we see both the training loss ( between predictions and smoothed labels ȳi ) and noisy validation loss ( between predictions and smoothed labels ȳj ) decrease . The clean validation loss ( between predictions and one-hot labels yj ) , however , initially decreases but at some point in training plateaus or begins to increase . At this point , the model has begun to learn the target function S. Accordingly , we introduce noise fitting , defined as the difference between the validation loss on one-hot labels and the validation loss on labels smoothed with the same function S that was applied to the training labels : noise fitting = clean validation loss− noisy validation loss ( 1 ) We compare different models by computing the minimum noise fitting achieved throughout training . A model with higher min noise fitting more readily fits the target function S than a model with lower min noise fitting . By choosing different functions S , we can probe nuances of spectral bias . Typically ( inspired by Rahaman et al . ( 2019 ) ) we choose a radial wave : S ( X ) = sin ( 2πf ( ‖X‖ − ED‖X‖ ) ) , where we can vary the frequency f to understand spectral bias at this global scale . We can also choose more targeted functions S ; for instance , if V is a direction of interest through the space of images ( i.e . V is an image-shaped vector of unit norm ) , we can construct S ( X ) = sin ( 2πf 〈X , V〉 ) . This allows us to vary both the frequency and direction of the target sinusoid , to understand model sensitivity along different directions through image space . However , label smoothing has a few limitations . It requires retraining each model from scratch , with substantial investment of time and computational resources . Because it involves perturbing the training labels , it can not directly answer questions about the interaction between the training dataset and spectral bias , such as the effects of data augmentation . For the same reason , we can not use this methodology to study the spectral decomposition of existing trained models .

Summary: This paper proposes two methods for estimating smoothness of the prediction function learned by a deep network :   1/ an extension of Rahaman et al. to multiclass classification, by adding a sine of desired frequency to the target vector, then training on the new target vectors. The "effective noise fitting" is then the difference between the validation loss on clean targets and the validation loss on modified targets.  2/ a measure of change of the logits in l2 norm between 2 training examples, averaged over many examples  Using their methods, they show the effect on the smoothness of the predicted function of:  - varying the number of parameters  - adding explicit regularization (weight decay/mixup)  - distillation