Spectral Bias in Practice: the Role of Function Frequency in Generalization
Authors: Sara Fridovich-Keil
Raphael Gontijo-Lopes
Rebecca Roelofs
Venue: ICLR 2022 Submitted (2022)
Keywords: spectral bias
generalization
function frequency
image classification

Abstract: Despite their ability to represent highly expressive functions, deep learning models trained with SGD seem to find simple, constrained solutions that generalize surprisingly well. Spectral bias – the tendency of neural networks to prioritize learning low frequency functions – is one possible explanation for this phenomenon,but so far spectral bias has only been observed in theoretical models and simplified experiments. In this work, we propose methodologies for measuring spectral bias in modern image classification networks.  We find that these networks indeed exhibit spectral bias, and that networks that generalize well strike a balance between having enough complexity (i.e. high frequencies) to fit the data while being simple enough to avoid overfitting. For example, we experimentally show that larger models learn high frequencies faster than smaller ones, but many forms of regularization, both explicit and implicit, amplify spectral bias and delay the learning of high frequencies. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive to the low frequencies prevalent in natural images. Our work enables measuring and ultimately controlling the spectral behavior of neural networks used for image classification, and is a step towards understanding why deep models generalize well.

Under review as a conference paper at ICLR 2022
SPECTRAL BIAS IN PRACTICE: THE ROLE OF FUNCTION FREQUENCY IN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Despite their ability to represent highly expressive functions, deep learning models trained with SGD seem find simple solutions that generalize surprisingly well. Spectral bias - the tendency of neural networks to prioritize learning low frequency functions - is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simplified experiments. this work, we propose methodologies for measuring spectral bias modern image classification networks. We find that these networks indeed exhibit spectral bias, and that interventions that improve generalization sometimes increase and sometimes decrease the frequencies of the learned function. For example, we experimentally show that larger models learn high frequencies more readily than smaller ones, but many forms of regularization, both explicit and implicit, inhibit the learning of high frequencies. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive to the low frequencies prevalent in natural images. Our work enables measuring and ultimately influencing the spectral behavior of neural networks used for image classification, and is a step towards understanding why deep models generalize well.
INTRODUCTION
Two fundamental questions in machine learning are why overparameterized models generalize and how to make them more robust to distribution shift. Resolving both of these questions requires understanding how complex our models should be.
For instance, it is thought that overparameterized models generalize well because there are implicit regularizers that constrain the complexity of the learned functions. However, the precise nature of these implicit regularizers, and their importance in practice, remains unclear. As for how to achieve robustness, no consensus has emerged regarding function complexity. On the one hand, Cranko al. (2018) and Leino et al. (2021) argue that Lipschitz smoothness of the learned function offers guarantee of robustness; on the other, (Shah et al. (2020) and Madry et al. (2019) argue that simplicity be harmful and robust model must actually be more complex than its non-robust counterpart.
One window into function complexity is spectral bias - the tendency of neural networks to learn low frequency (simple and smooth) functions early in training, gradually increasing the frequency (complexity) of the learned function as training proceeds. Foundational work in this area has shown theoretical evidence of spectral bias by analyzing convergence rates of neural networks towards functions of different frequencies and showing that they converge to low frequency functions at faster rate (Basri et al. 2019 Rahaman et al. 2019).
In practice, spectral bias is difficult to measure: the most direct method involves taking a Fourier transform with respect to the input, which expensive to compute due to the high dimensionality of images. Early experimental work focused on oow-dimensional synthetic data (Basri et al.] 2019) 2020) or used proxy measurements of spectral bias by inserting label noise of various frequencies during training (Rahaman et al. 2019). However, the label noise method is limited to binary classifica- tion/regression and involves modifying the training data, making it difficult to disentangle how other changes to the training data (such as data augmentation) affect the spectral content of the learned function. Thus, it remains an open question as to whether modern neural networks exhibit spectral bias and what role it plays in generalization.

---

Under review as a conference paper at ICLR 2022
In this work, we investigate model complexity through the lens of spectral bias, by introducing experimental methods to study the frrquency decomposition of the functions learned by modern image classification networks. We find that neither simplicity nor complexity is purely beneficial or detrimental to performance.
Contributions. We extend the label noise procedure of Rahaman et al. (2019) to enable measur- ing spectral bias in multi-class classification and apply this technique to understand the function frequencies present in high-accuracy models CIFAR-10 (Krizhevsky et al.). We also introduce second method for measuring the smoothness of a learned function via linear interpolation between test examples. This offers a proxy measurement of spectral bias without the need modify training labels, allowing us to probe the effects of the training data on the learned function frequencies.
Using these experimental methods, find that increasing the model size decreases spectral bias; larger models learn high frequencies faster. However, we observe that more accurate models are not always higher frequency. Several forms of explicit and implicit regularization, including weight decay, increasing the dataset size, and applying Mixup (Zhang et al., 2018) data augmentation, increase spectral bias and produce lower-frequency learned function. Our experiments show that these common training interventions affect the frequency content of the learned function in different ways, helping us understand their mechanism and offering a tool to aid design improved training methods in the future.
This perspective allows us to shed light on the mechanism behind aelf-distillation,, which a student model is trained to fit the predictions of teacher model. Using linear interpolation methodology, we observe that self-distillation produces a student model whose learned function is smoother than that of its teacher. This suggests that the teacher model acts sort of low-pass filter on the target function, perhaps making it more accessible to the spectrally-biant biased student.
Finally, we explore the relationship between image frequency and function frequency. By further extending the label noise methodology of Rahaman et al. (219)) to study spectral bias directions of interest through the input space, we find that models are most sensitive to the low image frequencies common in natural images.
RELATED WORK
Implicit bias. A common belief is that some form of implicit bias imposed by the training procedure
may account for the generalization ability overparameterized neural networks, and accordingly.
much research has been directed towards understanding these implicit biases (Soudry et al.) 2018
Zhang et al. 20117] Keskar et al., 2017 Gunasekar et al. 2019 19 2020 Hardt et al. [016 Hoffer
ct al. 2018 Belkin et al.) 2018 8
Neyshabur et al.) 2018 2015 2011 7 Mania et al. [019] Gunasekar
et al. 2019 Nakkiran et al. 2019). One of the most well- knolw forms of implicit bias comes from the optimization algorithm itself. which typically guides networks to solutions that generalize well
(Soudry et al. 2018 Keskar et al. 2017 Wilson et al. 2017).
Spectral bias. Spectral bias is form of implicit bias, with much recent attention including theoretical and experimental approaches (Oymak et al., 2019) Xu et al.) 2219 Rahaman et al.) 2019 Basri et al., 2019: Zhang et al., 2021): we describe some of these prior results here. [asri al. (2019) studied spectral bias using a linear model of SGD training dynamics to show that models learn low frequency (simple) functions early in training and then gradually learn higher frequencies training proceeds, assuming training data that is distributed uniformly on the hypersphere. Basri et al. (2020) extended the analysis to consider nonuniformly spaced training data, and found that learning faster where samples are denser; if sampling is nonuniform, then during training, the learned function will be higher frequency in regions with denser samples.
Rahaman et al. (219) used a more direct analysis to show the same spectral bias toward low frequency functions and also suggested that when training data lie on a manifold (as natural images are believed to), increasing the complexity of the manifold makes it easier for models to fit high frequency functions. Additionally, |aahaman et al. (2019) posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters, connecting back to the idea of flat optimization minima (Neyshabur et al. 2017) or smooth loss landscape Mehmeti Göpel et al. (202T). ) They proposed experimental methods to study spectral bias in image classification, but
2

---

Under review as a conference paper at ICLR 2022
focused on a binary subset of the relatively simple MNIST dataset (Deng 2012). with mean square error loss.
direct extension of Rahaman et al. 20 1019) to multiclass classification with the more common cross-entropy loss. Our linear interpolation experiments are somewhat similar to the recently-pon proppssed experimental methods in Zhang al. (2021), except that we sample along paths between images rather than in regions surrounding each image; this allows for a more global measurement of spectral bias.
Our label smoothing experiments
Model sensitivity to image frequency. While we focus on function frequency in this work, prior research aimed understand model sensitivity image frequencies. [To & Bengio (2017) found that CNNs are sensitive to Fourier statistics of the training data, even those irrelevant to human viewers. Ortiz-Jimenez et (2020b) studied the image frequency bias induced by using a convolutional archi- tecture, and Birtiz-Jimeezz et al. (220a)) argued that models are sensitive primarily to discriminative Fourier directions the training data. They posit that adversarial training induces large margins in orthogonal directions, thereby improving robustness. Yin et al. (2020) introduced a procedure to measure the sensitivity trained models to Fourier image perturbations, and showed that adversarial training and Gaussian data augmentation increase robustness to high frequency perturbations but increase sensitivity to low frequency perturbations. Our work adds to this line of research a study of the interplay between two notions of frequency: the function frequency involved in spectral bias, and the image frequencies present in the data.
METHODOLOGY
The goal of work is to measure the complexity of neural network functions through the lens of frequency. In low dimensions, measuring the frequency decomposition of a function is straightforward and tractable: evaluate the function dense, uniform sampling positions and compute its discrete Fourier transform. However, the function of interest in image classification is unavoidably high- dimensional, mapping images (with thousands of pixel values) to object classes. It would be intractable even to collect sufficient samples of this function to compute a discrete Fourier transform, let alone compute the transform itself. Instead, we employ two complementary approaches measure informative proxies of this frequency decomposition.
3.1 LABEL SMOOTHING
The core idea for measuring function frequency introduced by Rahaman al. (2019) to construct a sinusoid over the space of images, and to use that sinusoid as form of label noise during training.
Let D = {(,, y/:) J=n Risotn be the training examples and Dvalid = {(X,, ; y )=n Road be the validation examples, with X; an image and y; a one-hot class encoding, where Merain is the number of training examples, neat is the number of validation examples, d is the side length of image (assumed to be square for simplicity), c is the number of color channels, and M is the number of classes, so X; dxxxxx and y; E IR
To extend this procedure to the multi-class setting, we add noise of various frequencies to an M- dimensional label vector via label smoothing (Szegedy et al. 2015 Let S Rdxdxe [0. 1] be a target function that maps an input image X; to scalar value between O and 1. We apply label smoothing to each label yıs mapping it to y: = (y:11_((X S(X))) S(,)) retain a valid probability distribution. We then train from scratch using the original examples X; and their smoothed labels ū.- Finally, we evaluate on the validation images X,, comparing to both their original one-hot labels yj and smoothed labels Ø, y (1- s(X,) )) + M S(X,). Our experiments use the CIFAR-10 10 dataset (Krizhevsky et al.). where Merain = 50000, Neaa = 10000, d = 32, c 3, and M = 10; however, our experimental setup is sufficiently general as to transfer easily to any image classification task.
This method is introduced in Figure As training proceeds, we see both the training loss (between predictions and smoothed labels y.) and noisy validation loss (between predictions and smoothed labels y decrease. The clean validation loss (between predictions and one-hot labels yj), however, initially decreases but at some point in training plateaus or begins to increase. At this point, the model has begun to leam the target function S. Accordingly, difference between the validation loss on one-hot labels and the validation loss on labels smoothed with the same function S that was applied the training labels:
introduce noise fitting, defined as the
3

---

Under review as a conference paper at ICLR 2022
Noise Fitting
-3.5
Training Loss
Noisy Validation Loss
Validation Loss
-0.6
2.0
1.5
-2.7
1.0
-0.8
50
100
150
200
230
Epoch
Epoch
Epoch
Epoch
Figure 1: Noise fitting shows when and how much a network fits a target frequency function. At roughly epoch 150, noise fitting (Right), the difference between clean and noisy validation loss, exhibits a clear "dip" when the model begins to fit the target function: training (Left) and validation (Center Left) loss on the perturbed function drop, while improvement stalls on clean validation data (Center Right). By comparing the minimum value of noise fitting achieved by different models throughout training (illustrated by the gray line the right plot), we can compare the relative degree to which different models fit noise functions of varying frequency. A model with higher min noise fitting more readily fits the target noise function. Here, we train a wrn 32 model (wide-resnct with width 32) with radial wave label smoothing at frequency 0.04. For visual clarity, apply exponential averaging to all curves.
noise fitting = clean validation loss noisy validation loss
(1)
We compare different models by computing the minimum noise fitting achieved throughout training. A model with higher min noise fitting more readily fits the target function S than model with lower min noise fitting.
By choosing different functions S, we can probe nuances of spectral bias. Typically (inspired by Rahaman et al. (2019) we choose a radial wave: S(X) sin (2 f[] - Ep (I])), where we can vary the frequency to understand spectral bias at this global scale. We can also choose more targeted functions S: for instance, if V is direction of interest through the space of images (i.e. V is an amage-shaper vector of unit norm), we can construct S(X) sin(2x f (X, V)). This allows us to vary both the frequency and direction of the target sinusoid, to understand model sensitivity along different directions through image space.
However, label smoothing has few limitations. It requires retraining each model from scratch, with substantial investment of time and computational resources. Because it involves perturbing the training labels, it cannot directly answer questions about the interaction between the training dataset and spectral bias, such as the effects data augmentation. For the same reason, we cannot use this methodology to study the spectral decomposition of existing trained models.
3.2 LINEAR INTERPOLATION
To complement the label smoothing approach, we also propose measurement methodology based on linear interpolation between validation images. Although we cannot take dense. regularly-space samples that cover the entire input (image) space. we can sample along specifie paths through, and thereby glean glimpses into the spectral content of the learned function.
We consider two types of paths: those between images from the same class, and those between images from different classes. When considering paths between images in the same class, we choose 200 random, distinct pairs of images from each of the 10 CIFAR- 10 classes and average over the paths defined by these pairs. When interpolating between classes, we choose 200 random, distinct image pairs from each the 45 (10 choose 2) pairs of distinct classes, and average over the resulting paths.
For each pair of images (X,, X1), we vary X E [0. 1] to trace out a path, where each image in the path is given by X = AX + (1 Xy We choose As so that the distance between adjacent images on the path is constant, which means that each path may produce different number of samples depending on the total distance between X, and X, Before interpolating, we normalize Xn and X1- Figure 2 Bottom Left) shows illustrative example of two images and a corresponding interpolation path between them. For our experiments, we typically have 50 to 100 images along the interpolation path.
4

---

Under review as a conference paper at ICLR 2022
High Frequency Fraction of the Function
Smoothing Frequency
0.4
0.04
11/ -yoll
0.01
0.03
0.02
0.2
0.03
DFT
0.04
0.05
0.01
0.0
0.2
0.4
0.6
0.8
0.00
0.01
0.03
0.04
Label Smoothing Frequency
À 0.0
A 0.2
A 0.4
A 0.5
A 0.8
Figure 2: Linear interpolation measurements on functions of varying frequency. Left: Interpo- lation experiment between images of the same class on an oracle function: the true one-hot label perturbed by radial wave label smoothing of variable frequency. As the oracle function increases in frequency. the interpolating paths become less smooth. Right: Summary of this interpolation experiment via a discrete Fourier transform; as the frequency the oracle function increases, so does the proportion of the DFT magnitude allocated to the high frequency components.
Relationship between interpolation and label smoothing experiments. In order to validate our interpolation methodology, we consider the oracle function that maps directly from the interpolated image X, to the target label used in our label smoothing experiments. By increasing the frequency of the label smoothing noise, we create oracle functions with higher frequencies. We then verify that our interpolation measurement indeed measures higher frequencies on the higher frequency oracle functions.
In Figure Leftt) we show an example where we vary the label smoothing frequency from f to f 0.05. We plot the value of À along the interpolating path the axis, and the norm of the difference between the model output on the interpolated image ýi and the model output on the original image 9n on the y axis. The norm of the prediction difference allows to visualize how smooth the function is along the interpolating path; smoother prediction difference norm indicates a lower frequency function.
In order to quantify the frequency of the function along the interpolating path, we compute the per-class discrete Fourier transform (DFT). Letting (Xx, 1f" denote the of interpolated images, the DFT coefficient for frequency f is given by Y, [m] = Σt= UA, [m] exp(-i2n ft), where the imaginary number and m { Ad) denotes the class. In order to determine how the magnitudes of the DFT coefficients Y, are distributed between low and high frequencies, we quantify how much of the spectrum is high frequency by choosing a frequency threshold and computing the fraction of the total coefficient magnitudes above the threshold. In Figure (Right) we show this summary metric averaged over all within-class interpolating paths for the oracle functions.
Both sides of Figure 2 show that our interpolation methodology indeed measures the oracle functions that we constructed to be high frequency as high frequency. In Figure ( (Left). the higher frequency oracle functions have less smooth prediction norm differences, and in Figure (Right), they have higher fraction of DFT coefficient magnitudes in the high frequencies.
3.3 DATA AND MODELS
Our experiments use the CIFAR-10 (Krizhevsky et al.) dataset of low-resolution natural images from ten classes, a mixture of animals and objects. Images are pixel-wise normalized by the mean and standard deviation of the training images. We consider six different convolutional neural networks that have achieved high accuracy on this task: two wide-resmets (Zagoruyko & Komodakis 2017) of different sizes (wrn_32 and wrn_160), three shake-shake regularized networks (Gastaldi 2017 7 of different sizes (shake shake 32, shake shake 96, and shake shake 112), and pyramid net, which uses the even stronger shake-drop regularization (Yamada et al. 2019). Our implementations based
5

---

Under review as a conference paper at ICLR 2022
on Cubuk et al. (2019a); use the same optimizer (stochastic gradient descent with momentum) and cosine learning rate schedule. We train without data augmentation (to ensure all models are trained on exactly the same examples), except for experiments that explicitly vary data augmentation. Without data augmentation, the test accuracies of our models are: 89.4% (wrn 32), 90.2% (wrn 160), 92.2% and 95.8% (pyramid_net).
(shake maakesshakk_322 93.5% shaaee 93.6%
4 RESULTS
Throughout this section we show representative examples from our experiments to highlight our main findings. Full results (on all six models we tested) are included in section A
4.1 SPECTRAL BIAS AND MODEL ARCHITECTURE
We begin by applying our label smoothing methodology to find that modern image classification CNNs exhibit spectral bias, learning low frequency target functions early in training and learning higher frequency functions as training proceeds. Note that Figure 3 (Left) shows spectral bias over a small but illustrative range of frequencies (the full range frequencies we tested was 0 0. 0.1); target functions of sufficiently low frequency are fit almost immediately and target functions of sufficiently high frequency are never learned during the 250 epochs of training we tested.
Min Naise Fitting
-0.60
Min Noise Fitting
Min Noise Fitting
-0.65
-0.70
-0.88
-0.70
-0.70
-0.75
-0.72
-0.75
cas7 6.0.9 nnen
212
Label Smoothing Frequency
Wide-ResNet Width
Shake-Shake Width
(a) Varying frequency
(b) Varying model width
Figure 3: Modern CNNs exhibit spectral bias toward low frequencies, but larger networks learn high frequencies more readily. Higher min noise fitting denotes that the label smoothing noise function is learned more readily. Left: Min noise fitting for a shake shake 96 model with radial wave label smoothing of varying frequency; lower frequencies are learned more readily, Center: Min noise fitting for wide resnets of variable width frequency 0.039; the larger model learns this frequency more readily, Right: Min noise fitting for shake-shake models of variable width at frequency 0.039; larger models learn this frequency more readily.
Although all models tested exhibit spectral bias, found that the precise nature of the bias depends on the choice of model. For example, with all else fixed, increasing the width of model decreases its spectral bias, enabling more readily fit target frequency functions. This trend evident in Figure for the wide-resnet family (Center) and the shake-shake family (Right). Results of our label smoothing experiment on all six models we tested are included section A.2
4.2 SENSITIVITY TO NATURAL IMAGE DIRECTIONS
Although label smoothing with radial wave offers a convenient global picture of spectral bias, by replacing the radial wave with other target functions we can use the same methodology to test more localized aspects of spectral bias. We take step in this direction by considering the family of target label smoothing functions S(X; k) = s2m f X Fx).), where F is a diagonal Fourier basis image with frequency k and the same dimensions X. visualized in Figure (Bottom).
We consider (a subset of) Fourier basis images because the Fourier spectra statistics of natural images well studied (see, e.g. Tollurst al. (1922) natural images tend to be composed of Fourier basis images with amplitude proportional to their inverse spatial frequency. Indeed, in Figure 4 we find that shake shake_90 96 is more sensitive to label smoothing in low image frequency directions.
This finding is consistent with theoretical predictions of Basri et al. (2020) that models learn faster in regions of higher density of training examples: since low image frequencies more common

---

Under review as a conference paper at ICLR 2022
Min Noise Fitting
-0.6
-0.7
10
Image Frequency
k=0 k=2 k=3 k=5
k=6 k=8 k=13
k=21
Figure 4: Models are most sensitive to variations of low (but nonzero) image frequency, which dominant in natural images. Top: Effective noise fitting of wrn model with label smoothing of frequency 0.038 in various unit norm directions corresponding to Fourier basis images (Bottom) indexed by image frequency (scaled [0, 1] for visualization).
in natural images, the effective sampling density enjoyed by label smoothing target is higher in these directions. It is also possible that this bias is inherent to the convolutional model architecture regardless of the data distribution, discussed intorizz-immene et al. (2020b), that combination of both effects is at play. Determining the precise cause of this image frequency finding is an interesting direction for further study.
4.3 SPECTRAL BIAS AND REGULARIZATION
We also found relationship between regularization and spectral bias, which is explored in Figure and Figure Some forms of regularization, like weight decay and dropout, are amenable to study by either of our methodologies; Figure 5 demonstrates agreement between these methods in the context of varying weight decay. Further results showing agreement between these two measurement methodologies are presented in section A.4
In total we consider types of (explicit and implicit) regularization: weight decay, training set size, data augmentation (Cubuk al.) 2019h /Zhang al.) (2088), and the strength of Mixup augmentation (Zhang et al. 2018). The latter three involve changes to the training data and are therefore only directly studied via our interpolation methodology, which we use in Figure
High Frequency Fraction of the Learned Function
thin between
Nolse Fitting
0.4
-0.75
0.3
-0.05
0.000
0.001
0.002 0.003 0.004 0.002 Weight Decay
0.0011 6. 0.0005 0.001 Weight Decay
(a) Label smoothing
(b) Linear interpolation
Figure 5: Weight decay inhibits learning high frequencies. Results here are from shake shake_shake. _32 using the radial wave label smoothing methodology at frequency 0.038 (Left) and linear interpolation within and between classes (Right). Both measurement techniques show that increasing the penalty on the norm of the weights produces stronger spectral bias, in the form of diminished fitting of high frequency noise and a lower-frequency learned function.
7

---

Under review as a conference paper at ICLR 2022
Weight Decay. Weight decay is penalty on the square norm of the model parameters; stronger penalty encourages a model to learn morleer-norm weights. Figure 5 shows that increasing the weight decay also encourages a model to learn high frequencies slower (Left) and to ultimately be lower-frequency (Right). This result is consistent with the expectation that smaller-norm weights yield learned function with smaller Lipschitz constant and thus smoother changes in output (as discussed e.g. in [Szegeyy et al. (2014)). This experiment also demonstrates the nuanced relationship between the bandwidth or smoothness of the learned function and its accuracy; shake shake .32 trained with weight decay 0,0005 or 0.001 achieves higher test accuracy compared to the same architecture trained with either more or less weight decay. Neither complexity nor smoothness is, in itself, purely beneficial or detrimental: the trick is to balance them appropriately for the dataset.
High Frequecyy fraction the
Functton
High Hightresueee requeecy traction of the Leained Function
within
High Frequetryy Fraction the Leamnd Function
between
between
between
0.3
02
(a) Varying training set size
(b) Varying data augmentation
(c) Varying Mixup strength
Figure 6: Implicit regularization in the form of dataset size and Mixup augmentation produces lower-frequency learned function, but RandAugment and AutoAugment can increase func- tion frequency. Left: Interpolation experiment for wrn_160 trained with varying training set sizes: training with more examples produces a lower- freee-ffy learned function, particularly within-class Center: Training with Mixup (Zhang et al.) 2018) makes wrn 160 lower- freeuffee whereas training with Randnuggent (Cubuk al. (019bb) or Augment (Cubuk et al. (0T9a) makes it slightly higher-frequency. Right: Modestly increasing the Mixup strength produces freeuenyyy learned functions, but further increasing the Mixup can reintroduce high frequencies within-class (Top).
Training Set Size. As shown in Figure 6 (Left) on wrn_160, we find that increasing the number learned function. Note that all models in Figure 6
of training examples produces a (Left) were trained for the same number of epochs and with the same batch size, so the models trained with more examples were also trained for more gradient steps (since each epoch contained more batches). The models trained with more examples therefore had ample opportunity to fit the high frequencies present in their counterparts trained with fewer examples, and yet did not. This result supports the idea that additional examples serve as an implicit regularization by forcing the model away from more complex solutions that would predict the added examples incorrectly.
Data Augmentation. Data augmentation is a common strategy to increase the effective training set size without the expense of actually collecting additional examples. In Figure 6 (Center) we consider the effects of RandAugment (Cubuk et al.] 2019b), Auto/ugment (Cubuk et al. 2019a). and Mixup (Zhang et al., (018)) data augmentation strategies, each of which tends to improve the final accuracy of the trained model (with Ragment producing the most substantial benefits in our experiments). RandAugment and AutoAugment generate new images by applying geometric and lighting transformations, whereas Mixup produces new images by linearly interpolating between pairs of existing images. We find that training with RandAugment or AutoAugment tends to make the learned function slightly less smooth between examples of the same class (Top). Mixup (at strength 0.1) makes the learned function substantially smoother, both within-class (Top) and between-class (Bottom). This experiment reinforces the nuanced relationship between function complexity and performance; a model can be improved by the addition of helpful complexity (as in RandAugment and AutoAugment, which encode robust transformation priors) or by the removal of unnecessary complexity (as in Mixup, which encodes a simple linear prior in unsupervised input regions).
Mixup Strength. Mixup augmentation (Zhang et al. 2018) perturbs each batch of training data as follows: each example is matched with partner based on random permutation of the batch, and each example is then perturbed towards its partner by an interpolation amount À drawn from a symmetric beta distribution (with the same \ used for all examples in the batch). We refer to the parameter of this beta distribution as the Mixup strength, as it controls the degree to which the augmented images tend to lie close an original training image or close the average of two training images. A parameter
8

---

Under review as a conference paper at ICLR 2022
of 0 corresponds of 1 corresponds to the uniform distribution over À E [o, 1], and a parameter of oc corresponds to A 0.5, the exact midpoint between a pair of training images. Figure 6 (Center) uses Mixup strength 0.1 (one of the values recommended by /Zhang et al. (2)8)). Figure 6 (Right) shows that, as expected, increasing the strength of the Mixup augmentation generally makes the learned function lower-frequency, but that Mixup that is too strong can yield high frequencies. Interestingly, the impact of Mixup strength on the learned frequencies different within versus between classes, suggesting that perhaps tuning this parameter separately for these two cases may prove beneficial.
augmentation (A 0 or À 1, always using the original images), parameter
4.4 SELF-DISTILLATION
In a sense another form of regularization, we finally consider the effect of self-distillation on the smoothness of the learned function. In self-distillation, a 'teacher' model is trained with some form of strong regularization, our case combination of weight decay and early stopping based on training loss. A 'student" model with the same architecture is then trained from scratch to fit the pseudolabels produced by the teacher, instead the original one-hot training labels (alternatively, an interpolation between the pscudolabels and the original labels may be used). Prior research (Furlanello et al. 2018) found that this procedure can train student models that outperform both their teachers and a baseline (the same architecture trained to completion the one-hot labels).
Prior research has also sought to understand the mechanism behind knowledge distillation and seff-distillation in particular. Mobahi et al. (2020) finds that self-distillation acts regu- larizer by limiting the basis functions available to the student learn. We complement their theoretical work with our interpolation experi- mental methodology in Figure 7 where indeed we find that the student model learns lower-
High Frequency Fraction of the Learned Function
within
between
0.4
0.2
frequency function than its teacher, when both have the same training loss. We conjecture that this effect is analogous to low-pass prefiltering common in digital signal processing: high- frequency target function that we cannot ade- quately sample is first smoothed (in this case by being approximated by a regularized teacher) and then it can be modeled via samples without further loss in fidelity. Without this prefiltering, our samples are inadequate to capture the com- plexity of the target function, so we reconstruct an imperfect version corrupted by aliasing.
0.0
Teacher Self-Distillation
Student
Figure 7: Self-distillation produces student model with a lower-frequency learned function than its teacher. The teacher Shake_shake..a model is trained on one-hot labels and stopped early when training loss reaches a threshold. The student is trained to fit the pseudolabels produced by the teacher until the same training loss threshold is achieved, at which point it has higher validation accuracy than the teacher.
CONCLUSIONS
In this paper, we introduced two methods to measure spectral bias in modern image classification neural networks, and applied these methods towards the central question:
What kinds of function frequencies are needed for modern neural networks to generalize?
Specifically, we applied these methods to examine the impact of variety of training choices on the learned frequencies. Among training choices that improve validation accuracy, using larger model, training longer, and using RandAugment (Cubuk et al. (019b) or AutoAugment (Cubuk et al.) 2019a) tend to increase the frequency of the learned function. However, regularizing with modest weight decay, training with more examples, using modest Mixup augmentation (Zhang et al.) 2018), 8), and performing self-distillation all improve validation accuracy while decreasing the frequency of the learned function. Our findings support the common wisdom that, while some complexity is necessary to fit the data, unnecessary complexity can harm generalization. Our experimental methods offer a valuable window into precisely what kind of function complexity we should strive for.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
An anonymized copy of our code is included in the supplement. Our work uses the publicyy-availabl CIFAR-10 dataset (Krizhevsky et al.
ETHICS STATEMENT
Since our work is foundational rather than applied in nature, we do not anticipate any direct ethical concerns. However, as with any foundational research in machine learning. our work has the potential to improve the performance and controllability of machine learning models, particularly in the domain of image classification. These improvements in turn may find use in ethical (e.g. medical diagnosties, autonomous vehicles) or ethically questionable (e.g. surveillance) applications.
As for direct ethical impacts of our work, we acknowledge the energy costs of training and evaluating the models used in our experiments. One of our key contributions is proposing method (via linear interpolation) to measure the frequency content of learned function without retraining, which should enable future experimental research in spectral bias to be less energy-intensive.
REFERENCES
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies, 2019.
2019.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for imput of non-uniform density, 2020.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pp. 541-549. PMLR, 2018.
Zac Cranko, Simon Kornblith, Zhan Shi, and Richard Nock. Lipschitz networks and distributional robustmess, 2018
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data, 2019a.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space, 2019b.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 2012.
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks, 2018.
Xavier Gastaldi. Shake-shake regularization, 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Implicit bias of gradient descent on linear convolutional networks, 2019.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry, 2020.
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training neural networks, 2018.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On batch training for deep learning: Generalization gap and sharp minima, 2017.
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). htttp::/www.cscce.. www.wwcs.tee.. http://ww....oooooooooo/eeaaa/aa wwwt html
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse, 2019.
Christian H.X. Ali Mehmeti-Gööee David Hartmann, and Michael Wand. Ringing re (luuss: Harmonic distortion analysis of nonlinear feedforward networks. In International Conference on Learning Representations, 2021. URL httthhtt////ooo /ttps////pp/eeeeeeeeee .nt./oorum?dd-aaaaaoo qlxit
Hossein Mobahi, Mehrdad Farajtabar, and Peter I.. Bartlett, Self-distillation amplifies regularization in hilbert space, 2020.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang. Benjamin L. Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity, 2019.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. search of the real inductive bias: On the role of implicit regularization in deep learning, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring general- ization in deep learning, 2017.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over- wvr-paraaetriizaioon in generalization of neural networks, 2018.
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Hold me tight! influence of discriminative features on deep network boundaries, 2020a.
Guillermo Jimizeiimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Neural anisotropy directions, 2020b.
Samet Oymak, Zalan Fabian, Mingehen Li, and Mahdi Soltanolkotabi. Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian, 2019.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Pranceth Netrapalli. The pitfalls of simplicity bias in neural networks, 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data, 2018
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision, 2015.
DJ Tolhurst, Y Tadmor, and Tang Chao. Amplitude spectra of natural images. Ophthalmic and Physiological Optics, 12(2):229 232, 1992.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint 1rXiv:1705.08292 2017.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in
frequency domain, 2019.
Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regularization for deep residual learning. IEEE Access, 7:186126-18813366 186136, 2019. ISSN 2169-352 doi: 10. I 109/ access. 2019. 29600666. URL /xx www.dxxdoo...... wrg. 11099///...110//CC...... 2019. 2965666

---

Under review as a conference paper at ICLR 2022
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, and Justin Gilmer, A fourier perspective on model robustness in computer vision, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018
Xiao Zhang, Haoyi Xiong, and Dongrui Wu. Rethink the connections among generalization, memo- rization and the spectral bias of dnns, 2021.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 LINEAR INTERPOLATION: METHODOLOGICAL DETAILS
For each sampled path, we compute the discrete Fourier transform (DFT) of the prediction function separately for each of the M class predictions, take the (real) magnitude of the resulting (com- plex) DFT coefficients, and average them among the M classes. We then average these coefficient magnitudes among the many paths (all within-class paths and, separately, all between-class paths) and compute the fraction of this averaged DFT magnitude that allocated to the high frequency components. We (i.e. frequencies between and 0.05) "low" and the remaining (between 0.05 and 0.5) "high", but qualitatively results are not sensitive to this particular threshold. The highest frequency we measure is 0.5, the Nyquist rate corresponding to our sampling distance of 1; if a particular maximum frequency is desired for measurement all that must be changed achieve it is the sampling distance along the interpolating paths.
simple threshold that considers the lowest 10% of frequency components
A.2 LABEL SMOOTHING
2033
Noisa riting
Noize Fiting
Nobe Frttng
0638
2033
2035
0.29
2019
2019
0:04
304
3.44
Epoch
Ezoch
Epoch
(a) shake shake_shake..m 32
(b) shake shake _9
(c) shake. shake.shake
shake 112
botte Fiting
5038
Noize Fatina
5035
0:36
5017
5017
0257
5002
2013
-08
0.28
2019
2033
0239
H00
25
Epoch
Epoch
Epoch
(d) wrn_32
(e) wTn
(f) pyramid net
net
(A) Noise fitting
Min Noise Fitting
Min Noice Fitting
-0.65
-0.70
-0.50
-3.000
-0.85
-0.75
0015 0.36 0.377 поа8 note n.00 Label Smoothing Frequency
0.35 0.3s 0.027 a.ss a.z9 0.400
6035 0.37 n.s 6.099 0,000
Label Smoothing Freouency
Labe Smoothing Frequency
(b) _32
(c) shake shake 96
(d) Mlake_sshake_111
Fitting
Min Noise Fitzing
-0.0
Min Noise Fittina
-8.65
-0.05
-0.70
-0.75
-0.75
-0.90
00.55
0.03 0036 0.037 8.38 0039 8.40
0.255 0,12e 0,027 0.ss 2.39 up45 Label Smoothing Frequency
0,035 nome 0307 auss aus notp
Label Smoothing frequency
Label Smoothing Frequency
(e) wrn_32
(f) wrn_160
(g) pyramid net
(B) Min noise fitting summaries
Figure 8: All six models we tested exhibit spectral bias. Here show noise fitting when training each model with different frequencies of radial wave label smoothing.
A.3 SENSITIVITY TO NATURAL IMAGE DIRECTIONS
13

---

Under review as a conference paper at ICLR 2022
Notee ratmng
Notee FRAINa
Nase Fitina
2.6
4.0
Epach
(a) shake shake 32
(b) shake shake 96
(c) shake shake 112
Notoen Riting
Nobe SUUV
Nosee
100 Epoch
Tpoch
Egocn
(d) wrn.32
(c) wTn_160
(f) pyramid
(A) Noise fitting
Min Firting
-0.6
Min Nose Fisting
Min Noss Fitting
-0.7
-0.7
-0.7
-0.8
-0.8
-0.8
-0.9
0.0-
0.9
-1.0
-10
-10
Image Frequency
Image Frequency
Image Frequency
(b) shake shake 32
(c) shake shake
(d) shake shake
Firiing
-0.6
Min Noise Fitting
-0.8
Min Noise Fitzin
0.8
-.7
-0.7
-0.7
-0.8
-0.8
-0.9
-0.9
-0.9
-1.
20
10
10
15
Image Frequency
Image Frequency
Image Frequency
(e) wrn 32
(l) wrn_160
(g) pyramid nel
(B) Min noise fitting summaries
Figure 9: All models we tested exhibit sensitivity to variations low (but nonzero) image frequency, which are dominant in natural images. Here we show noise fitting when training each model with label smoothing of frequency 0.038 in various unit norm direction corresponding to Fourier basis images indexed by frequency k.
A.4 AGREEMENT BETWEEN LABEL SMOOTHING AND LINEAR INTERPOLATION
A.4.1 VARYING THE FREQUENCY OF THE RADIAL WAVE LABEL SMOOTHING
When the frequency of the radial wave used for label smoothing increases, models take time fit the smoothing noise. Figure 3 (Left) and Figure & show this using label smoothing; the corresponding interpolation experiment is shown Figure TO The target frequencies between 0.035 and 0.04 are close enough that, were the model to fit each perfectly, the interpolation curves would be nearly visually indistinguishable, as we can tell from F LLftt. However, the model actually smoother when trying fit higher frequency label smoothing noise, because fits this noise less well (in addition to fitting it later in training). We can see this effect, for instance, by noting that wrn_32, wrn. 160, and shake. shake. 32 fail to fit frequency 0.04, both in Figure & and Figure 10)
also worth noting that, in Figure 10 and repeatedly across our interpolation experiments, the learned function is smoother (lower-freus frequency) within-class and less smooth hhigher-fequenne between-class. This be expected since the model must change predictions somewhere along the path between examples from different classes.
14

---

Under review as a conference paper at ICLR 2022
00
0037
303
003
0032
0.033
0.39
0.039
4032
0.04
D1AM
0.04
06001
0.6
0.04
ssaks_shakee
(b) shake_sshhe Baakke__hake_9966
(c) Shaake_shake__. _112
0022
0.033
0/15
60
0.032
0.7
0.0
0.035
0.0
01/5
0923
0.07
0.37
105
00
32
0.
0.8
85
(d) wrn_32
(e) wrn_160
( pyramid_net
(A) Within-class
0.0
0.05
0.2
031
032
0.8
(a) Baaitte__ske____ .32
(b) Shhakesshake_p00
(c) shake_shakee111
5.00
0.
0.035
6.00
0015
0.037
0000
0.055
0035
0.099
0.03
0.035
5.00
0.04
0600
0.04
0.04
(d) wrn.32
(c) wrn_160
(f) pyramid_net
(B) Between-class
High Trequency traction of Loomed Function
Hon Frequency Fractio the Do0 norroed Functon
ictweer
letwee
Ietwor
Requency
62
02
417
Jabel Sensol ning Rriquennyy
Label Smeething Frecucncy
Label Smoothing Frequency
shake_shake_32
(b) shake shake 96
(c) shake shake 112
High trequrrryy trent 0f Learned Lesmeed Furciion
wthn
High Rrsquetcy Fracton the Loamed Functinn
Athin
Attin
Leaned
235 217 23*
hing Reqquency
Smoothing Frecuency
Smoothing Frecuency
(d) wrn_32
(e) wrn_160
(l) pyramid_nne
(C) DFT summaries
Figure 10: Spectral bias is evident via interpolation when training with radial wave label smoothing at various frequencies. Results here parallel those in Figure g but using the linear interpolation methodology.
A.4.2 LEARNING LOW FREQUENCIES FIRST
The label smoothing experiment presented in Figure 3 (Lett) and Figure shows that low frequency target functions learned earlier than higher frequency targets; also confirm this finding
15

---

Under review as a conference paper at ICLR 2022
0.2
062
165
100
240
242
24C
000
Seaakeshaak_32
(b) shake Laakke_haak_966
(c) Laakke_shak__11e _112
0.2
0.03
0012
0.0
6m
24C
(d) wrn.32
(c) wrn.160
(f) pyramid
(A) Within-clase
0.09
0.1
0.64
031
18
12
04
66
03
0.0
0.2
0.6
5.6
0.3
2.1
5.
03
(a) shake_shake.5 _hake_32
(b) chake_shake,
(c) shake_shake _12
0.00
< 034
162
0.00
90
162
160
242
282
20C
0.00
(d) wrm_32
(c) wTn_160
(f) pyramid
(B) Between-class
High Freqeeyy Fardon orehe Lestree R3 uncteen
wttn
High Fequercy Fradion Leatned surction
Athin Setwoer
Attin betvesr
between
Fequency
02
02
340
360
Epoch
Epoch
Epoch
(a) shake Shake_shake.32
(b) shake
(c) shake.shake_111
High High Trequency Iracian of Leseeed fundion
Hon Frequency FreuunyyFFeetii Frectio Learned Furction
Atten
wttin
Fiequency
80
80
Fpacch
(d) wrm_32
(c) wTD_160
(f) pyramid_net
(C) DFT summarics
Figure 11: Spectral bias is evident via interpolation when training as usual and comparing checkpoints at different epochs.
using interpolation with model checkpoints saved at different epochs throughout training (without any label smoothing). We see that indeed in the early stages training low frequencies predominate, with higher frequencies entering training proceeds. Surprisingly, we also that towards the end of training low frequencies again predominate; the reason for this is an interesting question for further research.
16

---

Under review as a conference paper at ICLR 2022
A.4.3 WEIGHT DECAY
Figure Secure/]shar shows on shake_shake.33 _22 via both label noise and linear interpolation that increasing the weight decay results in a smoother (lower frequency) learned function. Figure shows the same result across all six models we tested.
0.003
0.0033
0000
Nnoen Fiting
01025
Nose Fiting
0.0055
0.85
0.001
0.003
50
100
:53
266
350
50
320
150
303
250
30
386
150
283
250
(a) shake shake_32 .32
(b) shake.shake 96
(c) shake.shake 112
Notw siting
0.0055
Noise 5tingg
0,0055
0.002
0.002
0.05
0.003
0.005
Nobse
2.6
160
Epoch
Tpoch
Epocn
(d) wrm.32
(c) wrn wrn_160
(f) pyramid
(A) Radial wave label smoothing
Fitting
Min No No.se Fitting
Min Noise Fitting
-0.75
0.79
-0.70
-0.00
0.75
-085
-0.80
-0.80
0.000
0,002 ubUS DU04 0.0s
0,000
0.001
0,002 2,003 0,04 0.055 Weight Decay
0.000
0,01
0,002 3,003 0,04 0.05s Weight Decay
Weight Decay
(b) shake_shake_32
(c) shake shake 96
(d) shake shake 112
-0 -0.500
Min Noise Fitting
-0.56
Min Noise Fitting
-0.3
0.523
-0.58
-0.4
-0.550
-0.6
-0.72
Minn
-0.74
0,002 0.003 Weight Decay
8.022 ados
Decay
Weight Decay
(c) WTn 32
(f) wrn_160
(g) pyramid net
(B) Min noise fitting summaries of label smoothing. Note that for pyramid net, min noise fitting does not fully capture the results noise fitting: the "dips" are clearly separated across epochs, with higher weight decay inducing delayed dips, even if they achieve similar minimum noise fitting values.
17

---

Under review as a conference paper at ICLR 2022
0.44
0002
0400:
0.0022
0,0002
001
0.000
0.001
0.003
CUOOO
0.2
400
27
5.0
0.
032
(a) shake shake 32
(b) shake 96
shake
(c) shake shake 112
0.32
(100
(d) wrn.32
(c) wrn 160
(f) pyramid net
(C) Within class linear interpolation
0002
504
6.0002
0.0005
0001
0.001
0.005
0.005
0.005
6m
0:1
30
27
06
000
034
000
0.6
1.6
114
000
14
(a) chake_shake..o
(b) shake_shhhhe shake_96
(c) shake shake. 112
0.01
0.1
642
002
6.1
60t
0.01
(d) wrn_32
(c) wrn_160
(f) pyramid
(D) Between-elass linear interpolation
High tree aerry Fartoo the Loatmeed Funcilnn
wthin betwes
High Rreuur Fracton the Ieaine hunction
Athin betwoes
High Reequrrc Fration Learned
Attin betvesr
83
3 B
Weight Daay
Weight Docay
Weight Decay
shake shake 32
(b) shake shake shake 96
(c) shake.shake 112
shake
High requency frection Be Levined Funttin
wittin
Hegh Prqque ncy Frection of Learned purction
Attin
Athin
Reequecc the Learned
2. R. E
bu
05002 00000 3012 5,00
00000
B.0KS 0001 LOGS
BOOKS OOOI 200S 00001
Weigne Decay
Welight Decby
Weight Decay
(d) wrn_32
(e) wrn
(0 peramid_net
(E) DFT summaries of linear interpolation
Figure 12: Weight decay increases spectral bias, producing lower-frequency learned function.
18

---

Under review as a conference paper at ICLR 2022
A.5 TRAINING SET SIZE
Figure (Left) shows the regularization (frequency reduction) effect of increasing dataset size for wrn_160. Figure 13 shows the same effect on all six models we tested.
099
5007
600
0.:
(a) shake shake 32
(b) shake shake 96
(c) shake shake 112
25k
20k
0.0
20h
10c
30k
10k
354
436
002
60
30h
30h
601
34
04
00
0.5
060
03
03
0.5
06
32
(d) wrn 32
(e) 160
(f) pyramid net
(A) Within-class
53k
45k
10c
000
10c
30k
0.5
001
(a) shake_shake.B. shake_32 .32
(b)
(c) shakeshhsheee Baakkk_ssake_eeee 112
12k
0.64
50c
031
35k
358
204
61k
50h
50k
001
00
155
6m
022
66
02
08
(d) wrm.32
(c) wtn_160
(f) pyramid_net
(B) Between-class
High herqunty trect mn the Leaned Function
High Frequency Frest of The Leamadd mondiirrii Funttion
Atten
Recqueccy Frettn Funztion
ktten
24
60
10
50
26
Tuining
(a) shake shake 32
(b) shake shake 96
(c) shake shake 112
Highting Hightiuuungg treuurry fraition dhe Leatred Serreeeuurca Funcion
Helh Froqencyy Fration The toamadd Furccoon ton
High Fesuercy Faition Loairaid harcion
Ex Fa
befowen
betweer
Traniing Set. Size
Training Set Size
Training Size (1 pyramid.net
(d) wrn_32
(e) wrn.160
(C) DFT summaries
Figure 13: Increasing the number of training examples reduces the frequency of the learned function.
19

---

Under review as a conference paper at ICLR 2022
A.6 DATA AUGMENTATION
Figure ( (Center) uses linear interpolation to study the effect of common data augmentation pro- cedures on the learned frequencies for wrn_160. Figure 14 shows the same experiment on all six models we tested.
Vote
Vote
Vote
lard
Asto
land
0015
land
Meeup
0010
0000
0000
(a) shake shake .32
(b) shake..sake..... 96
shake
(c) shake
Rand
Auto
Meup
0010
Vioup
MoUp
0.3
0.4
3.2
0.00
0.4
C6
0.32
(d) wrn.32
(e) wrn.160
(f) pyramid_net
(A) Within-
0.2
land
Vote
2003
land
Mone
000
Wa
Tand
0.33
Autn
Satn
0.2
0m
0.5
0:00
0.6
(a) shake shake 32
(b) shake shake 96
(c) shake shake 112
000
Sand
0.0
Sand
Acto Wicip
Auto
1987
Weap
002
Vicup
0.0
601
081
40
0.5
66
3.2
0.00
0.5
60
0.2
0.5
06
0.2
(d) Wrm_32
(c) wTn_160
(f) pyramid_net
(B) Between-class
High treig mency Freeiion
of Be Levimed Function
wittin betwecn
High Requeenc Freciion
of the Leameed Purk purttoon
wittin
Requenic Fracton
Attin
betwrce
82-
&.
Here Band AuTo Mixu
No44 Bund AdD Maua AugIT entaaionn
km
No44 Data AugIT entation
Pannd AND Mixa
Rugmentation
(a) shake shake .32
(b) shake-shake.ann 99
(c) shake.shake.11 shake_
High Highh Fequeryy fraction of Leatne function
High F Eequency Fradion of ah Learned surgtion
Fiequency purction
03
62
Rand Auc Shrup
furd
Mexa
Augieentation
Data
(d) wrn_32
(c) wrn.160
(f) pyramid
(C) DFT summaries
Figure 14: Mixup augmentation produces a lower-frequency learned function on all models tested. On some models, RandAugment and AutoAugment produce higher-frequency learned function.
20

---

Under review as a conference paper at ICLR 2022
A.6.1 MIXUP STRENGTH
Figure 6 Rightt) uses linear interpolation to show that training with Mixup data augmentation ZZhng al., 2018) causes wrn_160 to learn lower-frequency function, but much Mixup can produce
hugher-freueng function. Figure I5 repeats the experiment on all six models we tested.
0.00
0013
02
04
0,4
5.00
0.8
0000
2.00
0.5
0.5
0000
00
0.2
0.4
06
0000
00
0.
0.
66
2hankesshake322
(b) shake shake. 96
(c) shake shake 112
0320
00133
0013
0021
3.8
(d) wrn
(e) wrn
(f) pyramid net
(A) Within-clan
0.0
00
0.5
0.
38
0.2
0.4
66
0.2
66
08
(a) shake shake_
Shake-shake.aae .32
(b) Siakkesshake _96
(c) shake. shake. 112
0.2
0.8
3.5
3.5
032
014
000
0.0
(d) wrn 32
(e) wrn. 160
i pyramid net net
(B) Setween-class
High hequancy traction the Learned Function
wiht Ietwoe
High Requuncc Frecton Laamed nunrion
Athmn letwee
Atti ictwee
64
Freuueccy
01
R7: RS mX 1.
DVE
RT PC
kopp Strength
Micip Strength
Moxp Strength
(a) shake shake 32
(b) shake shake 96
(c) shake shake 112
Pheppery freet ben the Lzomeed Funciion
High High Feecuency Fraction unamed Eutttioon
kdtin
Attin
Froquency
02 U.C.
(d) wrn.32
(e) wrn.160
( pyramid.net
(C) DFT summaries
Figure 15: Modest Mixup augmentation produces lower-frequency learned function, but Mixup that is too strong can induce higher frequencies.
21

---

Under review as a conference paper at ICLR 2022
A.7 SELF- SELF-DISTIAN DILL-DIISILLAIION
Figure 7 uses linear interpolation to show that self-distillation with Shake.shake.00. 96 produces a student model that is lower-frequency than its teacher, even though both are trained the same training loss and the student has higher validation accuracy than the teacher. Figure T6 shows the same result across all six models we tested.
Teacher
Teacher
Teacher
Studett
Stident
Stddent
12.0.3
12 992
08:
0.2
0.2
0.
(a) shake shake _32
(b) shake.shake.aaa 99
(c) shake.shake... 112
0.51
0.64
Teacher Student
0.03
0.2
0992
09:
0/1
0.4
3.5
0.01
60
5.0
(d) wrn_32
(e) wrn.160
(f) pyramid
(A) Within-clat
0.23
Rocher Sudent
5mm
Teacher 2uudent
tocchr
2..ent
3.1
106
022
032
0.6
(a) shake shake .32
(b) shake_shake,_ee _9
(c) shake Baacee__sake___l_ _112
0.44
Student
2t.dent
11 Fall
St.dent
0/02
000
083
0.
01.
3.
0.00
12.
06.
124
110
06
(d) Wrm_32
(c) wrn.160
(f) pyramid_net
(B) Between-
High Frequeccy frarle Bee Lewmee Lrcdoon
wihin
Hit Froo Rec.ency Fratt the Loamed nunrioon
kthm betueer
High Resuency Facton tuntiion
ktmm betweer
between
00
00
Distiletoon
Disillation
Disiilatioo
(a) shake shakeesshaae.... .32
(b) shake_shake_000
(c) ssakesssakeeee _12
High Erequncyy traction of Learned functen
High Frequency Fracion af Learned do sunctiin
Learrne punction
Frequency
60
Texder
Sell-Darilattion
Sell-D
(d) wrm_32
(c) 160
(f) pyramid
(C) DFT summaries
Figure 16: Self-distillation produces a lower-frequency student compared to its teacher.
22


Reviews:
[{"review_id": "rui4dvu7lXT", "reviewer": "Reviewer_axgN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths:\n\n-  The text is mostly readable (apart from a few parts as denoted below). \n-  The links to model distillation are interesting, but not explored enough. \n-  The motivation is clear.\n\n# Clarification questions: \n\n-  How is sec. 3.1 (fig. 1) different to classic overfitting? \n-  In the first paragraph of sec. 3.2 the paper mentions ‘grouping of images in pairs’, however it is not mentioned in that paragraph how the grouping is done.\n-  What is the ‘effective amount of interpolation’ mentioned in sec. 3.2?\n-  One of the most popular networks (and the almost default baseline) is ResNet, so it is surprising that ResNet is not used. In addition, what are the clean accuracy of the networks used? \n-  Why are ablation studies on different regularization schemes (sec. 4.3) conducted in different networks? Why are wide-residual nets preferred for weight decay and shake-shake for the training set size?\n-  Why does RandAugment add ‘necessary complexity’? How does RandAugment differ from Mixup or other training schemes that demonstrate low-frequency preference?\n-  Frequently, when training image recognition networks on CIFAR10 data augmentation is used, however it is not explicitly mentioned (other than in sec. 4.3). Was there any data augmentation used in sec. 4.1 and 4.2? \n-  In sec. 4.2, is there some intuition for stopping at k=21 (fig.4)? \n\n\n# Improvements:\n\n-  The paper includes empirical results and decisions without mentioning any motivation or theoretical grounding for those. For instance, see the comment about different networks in the ablation studies, or why those networks were used over state-of-the-art networks or the popular ResNet. In addition, neither an algorithm or the source code is provided, which makes it harder to assess the decision choices on the code and the plots. \n-  I would recommend sharing further information on the function S (sec. 3.1); sinusoidal might not be ideal for image domain applications, which is again not mentioned or discussed in the paper. \n-  The paper focuses on a single experiment, i.e. image classification on CIFAR10, which makes it hard to appreciate whether it generalizes beyond this setting. For instance, do the same results emerge in self-supervised learning? \n-  In a similar vein to the previous comment, it would be more convincing if the results on different datasets (e.g. STL10, Imagenet) were presented along with additional training algorithms to demonstrate that the observations indeed generalize well. \n-  One implicit assumption of the paper is that CIFAR10 shares the same high-frequency statistics as the datasets used in practice. Do higher-dimensional datasets (e.g. Imagenet, CelebA-HQ, COCO, iNaturalist) share the same statistics and do the properties presented here generalize to those?\n-  The sec. 3.2 (which is probably the most critical in the method) could be improved in terms of writing. Currently, a number of questions are not answered. For instance: why do the path lengths vary? What do the different colors in Fig. 2 depict? Is y_lambda (fig. 2) and y_t (sec. 3.2) different? \n-  Is there some theoretical foundation on how the rapid changes in the cumulative difference of the norm (sec. 3.2) are related to the high-frequency components? \n\n\n# Minor comments: \n\n-  The ‘shake-shake’ (sec. 3.1) is mentioned without defining that it is a network or providing any citation; the reader needs to go all the way to the experiments to understand what this refers to. \n-  Lambda is not a formula (sec. 3.2). \n-  The legends are currently broken, e.g. in Fig. 3 they are blocking a critical part of the figure (i.e. > 200 epochs). \n-  Is the lambda mentioned in sec. 4.3 (Mixup strength) the same as in sec. 3.2? \n", "summary_of_the_paper": "The paper focuses on the spectral bias of neural networks, i.e., their tendency to learn first the low-frequency information. The goal of this work is to extend the spectral bias into practical image recognition networks, i.e., beyond the fully-connected nets and the NTK regime that it was previously studied. To that end, the paper conducts a number of studies in both explicit and implicit regularization schemes used in practice, and provides links for the success of model distillation. ", "summary_of_the_review": "In my opinion, the spectral bias can be a useful tool in analyzing the inductive bias of networks, but the current format of the manuscript is not ready for acceptance. There are a number of questions on the methodology and how this relates to the spectral bias, that is the main contribution of this work. In addition, the experiments are not considering several cases (e.g. other datasets, or alternative learning algorithms) that might have correlations with their results. Furthermore, previous results about the manifold of the data (ie. that it impacts the frequencies learnt) are not discussed in the paper. If the revised version addresses the improvement points, I can reconsider my rating. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dyUUxN0y_zA", "reviewer": "Reviewer_tm3W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nEven though similar methodology has been used before (e.g. in Rahaman et al.), the proposed label smoothing technique seems to be novel in being applicable to multi-class classification. The experiments are extensive and well thought-out. The effects of training parameters on spectral bias (or its proxy) is investigated from various different directions. The paper is well-written and easy to follow. Overview of related work is sufficient.\n\nWeaknesses:\nIt is not clear to me what tangible novel insights we learned about generalization beyond what has already been investigated and observed in the literature (such as Oymak et al. and similar works), namely that over-parameterized networks learn structure from data first, but can overfit to any noise over time. In other words, the paper makes observations on the effect of different regularization techniques on function frequencies but the *role* of function frequency in generalization is not investigated directly.\nMoreover, the significance of the work is somewhat limited in my opinion because it is not clear how it can aid training choices. The key conclusion seems to be that 'an ideal function should include high enough frequencies to fit the data but avoid unnecessary high frequencies that can harm generalization', which is somewhat evident and it is not clear to me what consequences it has. For instance, can we use this methodology to inform us how to choose implicit/explicit regularization or network size?\n\nOymak, S., et al. \"Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian.\" arXiv preprint arXiv:1906.05392 (2019).\n\nRahaman, N., et al. \"On the spectral bias of neural networks.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "This paper proposes a set of tools to probe spectral bias of deep neural networks, that is their tendency to learn low frequency, simpler functions earlier in training, whereas high frequencies are fit later. Authors propose adding noise to the labels through a target function via label smoothing, where the frequency and direction of the target function in image space can be varied. Moreover, they introduce a linear interpolation technique on validation data to probe the smoothness of the learned function along paths connecting natural images. Authors perform extensive experiments to demonstrate how the proposed tools can be used to investigate the spectral effect of training parameters such as model size and different forms of explicit and implicit regularization.", "summary_of_the_review": "Overall, I think the paper has merit and the proposed tools have potential to reveal novel insights on generalization properties of deep neural networks. On the other hand, the core novelty and significance of the findings are somewhat limited in my opinion, therefore I recommend marginal acceptance. I am however open to change my score either way after hearing back from the authors and after further discussion with other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cmfQMnw1RmC", "reviewer": "Reviewer_Vhe6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. **Research question**: I find the proposed research question of great interest to the community. The spectral bias of deep learning is mentioned very often as a justification of many practical tricks in the literature, but we are lacking a proper evaluation of this bias in practice.\n2. **Analysis of many factors**: I admire the effort made by the authors in studying the effect that many factors of variability have in the spectral bias of SOTA convolutional neural networks. In particular, I find the study of the role of architecture and network capacity in the strength of the spectral bias very compelling.\n3. **Connection between directional bias and spectral bias**: The directional inductive bias in the input space and the spectral bias in the function space are easily confused in the literature. In this sense, I find the analysis bridging the gap between these two inductive biases very interesting. In particular, I honestly believe that moving forward, the research community should strive to understand the interplay between the different types inductive biases present in neural networks, so I find that some of the observations in this work are a good step in the right direction.\n\n## Weaknesses\n\n1. **Evaluation metrics**: My main concern with this work is the fact that the choice of metrics used to evaluate the spectral bias of a neural network are highly subjective and difficult to interpret. Specifically, both the effective noise fitting curve and the cumulative difference norm in the linear path require to inspect a noisy curve to measure the spectral bias. This makes the final assessment of the results very qualitative. In my opinion, this seriously diminishes the strength of the presented results, and this has greatly affected my final score.\n\nNonetheless, I do not think this problem is unsolvable and I am willing to increase my score if this problem is addressed during the rebuttal. In particular, I would highly encourage the authors to look for alternative metrics to quantify the spectral bias, and perform some experiments to validate their qualitative results with a more numerically precise metric. As a suggestion, I propose the following two metrics:\n\n   - **Test accuracy when training to predict sinusoidal labels of different frequency**: One way to precisely quantify the spectral bias would be by training multiple times on the same dataset with target labels of different frequency. In particular, I propose to generate a benchmark consisting on CIFAR10 samples and different target labels $S(X)$, or similar. The test performance obtained when training to predict different frequency targets should be a good indication of the spectral bias of that network.\n   - **Roughness of loss landscape wrt the input**: One could use a similar metric as Mehmeti-Gopel et al. 2021 but in the input space to measure the frequency content of the final loss landscape of a trained neural network.\nIn any case, I encourage the authors to defend their choice of metric during the rebuttal if they believe there is no better metric to capture this phenomenon.\n\n2. **Inconclusive results**: Another source of concern for me has been the fact that some of the presented analysis are rather speculative and not based on facts. In particular, when discussing the role of different types of regularization in the spectral bias, the authors refer multiple times to the connection between their observed spectral bias and the accuracy of the different models: \"*This experiment reinforces the nuanced relationship between function complexity and performance...*\", \"*we find that [..] some smoothness [...] is beneficial, [but] too much causes detrimental underfitting*\" or \"*Neither complexity nor smoothness is, in itself, purely beneficial or detrimental: the trick is to balance them appropriately for the dataset.*\" All these statements, however, are not supported by the data, as nowhere in this manuscript is the spectral bias of the different training runs compared to the performance of these models. Please, note that I am not implying that these statements are not true, I am just merely stating that these are not supported by the results presented in this manuscript. Reaching those conclusions could be possible if one juxtaposed some quantitative metric of the spectral bias to the final performance reached by these models, but I am afraid that with the current qualitative metrics, this is not possible.\n\n## Other comments and additional questions\n1. **Missing citations**: I believe some parts of this work could benefit from the inclusion of a few missing citations. In particular, prior to Ortiz-Jimenez et al. 2020, Yin et al. 2019 had also previously studied the robustness of computer vision classifiers to different frequency components. Also, on the implications of the spectral bias, Shah et al. 2020 argued that the simplicity bias was detrimental for robustness. The directional inductive bias presented in Sec. 4.2. was extensively studied in Ortiz-Jimenez et al. 2020. And finally, Mehmeti-Gopel et al. 2021 recently studied the roughness of the loss landscape of neural networks in an empirical campaign analogous to the one in parts of this work, although with respect to the weight space, rather than the input space.\n   - Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin Gilmer. \"A Fourier Perspective on Model Robustness in Computer Vision\". NeurIPS 2019\n   - Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain and Praneeth Netrapalli. \"The Pitfalls of Simplicity Bias in Neural Networks.\" NeurIPS 2020\n   - Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard. \"Neural Anisotropy Directions\". NeurIPS 2020\n   - Christian H.X. Ali Mehmeti-Göpel, David Hartmann and Michael Wand. \"Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks\". ICLR 2021\n\n2. **Possible methodological issues in linear interpolation**: I have two questions regarding the experiments measuring complexity of the loss landscape:\n   - Why do the authors choose to work with segments of the same size, and do not fix the number of segments to query in the linear path? Do the results change significantly if one changes this element of the evaluation?\n   - Doesn't the exclusion of the incorrectly labeled samples bias the final metric towards smoother values? Specifically, if one repeats this experiment without removing those samples, is the order of complexity of the different methods retained?\n\n3. **Choice of architectures**: Could the authors elaborate on the reasons behind their choice of model architectures? Specifically, why did the authors decide to go with shake-shake models and very wide-resnets and did not provide also results for more manageable ResNets?\n4. **Role of data distribution**: The authors blame the data distribution for the interplay between directional inductive bias and spectral bias in Sec. 4.2. However, this statement is also not supported by any evidence. In this sense, I wonder if the authors have tested whether the differences in the spectral bias for different directions are not present in isotropically distributed data, e.g., a normal distribution. If so, this statement would be better supported. Nevertheless, note that Ortiz-Jimenez et al. 2020 found that the directional inductive bias was mostly due to the flow of information in the architectures, so I would not expect that to be the case. In general, it would be good if the authors could comment on what do they expect if instead of using Fourier basis functions, they repeated their experiments using Neural Anisotropy Directions, instead.\n\n", "summary_of_the_paper": "This work presents an empirical study of how different training aspects affect the spectral bias of neural networks in practice. To that end, the authors propose to inject label noise of different frequencies to the CIFAR10 dataset and suggest that the time a neural network takes to start overfitting this noise is a good metric of its spectral bias. They also propose to measure the variability of the loss landscape in the linear interpolation path between two images as proxy for spectral bias. Experiments on the effect of model architecture, explicit regularization, and data augmentation suggest that deep neural networks exhibit a strong spectral bias, in practice, which can be modulated by different design factors.", "summary_of_the_review": "Although I believe the proposed research question is of great interest to the community, and I admire the effort made by the authors in studying the effect of many training factors in the variability of the spectral bias in CNNs; I think that the overreliance on qualitative metrics and lack of quantitative results makes most of the observations in this work inconclusive. I am open to increase my score if the authors address these issues during the rebuttal, but at this stage a lean towards rejection.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VtKzlx7swmQ", "reviewer": "Reviewer_9z2T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the topic is certainly relevant and interesting, I think that the proposed methods are a bit hazy and I found some important missing details in the empirical exploration.\n\n### Smoothness estimation methods:\n\n1/ is a straight forward extension of Rahaman et al, applied to multiclass.\n - I don't really understand why you had to introduce \"effective noise fitting\" instead of just looking at the validation loss as in Rahaman et al?\n - What is the purpose of the 1/k added to the target vector ?\n - How did you choose the range of frequencies. E.g. in the exp in fig 3.a the frequency ranges from 0.035 to 0.04 which seems a rather tiny range compared to the experiments in Rahaman et al where the high frequency is 5x the low frequency\n - How can you tell from fig 3.a e.g. that the high frequencies are learned later than the low frequencies ? I see that the 0.035 yellow curve takes its minimum at around 50 epochs whereas the \"high\" frequency 0.04 dark blue one takes its minimum at around 30 epochs. Similarly in fig 3.b left where exactly is the minimum for wrn_160 ?\n \n2/ is difficult to interpret precisely:\n - The role of delta is not defined, for instance what does it mean that \"delta = 1\" in section 3.2 ?\n - Why is it called *cumulative* difference norm? What is cumulative in this measurement?\n - In fig 2. if both images of the same class, why is ||y_lambda - y_0|| high at lambda=1 (i.e. exactly the other image)\n - How precisely do you see the smoothness in these plots? In some cases it is rather ambiguous e.g. in fig 6.c bottom which experiment in the range [0.1; 1.6] is the smoothest one?\n\n\n### Empirical evaluation\n\n - The paper is lacking experimental details, such as choice of optimizer and hyperparameters. This is especially true as I would expect the learning rate to play an important role in how fast each frequency component is learned.\n - It would be interesting to include an ablation study when varying the learning rate.\n\n\nYou can also take inspiration from the method in [Zhang et al 2021] who study the spectral bias in the context of epoch-wise double descent, also using training examples, but by performing a full Fourier decomposition.\n\nReferences:\nZhang, Xiong and Wu, Rethink the Connections among Generalization, Memorization, and the Spectral Bias of DNNs, IJCAI-21", "summary_of_the_paper": "This paper proposes two methods for estimating smoothness of the prediction function learned by a deep network : \n 1/ an extension of Rahaman et al. to multiclass classification, by adding a sine of desired frequency to the target vector, then training on the new target vectors. The \"effective noise fitting\" is then the difference between the validation loss on clean targets and the validation loss on modified targets.\n 2/ a measure of change of the logits in l2 norm between 2 training examples, averaged over many examples\n\nUsing their methods, they show the effect on the smoothness of the predicted function of:\n - varying the number of parameters\n - adding explicit regularization (weight decay/mixup)\n - distillation", "summary_of_the_review": "Overall, I find the paper in its current state not ready for publication. I think that the methodologies for measuring smoothness can be improved, and the experiments should include experimental details as well as an ablation study of the role of different hyperparameters on the smoothness of the function obtained after training.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper expands the spectral bias, which has been studied in a constrained situation such as the fully-connected network, to a more practical situation of a multi-class classification situation, and proposes a novel technique that can measure the smoothness through linear interpolation of test examples.

Two reviewers highly evaluated the importance of the research question considered in this study and the value of diverse experiments applying the proposed method in various directions, and suggested acceptance. On the other hand, two other reviewers suggested rejection due to the lack of rigor in writing and experiments. I strongly agree with the reviewer's concern that  the method was only verified  on CIFAR10 and the rigor of the experiment was lacking. Unlike the spectral bias paper, which is the basis of this study, this submission is not a theoretical paper, but rather an experimental paper. I admit that it is impossible to verify in various domains as mentioned by the author. However, I believe that verification on more diverse, especially larger-scale datasets is essential at least focusing on the image classification task.
Decision: Reject