Spectral Bias in Practice: The Role of Function Frequency in Generalization
Authors: Sara Fridovich-Keil
Raphael Gontijo-Lopes
Rebecca Roelofs
Venue: NeurIPS 2022 Accept (2022)
Keywords: spectral bias
generalization
data augmentation
self-distillation
image classification

Abstract: Despite their ability to represent highly expressive functions, deep learning models seem to find simple solutions that generalize surprisingly well. Spectral bias -- the tendency of neural networks to prioritize learning low frequency functions -- is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simplified experiments. In this work, we propose methodologies for measuring spectral bias in modern image classification networks on CIFAR-10 and ImageNet. We find that these networks indeed exhibit spectral bias, and that interventions that improve test accuracy on CIFAR-10 tend to produce learned functions that have higher frequencies overall but lower frequencies in the vicinity of examples from each class. This trend holds across variation in training time, model architecture, number of training examples, data augmentation, and self-distillation. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive to the low frequencies prevalent in natural images. On ImageNet, we find that learned function frequency also varies with internal class diversity, with higher frequencies on more diverse classes. Our work enables measuring and ultimately influencing the spectral behavior of neural networks used for image classification, and is a step towards understanding why deep models generalize well.

Spectral Bias in Practice: the Role of Function Frequency in Generalization
Sara Fridovich-Keil University of California, Berkeley sfkθeecs. berkeley. edu
Raphael Gontijo-Lopes
Rebecca Roelofs Google Brain
Google Brain iraphael@google.com com
rofls@google.com com
Abstract
Despite their ability to represent highly expressive functions, deep learning models seem to find simple solutions that generalize surprisingly well. Spectral bias the tendency of neural networks to prioritize learning low frequency functions is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simplified experiments. In this work, we propose methodologies for measuring spectral bias modern image classification networks on CIFAR- 10 and ImageNet. We find that these networks indeed exhibit spectral bias, and that interventions that improve test accuracy on CIFAR-10 10 tend to produce learned functions that have higher frequencies overall but lower frequencies the vicinity of examples from each class. This trend holds across variation in training time, model architecture, number of training examples, data augmentation, and self-distillation. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive the low frequencies prevalent in natural images. On ImageNet, we find that learned function frequency also varies with internal class diversity, with higher frequencies on more diverse classes. Our work enables measuring and ultimately influencing the spectral behavior of neural networks used for image classification, and is step towards understanding why deep models generalize well.
Introduction
Two fundamental questions in machine learning are why overparameterized models generalize and how to make them more robust distribution shift and adversarial examples. Resolving both of these questions requires understanding how complex our models should be.
For instance, it is thought that overparameterized models generalize well because there are implicit regularizers that constrain the complexity of the learned functions. However, the precise nature of these implicit regularizers, and their importance in practice, remains unclear. As for how to achieve robustness, no consensus has emerged regarding function complexity. On the one hand, Cranko et al. 4 and Leino et al. 24 argue that Lipschitz smoothness of the learned function offers guarantee of robustness; on the other, Shah et al. B7 and Madry et al.
25 argue that simplicity can be harmful and robust model must actually be more complex than its non-robust counterpart.
One window into function complexity is spectral bias the tendency of neural networks learn low frequency (simple and smooth) functions first, then gradually increase the frequency (complexity) of the learned function training proceeds. Foundational work in this area has shown theoretical evidence of spectral bias by analyzing convergence rates of neural networks towards functions of different frequencies [| 35
Work done as an intern and student researcher at Google Brain.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In practice, spectral bias is difficult to measure: the most direct method involves taking a Fourier transform with respect to the input, which impractical to compute due to the high dimensionality of images. Early experimental work focused on low-dimensional synthetic data [1 2 or used proxy measurements of spectral bias 44 for instance by inserting label noise of various frequencies during training [55] However, the label noise method is limited to binary classification and involves modifying the training data, making it difficult to disentangle how other changes to the training data (such as augmentation) affect the spectral content of the learned function. It remains open question whether modern neural networks exhibit spectral bias and what role it plays in generalization.
In this work, we investigate model complexity through the lens of spectral bias, by introducing experimental methods to study the frequency decomposition of the functions learned by modern image classification networks. We find that high-accuracy models learn function that is high- frequency in regions between different image classes but low-frequency within each class, validating the intuition that a "good" model should have sharp decision boundaries to delineate different classes, but smooth behavior within each class.
Contributions. We extend the label noise procedure of Rahaman et al. 35 to enable measuring spectral bias in multi-class classification via label smoothing, and apply this technique understand the function frequencies present in high-accuracy models on CIFAR-10 10 [36
We also introduce a second method for measuring the smoothness of learned function via linear interpolation between test examples, and apply this method to both CIFAR- 10 and ImageNet |
models. We use this method to probe the effects of the training data on the learned function frequencies. and distinguish the spectral content in paths between images from the same class ("within-class paths") and paths between images from different classes ("between-class paths").
We find that higher-test-accurad between these path types, with lower frequencies within-class and higher frequencies between-class. On CIFAR-10, this trend holds as we improve the model architecture, train longer, increase the size of the training dataset, improve the data augmentation strategy, and apply self-distillation On ImageNet, this trend appears again as we consider models with more diverse architectures.
models demonstrate greater separation in frequency content
We also explore the relationship between image frequency and function frequency. By further extending the label noise methodology of Rahaman et al. (35) to study spectral bias in directions of interest through the input space, we find that CIFAR-10 models most readily learn functions of the low image frequencies common in natural images.
Via linear interpolation within ImageNet classes, we find that many models are higher-frequent in more internally diverse classes and lower- weereffre in more internally consistent classes.
2 Related Work
Implicit bias. A common belief is that some form of implicit bias imposed by the training procedure or optimization algorithm may account for the generalization ability of overparameterized neural networks, and accordingly, much research has been directed towards understanding these implicit
biases 38 4%
22
[5
29 3O 26
20
Spectral bias. Spectral bias, also called the frequency principle, is form of implicit bias with much recent attention including theoretical and experimental approaches 32 44
35
I1
50) Basri et al.
I studied spectral bias using a linear model of SGD training dynamics to show that models learn low frequency (simple) functions faster, assuming training data that is distributed uniformly on the hypersphere. Basri et al. [2] extended the analysis to consider nonuniformly spaced training data, and found that learning is faster where samples are denser; if sampling is nonuniform, then during training, the learned function will be higher frequency in regions with denser samples. Rahaman et al. [35] used a more direct analysis to show the same spectral bias toward low frequency functions and also posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters, connecting back to the idea of flat optimization minima 30 or smooth loss landscape 27 They proposed experimental methods to study spectral bias in image classification, but focused on binary subset of the relatively simple MNIST dataset M with mean square error loss.
Our label smoothing experiments are direct extension of Rahaman et al. +5 to multiclass classifi- cation with the more common cross-entropy loss. Our linear interpolation experiments are somewhat
2

---

be the validation similar to the excentlypropoosee experimental methods in Zhang et al. 150, except that we sample along paths between images rather than in regions surrounding each image; this allows for a more global measurement of spectral bias that we apply to a broad and distinct set of training procedures beyond the double descent regime studient in Zhaag et al. (50)
Model sensitivity to image frequency. While we focus on function frequency, prior research aimed to understand model sensitivity to image frequencies. Jo and Bengio [9] found that CNNs are sen- sitive to Fourier statistics of the training data, even those irrelevant to human viewers, Ortiz-Jiménez et al. [R] studied the image frequency bias induced by using a convolutional architecture, and Ortiz- Jiménez et al. 32 argued that models are sensitive primarily to discriminative Fourier directions in the training data. Yin et al. [6 introduced a procedure to measure the sensitivity of trained models to Fourier image perturbations, and applied it to study the effects of adversarial training and data augmen- tation. Our work adds to this line of research a study of the interplay between two notions of frequency: the function frequency involved in spectral bias, and the image frequencies present in the data.
3 Methodology
The goal of our work is to measure the complexity of neural network functions through the lens of frequency. In low dimensions, measuring the frequency decomposition of function is straightforward and tractable: evaluate the function dense, uniform sampling positions and compute its discrete Fourier transform. However, the function of interest in image classification is unavoidably high- dimensional, mapping images (with thousands of pixel values) to object classes. would be intractable even to collect sufficient samples of this function to compute a discrete Fourier transform, let alone compute the transform itself. Instead, we employ two complementary approaches measure informative proxies of this frequency decomposition.
3.1 Label Smoothing
The core idea for measuring function frequency introduced by Rahaman et al. [5 is to construct a sinusoid over the space of images, and use that sinusoid as form of label noise during training. Let D = {(x; y4) ) be the training examples and Dvalid = (X,, } examples, with X; an image and y; a one-hot class encoding, where Merrin is the number of training examples, naat is the number of validation examples, d the side length of an image (assumed to be square for simplicity), e is the number of color channels, and M is the number of classes, so X; E dxdxxx and y: E A5
To extend this procedure to the multi-class setting, we add noise of various frequencies to an M-dimensional label vector via label smoothing, originally introduced regularization approach in Szegedy et al. [39]. Intuitively, label smoothing removes some probability from the correct class and redistributes it equally among the remaining classes. Let S Rxxx - [0. 1] be a noise function that maps an input image X; to scalar value between O and 1. We apply label smoothing to each label ye. mapping to D, = y S(X.)) + ((x)) to retain valid probability distribution, as in Szegedy et al. [39 but with different functions S. Rather than using label smoothing as regularizer to improve performance, we use it as form of variable frequency label noise to study spectral bias. We train from scratch using the original examples X; and their smoothed labels ÿ. We evaluate on the validation images X,. comparing to both their original one-hot labels yj and smoothed labels y = 01 - S(X,)) )) + 3f (X,). Our label smoothing experiments use the CIFAR-10 dataset (3) where Mtrrin 50000, neal = 10000, d = 32, = 3. and M 10.
This method introduced in Figure M As training proceeds, we see both the training loss (be- tween predictions and smoothed labels ÿ.) and noisy validation loss (between predictions and smoothed labels yj) decrease. The clean validation loss (between predictions and one-hot labels ,j), however, initially decreases but at some point in training plateaus or begins to increase. At this point, the model has begun to learn the noise function S. Accordingly, we introduce noise fitting, defined as the difference between the validation loss on one-hot labels and the valida- tion loss on labels smoothed with the same function S that was applied to the training labels: noise fitting clean validation loss - noisy validation loss. We compare different models by com- puting the minimum noise fitting achieved throughout training. A model with higher min noise fitting more readily fits the noise function S than model with lower min noise fitting.
3

---

Noise Fitting
-0.5
Training Loss
Noisy Validation Loss
Validation Loss
-3.6
-0.7
-3.0
100
200
160 250
160 200
50
1000
150
260
250
Epoch
Epoch
Epoch
Epoch
Figure 1: Noise fitting shows when and how much a network fits a noise function of given frequency. At roughly epoch 150, noise fitting (Right), the difference between clean and noisy validation loss, exhibits a clear "dip" when the model begins to fit the noise function: training (Left)) and validation (Center Left) loss on the perturbed function drop, while improvement stalls on clean validation data (Center Right). By comparing the minimum value of noise fitting achieved by different models throughout training (the gray line in the right plot), we can compare the relative degree to which different models fit noise functions of varying frequency. A model with higher min noise fitting more readily fits the noise function. Here, we train WideResNet32 model (wide-resnet with width 32) with radial wave label smoothing at frequency 0.04. For visual clarity, we apply exponential averaging to all curves.
By choosing different functions S, we can probe nuances of spectral bias. Typically (inspired by Rahaman et al. 351 ) we choose a radial wave: S(X = a(1 + sin - [)]]\|(()))) where || denotes the Euclidean norm of the vectorized image, and a [0, 0.5] to ensure that S S(X) [0. 1]. We can vary the frequency f to understand spectral bias at this global scale. We can also choose more targeted functions S; for instance, if V is direction of interest through the space of images (i.e. V is an image-shaped vector of unit norm), we can construct S(X) = a(1 + sin f (X, V))). This allows to vary both the frequency and direction of the noise function, understand models' learnability along different directions through image space.
However, label smoothing has few limitations. Because it involves perturbing the training labels, it cannot directly study the interaction between the training dataset and spectral bias, such as the effects of data augmentation. It also requires retraining each model from scratch, with substantial investment of time and computational resources. We address these limitations with our linear interpolation technique in the next section.
3.2 Linear Interpolation
High Frequency Fraction
0.4
Label Smoothing Frequency
of the Function
0.05
119 Voll
0.04
0
DFT
0.03
0.02
0.03
0.01
0.0
0.05
0.00
0
0.01
0.22 0.03 0.04 0.05
Label Smoothing Frequency
= 0.0 A = 0.2 A= = 0.4 À 0.6 = 0.8
Figure 2: Linear interpolation measurements on functions of varying frequency. Left: Interpo- lation between images of the same class on an oracle function: the true one-hot label perturbed by radial wave label smoothing of variable frequency. As the oracle function increases in frequency. the interpolating paths become less smooth. Right: Summary of this interpolation experiment via a discrete Fourier transform; as the oracle frequency increases, so does the proportion of the DFT magnitude allocated to the high frequency components. In all interpolation figures, error bars report the standard error of the mean high frequency fraction over all the interpolating paths measured.
To complement the label smoothing approach, we also propose a methodology based on linear interpolation between validation images. Although we cannot take dense, regularly-spacee samples that cover the entire input (image) space, we can sample along specific paths and thereby glean glimpses into the spectral content of the learned function.

---

We consider two types of paths: those between images from the same class, and those between images from different classes. When considering paths between CIFAR- 10 images in the same class, we choose 200 random, distinct pairs of validation images from each of the 10 CIFAR-10 10 classes and average over the paths defined by these pairs. We do the same for ImageNet, choosing 50 random, distinct paths for each of the 1000 ImageNet classes. When interpolating between CIFAR-10 classes, we choose 200 random, distinct validation image pairs from each the 45 (10 choose 2) pairs of distinct classes, and average over the resulting paths; for ImageNet we choose 1000 random, distinct between-class paths, ensuring that each class is represented in exactly two paths.
For each pair of images (Xu, X1), we vary À E [0, 1] to trace out a path, where each image in the path given by X, XX1 + (1 X) Xy. We choose As so that the distance between adjacent images on the path is constant; each path may produce different number of samples depending on the total distance between X₂ and X1. Figure 2 (Bottom Left) shows an illustrative example of two CIFAR- 10 images and corresponding interpolation path between them. For our experiments, we typically have 50 to 100 images along the interpolation path.
Relationship between interpolation and label smoothing experiments. To validate our interpolation methodology, we consider the oracle function that maps directly from the interpolated image Xx to the noised label used in label smoothing experiments. We only consider within-class paths for this experiment, so that the original one-hot label (before smoothing) is fixed across the entire path, and the only change in the oracle function along the path is due the noise function used in label smoothing. By increasing the frequency of the noise function, we create oracle functions with higher frequencies. We then verify that our interpolation measurement indeed measures higher frequencies on the higher frequency oracle functions.
In Figure 2 Left), show example where vary the label smoothing frequency from f 0 to f f 0.05. We plot the value X along the interpolating path on the x axis, and the norm of the difference between the model output (softmax probabilities) on the interpolated image ýı and the model output on the original image yo on the y axis. This difference norm always starts at zero (although plot it by averaging many paths over buckets in X, so the curves do not actually touch the axis), and tends to increase with as the model predictions change along the interpolating path. The norm of the prediction difference allows to visualize how smooth the function is along the interpolating path; smoother prediction difference norm indicates a lower frequency function.
To quantify the frequency of the oracle function along the interpolating path, compute the per-class discrete Fourier transform (DFT). Letting (Xx, |T= denote the set of interpolated images, the DFT co-
efficient for frequency/ f is given by Y, [ Et= ÿx, [m] exp ft), where i is the imaginary number and m E {1,.... , .M) denotes the class. To summarize how the magnitudes of the DFT coeffi- cients Y, are distributed between low and high frequencies, we average the coefficient magnitudes for each of the classes and compute the fraction of the total average coefficient magnitude that is allocated to frequencies above a threshold (0.05). In Figure 2 (Right) we show this summary metric averaged over all within-class interpolating paths for the oracle functions, along with its standard error. In Fig- ure 2 (Left,, the higher frequency oracle functions have less smooth prediction norm differences, and Figure Riihht, they have a higher fraction of DFT coefficient magnitudes in the high frequencies.
3.3 Data and Models
We use the CIFAR-10 [23] dataset of low-resolution (32 x 32) natural images from ten animal and object classes and the ImageNet [X] dataset of higher-resolution (224 x 224) natural images from 1000 classes. We consider diversity of model architectures CIFAR-10 10 (6 total) and ImageNet (10 total). See Section A.T for more details on the data processing, models, and their accuracies.
4 Results on CIFAR-10
Throughout this section we show representative examples from our CIFAR-10 experiments; full results (on all six models we tested) are included in Section A.3
5

---

-2,000
Hlih Frequency frection
Min Noise Fiting
-2.675
-0.050
-9.75
0.2
2.700
-2725
-0.700
Lobel Smeathing Frequency
Epoch
(a) Label smoothing
(b) Linear interpolation
Figure 3: Modern CNNs have spatially-dependeer spectral bias. Higher min noise fitting denotes that the label smoothing noise function is learned more readily. Left: Min noise fitting for Shake- Shake96 model with gariable-frequent radial wave label smoothing; lower frequencies are easier to learn. Right: Linear interpolation for ShakeShake96 model throughout training; as training proceeds, the learned function becomes lower-freyeency within-class and
between-class.
-0.65
High Frequency Fraction the Leamed function
Min No se Hitting
-0.05
Min Noise Fitting
-0.40
0..00
-0.75
-0.00
0.0
h200 wd2
222 200 1132 PM
Width
Shake Shake Width
Model Architecture
(a) Label smoothing
(b) Linear interpolation
With n-Claas Paths
Betwean-Class Paths
Figure 4: Larger models learn high frequencies more readily; higher-accuracy models are higher-frequency between classes and lower-frequency within each class. Left: Min noise fit- ting for Wide-ResNets and Shake-Shake models of variable width at frequency 0.039; the larger models learn this frequency more readily. Right: Linear interpolation frequencies for all six models we tested on CIFAR-10. ordered by test accuracy; higher- models are lower-frequency between-class.
within-class and
4.1 Spectral Bias
We begin by applying our label smoothing methodology to find that modern image classification CNNs exhibit spectral bias, learning low frequency functions early in training and learning higher frequency functions as training proceeds. Note that Figure 3 (Left) shows spectral bias over a small but illustrative range of frequencies (the full range of frequencies
tested 0 to 0.1); noise functions of sufficiently low frequency are fit almost immediately and noise functions of sufficiently high frequency are never learned during the 250 epochs of training we tested.
Using our linear interpolation methodology and distinguishing between within-class paths (in which both endpoint images of the same class) and between-elass paths (where the path must traverse class boundary) in Figure Right), we find that this spectral bias is highly variable over the domain of images. In particular, while the learned function does increase in frequency for between-class paths, it decreases in frequency for within-class paths, indicating that the model learning to cluster and separate the different classes. Throughout our CIFAR-10 experiments, we find that higher-accuracy models have greater separation between the frequencies of their within-class and between-class paths, with fower-fryy behavior inside each class and higher-frequenc behavior in between.
4.2 Spectral Bias and Model Architecture
Although all models we tested exhibit spectral bias, we found that the precise nature of the bias depends the choice of model. For example, with all else fixed, increasing the width of model decreases its spectral bias, enabling it to more readily fit noise functions of a given frequency. This trend is evident in Figure g for the Wide-ResNet family (Lefi) and the Shake-Shake family (Center).
Although larger models learn high frequencies faster, we found that higher-accuracy models are not uniformly higher-frequern Instead, using linear interpolation in Figure 4 (Right) we find that higher-

---

accuracy models (ordered left to right by increasing test accuracy) are higher-frequent along between- class paths but lower-frequency along bithin-class paths, perhaps evident of a better class clustering.
4.3 Sensitivity to Natural Image Directions
Label smoothing with radial wave offers convenient global picture of spectral bias, by replacing the radial wave with other noise functions we can use the same methodology to test more localized aspects of spectral bias. We take step in this direction by considering the family of noise functions S(X; k) = a(1 + sin f (X, FR))), where Fx is diagonal Fourier basis image with frequency k and the same dimensions as X, visualized in Figure 5 (Bottom).
Noise Fitting
-0.7
-0.8
-0.9
Image Frequency
k=0
k=2
k=5
k-21
Figure 5: Models are most sensitive to variations low (but nonzero) image frequency. Top: Effective noise fitting of WideResNet32 model with label smoothing of frequency 0.038 in various unit norm directions corresponding to Fourier basis images (Bottom) indexed by image frequency (scaled to [0, 1] for visualization).
We consider (a subset of) Fourier basis images because the Fourier spectra statistics of natural images are well studied (see, e.g. Tolhurst et al. [4]]:: natural images tend to be composed Fourier basis images with amplitude proportional to their inverse spatial frequency. Indeed, in Figure 5 we find that ShakeShake96 is more sensitive to label smoothing in low image frequency directions.
This finding is consistent with theoretical predictions of Basri et al. [2] that models learn faster in regions of higher density of training examples: since low image frequencies are more common in natural images, the effective sampling density enjoyed by noise function is higher in these directions. also possible that this bias is a byproduct of the lower Fourier magnitudes of high frequency components in natural images [36], or is inherent to the convolutional model architecture regardless multiple effects is at play. Determining the precise cause of this image frequency finding is an interesting direction for further study.
of the data distribution (3 or that a combination
4.4 Spectral Bias and Training Data
In our CIFAR- 10 interpolation experiments, we consistently observed that variations in training that improve accuracy create greater disparity between the frequencies of within-class and between-class paths, with lower frequencies within-class and higher frequencies between-class. This trend is evident Figure Rightt) as a model trains for more epochs and in Figure (Right) as the model architecture improves. In Figure C we find that the same trend also holds as we train on more data (Left), apply more effective data augmentation (Center), or apply self deff-disillation (Right).
Training Set Size. As shown in Figure (Lef)) on WideResNet160, find that increasing the number of training examples increases the difference in frequency content between within-class interpolating paths and between-class interpolating paths. Training with more data makes our CIFAR- 10 models behave lower-frequency within each class but fiigher-frquuncyy between classes, perhaps indicating that the models are learning better clustering of the classes they are provided with additional data.
Data Augmentation. Data augmentation is common strategy to increase the effective training set size without the expense of actually collecting additional examples. In Figure (Center))) Center) we consider
the effects of Mixup (with strength 0.1) 49]. AutoAugment 66, and RandAugment [5], each of
7

---

High Frequencyy Fraction of the Learned Function
High Frequency Frection
Leemed Junzion
High Frequency frection the Lzemed Function
High Frequency Fraction of Leamed Function
02:
0.2
0.2
01
Training
Size
Augmentation
Strength
Self-Disillation
Withn- Paths
Between Detween-Class Paths
Figure 6: nigher-accurage models are lower-frequency within-class and higher-frequency between-class. Test accuracy increases from left to right in each figure as we train WideResNet160 with more data (Left), better data augmentation (Center Left), and self-distillation (Right); in each case the frequency separation between within-class and netween-class paths increases as accuracy improves. We also find that varying the Mixup strength (Center Right) within roughly the range 0.1 to 0.4 recommended by the original authors [9] tends to increase this frequency separation, but using stronger Mixup has the opposite effect.
which improves the final accuracy of the trained model (with RandAugment producing the most substantial benefits in our experiments, followed by AutoAugment and then Mixup). RandAugment and Autonug generate new images by applying geometric and lighting transformations, whereas Mixup linearly interpolates between pairs of existing images. Much like our experiments with training set size, we find that augmentations that improve accuracy more also produce greater frequency separation between within-class paths and between-class paths.
Mixup Strength. Mixup augmentation 4 perturbs each batch of training data by randomly pair- ing the examples and perturbing each example towards its partner by interpolation amount À drawn from a symmetric beta distribution (with the same À used for all examples in the batch). We refer to the parameter of this beta distribution the Mixup strength, as it controls the degree to which the augmented images tend to lie close to an original training image or close to the average of two training images. A parameter of 0 corresponds images), a parameter of 1 corresponds to the uniform distribution over À E 1], and a parameter of oo corresponds to À 0.5, the exact midpoint between pair of training images. Figure [6 (Center) compares a range of Mixup strengths and finds that relatively low values that only slightly perturb the original images tend produce greater frequency separation (and higher test accuracy), whereas per- turbing the images too much with stronger Mixup has the opposite effect on both frequency separation and accuracy. Of note, the range of Mixup strengths that increase or do not affect frequency separation in our experiments are well aligned with those recommended by the original authors 49 0.1 to 0.4.
augmentation (A
À 1, always using the original
Self-Distillation. Finally, we consider the effect of relf-distillation on the frequency content of the learned function. In seff-distillation, "teacher" model is trained with some form of strong regularization, in our case combination of weight decay and early stopping based on training loss. A "student" model with the same architecture is then trained from scratch to fit the pseudolabels produced by the teacher Prior research []]] found that this procedure can train student models that outperform both their teachers and baseline model trained as normal.
Prior research has also sought to understand the mechanism behind eeff-distillation. Mobahi al. finds that self-distillation acts as regularizer by limiting the basis functions available to the student learn. We complement their theoretical work with our interpolation experimental methodology in Figure 5 (Right), where indeed we find that the student model learns a greater frequency separation between within-class and between-class paths than its teacher, while also achieving higher test accuracy. We conjecture that this effect is analogous to low-pass prefiltering common in digital signal processing: a high-frequent
noise function that we cannot adequately sample first smoothed (in this case by being approximated by regularized teacher) and then it can be modeled via samples without further loss in fidelity. Without this prefiltering, our samples are inadequate to capture the complexity of the noise function, reconstruct imperfect version corrupted by aliasing.

---

Results on ImageNet
Finally, we apply linear interpolation methodology study the frequency content of range of pretrained models on ImageNet [8] Here we show results on ResNet50; results on all ten models we tested ImageNet are included in Section A.4
5.1 Training Time
As we observed for CIFAR-10 10, our ImageNet models tend increase the frequency gap between the two types of interpolating paths as training proceeds, with lower-frequenee within-class and higher- frequencies hetween-class. We visualize this trend for ResNet50 Figure (Left). For our CIFAR-10 models, this increase in frequency difference between path types was achieved by combination of increasing the average high frequency content on between-class paths and decreasing the average high frequency content on within-class paths. On ImageNet, we again find that average frequency content on between-class paths increases during training, but that within-class frequencies sometimes decrease during training (e.g. for ResNets) but sometimes increase (slower than between-class paths) during training (e.g. for CoATNets). We speculate that the trend decreasing within-claa frequency training time progresses is less consistent ImageNet than CIFAR- 10 because, explore in the next section, ImageNet classes are more internally diverse than CIFAR- 10 classes.
5.2 Class Coherence
In addition to containing 100x as many classes as CIFAR-I0, ImageNet classes can also be more internally diverse. This notion of within-class "consistency" is captured by C-Scores [8. which assign to each training example value between O and denoting how typical that example is. One way of capturing the internal consistency diversity of a class by studying the distribution of its C-Scores. In Figure 7 (Right) we plot the mean and standard deviation of these scores for each of the 1000 ImageNet classes in gray and highlight the 10 classes whose ResNet50 interpolating paths between validation images are most high-frequenc, (red stars) and least high-freas figh-freyuency (blue squares). We find that ResNet50 lower-frequency on more internally consistent classes and higher-frequent on more diverse classes, supporting the intuition that more diverse classes require more complex functions.
Stoev C-Score (per class)
High Frequency Fraction of the Learned Funczion
0.40
0.08
0.35
0.06
0.30
DD4
0.20
0.02
0.15
0.00 15k
20k
25k
28k
0.3 0.4 U.S U.b 0./ 0. 0.9 1.0 Mean C-Score (per class)
Epoch
Figure 7: Within vs. between-class frequency gap widens during training on ImageNet, and in- ternally diverse classes have higher within-class function frequencies. Left: The frequency separa- tion between within-class and between-class paths for ResNet50 grows during ImageNet training. Right: Highlighted points show the 10 classes for which ResNet50 is most (red stars) and least (blue squares) high-frequency Gray points show the distribution of C-Scores among 1000 ImageNet classes.
Conclusions
In this paper, introduced methods to measure spectral bias in modern image classification neural networks, and applied these methods towards the central question:
What kinds of function frequencies needed for modern neural networks to generalize?
Specifically, we applied these methods to examine the impact of variety of training choices on learned frequencies. On CIFAR-10, we found that higher-accuracy models typically have greater frequency separation between within-class and between-class paths, with lower frequencies within- class and higher frequencies between-class. This trend holds regardless of whether high accuracy

---

is achieved through longer training, choice of model architecture, increase in dataset size, data augmentation, or self-distillation. On ImageNet, we again found that this frequency separation increases during training, and noted that models are higher-frequency within more internally diverse classes. Our experimental methods offer window onto the frequency structure of neural networks for image classification and its relationship to model performance. In particular, although our primary goal is to improve our understanding of how neural networks generalize, our work is likely to have practical implications for improving generalization as well as robustness image perturbations and distribution shifts. We hope that future work realizes these implications by applying our methods to study the spectral bias of robust models, and to induce model robustness directly, perhaps by incorporating our metrics as regularizers during training. Future work may also benefit from extending methodology to interpolating paths that adhere more closely to the hypothesized manifold of natural images, such as those produced by generative networks
Reproducibility Statement
Our CIFAR-10 code is available at https://github http://ggtthub..ccmoooooo.ooo/// .co./googel--eeesaarch for ImageNet we apply the same inter- polation method to pre-cropped images and pretrained model checkpoints. Our work uses the
publicly-available CIFAR-10 23 and ImageNet [8] datasets.
Acknowledgments
Many thanks to Yasaman Bahri, Ekin Dogus Cubuk, Hossein Mobahi, Michael Mozer, Maithra Raghu, Samuel Schoenholz, Jonathon Shlens, and Piotr Teterwak for productive conversations and helpful pointers.
SFK is also funded by NSF GRFP.
References
[1] Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages pages 4763-4772, 2019.
neurips........ wwtuuips.ccppapaeee wecp/app////////e 2019/ Abstract. html
[2] Ronen Basri, Meirav Galun, Amnon Geifman, David W. Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In Proceedings of the 37th International Conference on Machine Leurning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 685-694. PMLR, 2020. URL http::////proccdiggg.... mlr hagp://dearaaaaaaarrrrr/////aaaa html
[3] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 548. PMLR, 2018. URL http::/pproceeding/ mlr htt..pprr/////blliinnnt. html
[4] Zac Cranko, Simon Kornblith, Zhan Shi, and Richard Nock. Lipschitz networks and distribu- tional robustness. CORR, abs/1 abs/1809.00 129, 2018. htRlpttp::/ http::/tp:::iiii httpp::://.aii.ooo///bb///////// httg/bbb/18099 129
[5] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space. ence on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pages 3008-3017. Computer Vision Foundation / IEEE, 2020. URL htttps://opeaaccesss thecvf com/ Randaugment Practical Camdauggent_PPactt.aa__Attooaa
2020 IEEE/CVF Confer-
doi: 10. 10.10090CVPR5.0000.222222 2039.00399 content, Centent_CVPRRW.... 2020 11100/html/////m http:/httm/////bbuubkk http:///on.uuuuumaaaanneenennt Augmentation, With a_ Seduche_SSearch.. SaacchhhhSppeeeeeeeeeeeeeeeeeeee CVPRW CWWRW_202O000,CC2222222222202220 _0___222000_____00
DData
20000000pppp.___1111 html
10

---

[6] Ekin Dogus Cubuk, Barret Zoph. Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Au- toaugment: Learning augmentation policies from data. CoRR, abs/1805.09501, 2018, URL
http://arxiv> http://axvv.v.o httpi//.rrgabbb///eeeee 1805. 1801099011000011
[7]
Zihang Dai, Hanxiao Liu, Quoe V. Le, and Mingxing Tan. Coatnet: Marrying convolution and at- tention for all data sizes. Marc' Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, De- cember 6-14, 2021, virtual, pages 3965-3977, 2021. URL Utt/tttss//prrreeeddngg .nuggipe
wwtere--btttttttttcac html
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[9] Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. JEEE Signal Process. Mag. (9()): 141-142, 2012. doi: 10.1 10.11090MMP:200222111777 URL. https://doi http://oi.rgg///// 10. 1109/ MSP 2012. 2211477
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL URL /ttp::tt/////eeeeeieie net/forunii id- -idiidd--yibTTTT
[11] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In Jennifer G. Dy and Andreas Krause, edi- tors, Proceedings of the 35th International Conference Stockholmsmmsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- chine Learning Research, pages 1602-1611. PMLR, 2018. URL Utthhttp///rooeeednggs
Machine Learning. ICML 2018,
Fress/v800ffrraaeelloo html
[12] Xavier Gastaldi. Shake-shake regularization. CoRR, abs/1705.07485, 2017. URL http:
07485
[13]
Suriya Gunasekar, Jason D. Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüs- san, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1827-1836. PMLR, 2018. URL http://prooceedingg mlr. wol...pres//vee press/v80/ Junasekar18a.. html
[14] Suriya Gunasekar, Jason D. Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gra- dient descent on linear convolutional networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada. pages 9482- 1491, 2018 URL 0e98aeeb54acf61 httpeaaaaee62220eee4442222222222
nwwrine.rnccmm ccc htcc//appe//0000 happer/201//hsh/ Abstract.
[15] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning. ICML New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages JoLR.org, 2016. URL http::/prrceediinn
mlr pwwss/ves33 2a////hard116.1 html
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10. 10111/999..226.....
URL https::/:////// //op::://doi. org/10. 1109//VVR..... 2016. .00
[17] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference
11

---

on Neural Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA. pages 1731-1741, 2017. URL https://prooceeddngggs https//rr..rpsspppp///////a/aaaa cctpppper//01///aaa. 1580fff2260008455fff111188888881
[18] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Characterizing structural regularities of labeled data in overparameterized models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18- 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5034-5044. PMLR, 2021.
http://inwnnn.....eneeeeeeeeeeee prwss/v139) / html
[19] Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. CoRR, abs/ 2017. URL http::/arxiv..orrrr http:///aaravv.orraaa////1/11 11561
[20] Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang. Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of in- creasing complexity. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' A'Achéé-uuc Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3491-3501, 2019. URL https: http://///oceee....//.o b432f34c5a997
Abstract. nwwrinuuii... httpp//nwnnneippppppprrrr//////e 2019/hash/
[21] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten. Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of stylegan. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8107-
8116. Computer Vision Foundation/ IEEE, 2020. doi: 10. 10.11////2222000000000.00
URL
https://openacceess: thecvf com wwwtenttttttcccccoccoccc Cww.oeeett_cReeee_ot 2020/htm1/Karraa http///all/krrrr_aaaaaaiiii Analyzing- and wwwprmmingg.....mm. the
of CVPR 10020_paper.htte htmm
[22] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreviex www./eeennnneeeeee http:oorrrmmddloooylylyye
[23] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- (canadian institute for advanced research). http::/t:///w/r
toronto eoronto.edu/"Mttt........ wwwpeddduudciiiieeeee .htll
[24] Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In Ma- rina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Pro- ceedings of Machine Learning Research, pages 6212-6222. PMLR, 2021. URL http: //proceedings. mlr paress/v139/1eeiii
html
[25] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representattinns ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, net/
Conference Track Proceedings. OpenReview.net, 2018. URL
forum?id-rJ2IBFZ2.
[26] Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-BBuc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9903- 2019. URL http:: httttp::/pprceeeingg. neur www. pttpc/pppp///////aaaaaae 45227789222e88877442222222666663 63312213112666333333333333333333 wtt.aeeaa-333tttaattttto Abstract. html
[27] Christian X. Ali Mehmeti-Gööe Göpel, David Hartmann, and Michael Wand. Ringing relus: Har- monic distortion analysis of nonlinear feedforward networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021. URL //tttss:://ppmmme http:::///ppeeeeeeeeeeeee wwwpeeewwwwwwweeeeeeeeeeee teet/forummid=-Tatv--gl///
[28] Hossein Mobahi, Mehrdad Farajtabar, and Peter L.. Bartlett. Self-distillation amplifies regulariza- tion hilbert space. In Hugo Larohelle,, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin, editors, Advances in Neural Information Processing Systems 33:
12

---

Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL
[29]
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias:
On the role of implicit regularization in deep learning. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL lattp://aree //rxiv. www..org/aaa arg/abs/ 1412.6614
[30] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Explor- ing generalization in deep learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, edi- tors, Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 5947-5956, 2017. URL httpp::/pproceedinnggee//// neur wps.rps.ccom wwtpc//ppp///////////// (00e@aaaeee000771e22333333333333 wtt.eeeatatttaaaataaatttt...aaea html
[31] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of over-parametrization in generalization of neural networks. CoRR, abs/1805. 12076, 2018. URL http::////axxx org/ hetg/bbs/8800 1805/ 12076
[32] Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfoolii and Pascal Frossard. Hold me tight! influence of discriminative features deep network bound- aries. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 12. 2020, virtual, 2020. URL 1a997788666645000111666442271123
www.neiiss.cpppaee//////o.. Awttrrtttttttttttca.m html
[33] Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Neural anisotropy directions. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL/httpss:/pproceeiii wtwpeeeeeddinnnnenenei..eee neurips cc/paper/2020 hace/////hhh//ccce cf cft.:fff0f2faamcmmm htt.://///eeeaa/aaaaaaaaaaaaaaa. Abstrattt
[34] Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guar- antees for neural networks via harnessing the low-rank structure of the jacobian. CoRR, abs/1906.053922, 2019. URI. http:///t/// org
[35] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages //proceedings. mlr. www..ppees//... prees/v99/// maaess/v9//raaaaaan www.aaaaaaaaaa.a. html
PMLR, 2019. http
[36] Katja Schwarz, Yiyi Liao, and Andreas Geiger. On the frequency bias of generative mod- els. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 18 116-18813 2021. URL https: //poceeddiggg neur cc. httpe//2222//aass/955555/22 11tp://.55..1.111161111111111111 www.:efffee44222eeeee Abstract. tattsttratt.tttllll
19504ff
[37] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information
Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020,
NeurIPS 2020, December 6- 2020, virtual, 2020. URL/attss:/prroeeeiii wwwpiiinnnnn.uuunnnee neurip html
hash http:////e.//0002222222222222220
[38] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Mach. Learn. Res., 19:70:1-70:57, 2018.
URL htttp://jmm org wwwpp//aggg//aaeeeeeeee. wttpppppp////1..... 11111999111188111
html
13

---

[39] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, URL.
2016, pages 2826. IEEE Computer Society, 2016. doi: 10.1 N.1109/CVP..2016..0
https://doi. 1ttp///1.119//C/PP/ 1109/CVPR. 2016.308
[40] Piotr Teterwak, Chiyuan Zhang, Dilip Krishnan, and Michael C. Mozer. Understanding invari- ance via feedforward inversion of discriminatively trained classifiers. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pages 10225-10235 PMLR, 2021. URL http://t///rr mlr wttpaag........ee/////eea press/v139/ tetervak21a html
[41] DJ Tolhurst, Y_ Tadmor, and Tang Chao. Amplitude spectra of natural images. Ophthalmic and Physiological Optics, 12(2):229-232) 1992.
[42] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 2021. URL http:/ pro.eedinge html
mlr. Freess/v139/touvrorn
[43] Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4148-4158, 2017. URL Abtrr44rr..aabbba...aatttaat... html
.eerinnerriie meurips... . cc/ wwcc//aper/2e/ 2017/hash/
[44] Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In Tom Gedeon, Kok Wai Wong, and Minho Lee, editors, Neural Information Processing- 26th International Conference, ICONIP 20/9, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part I, volume 11953 of Lecture Notes in Computer Science, pages 264-274. Springer, 2019. doi: 10. (1077/978-3-03--670888 //doi.org/10. 1 /978-377-33-11 3- 36708-4-77----2-
URL https
[45] Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regulariza- tion for deep residual learning. IEEE Access, 7:186 26-186136 16,6 2019. doi: 109//CESS
1199/ACESSS URL hhttps://ooiii htt.//oi.oog//// org/10 10. 1 1109/ ACCESS. 2019 2960566
[46] Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 14, 2019, Vancouver, BC, Canada, pages 13255- 2019. URL pttttp:://proceeddngg www.ppooooee.........no.o.....p. neur ct.ppperrrcpppp
[47] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URL
http://www. www.. org httg://vc////// 2016 http://///221//ee/e/eee/ http:///ppppr//pppee/peee// http://www.g.//////////////////a
/inweddxx..//i//oo/ee html
[48] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, 2021. doi: 10/3444/34 htLpstss:: //doi www.oogg//11 0ee.e1144344445
[49] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Cunada, April 30- May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018 URL atttps://ppereett
net met//Eorummid-r1ddd1-111
[50] Xiao Zhang, Haoyi Xiong, and Dongrui Wu. Rethink the connections among generalization, memorization, and the spectral bias of dnns. In Zhi-Hua Zhou, editor, Proceedings of the
14

---

Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3392-33223 3398. ijcai.org, 2021. doi: 10.24963/jcai.. 2021/467. URLhttpps / //doi//ooio. org wwwp.//...............oo
2 2021/467
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations of the label smoothing methodology are discussed at the end of Section 3.1 Limitations of the interpolation
methodology are discussed in Section A.2
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] Code is provided GitHub.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Some of these details are provided in the appendix, and the rest evident from the code.
(c) Did you report error bars (e.g.. with respect to the random seed after running exper- iments multiple times)? [Yes] Standard errors are reported for linear interpolation experiments.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs. internal cluster, or cloud provider)? [No] These results did require substantial GPU compute, but we do not quantify the exact amount. Each point in a label smoothing figure required training model from scratch, and each value in interpolation figure required evaluating a pretrained model on hundreds of thousands of images.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators of CIFAR-10 2 and ImageNet g the image datasets use, well the creators
of the models we use [4 12
16 1O,
42 7
(b) Did you mention the license of the assets? [No] The licenses of each dataset and model
we use are available on their websites.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "vOvqANCQ5UV", "reviewer": "Reviewer_KubS", "summary": "This paper investigates the generalization of neural networks by investigating the spectra of the learned functions. While existing literature focuses on simple architectures like 2-layer MLPs, this work studies spectral bias in modern image classification networks. Two methodologies/metrics are developed for analyzing the spectral bias: i) label noise to enable measuring spectral bias in multi-class classification via label smoothing and ii) measuring the smoothness of a learned function via linear interpolation between test examples.The key finding is that high-accuracy models learn a function that is high frequency in regions between different image classes but low-frequency within each class. This finding is demonstrated to be valid under different settings, e.g., for different model architectures and training strategies. ", "questions": "- It seems that noise fitting happens when the validation loss starts to plateau (Figure 1). A simple baseline would hence be to simply determine the noise fitting point via the slope of the validation loss. It would be interesting to see if this is an equally good measure to justify the complexity of perturbing labels and training models from scratch.\n- Why does sampling along paths between images rather than in regions surrounding each image allow for a more global measurement of spectral bias (L.93)? Could the methodology by Zhang et al. also be used as a metric in this setting? If not, please explain why?\n- L.171 why is the distance between adjacent images kept constant as opposed to the number of samples?\n- How do the results from section 4.2 relate to the results for MLPs of Rahaman et al (On the spectral bias of neural networks)?\n- The findings of Basri et al., that the learned function will be higher frequency in regions with denser samples (L.80), seem to contradict the finding of this work that the learned function of high-accuracy models is low-frequency within each class. Could you clarify how these findings can be aligned?\n- Could you give an intuition for the formula for label smoothing in L.129?\n", "limitations": "The limitations were addressed adequately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is overall well-written and easy to follow. \n- The proposed metrics/tools for analyzing spectral bias in more complex classification architectures are well-motivated and ablated. While the proposed metrics appear not completely novel (see weaknesses), applying these metrics to quantify spectral bias in neural networks is a valuable contribution and might be useful tools for future works in this research area.\n- Generally, I found the paper insightful and interesting to read.\n\n\nWeaknesses:\n- The paper aims to explain the generalization ability of neural networks by investigating their spectral bias. However, there are no experiments in the paper that actually demonstrate that the proposed metrics are indeed linked to better generalization of the network (i.e. there is no metric to report generalization ability in the experiments).\\\n- The proposed label smoothing appears very similar to the label-smoothing regularization proposed in [37] and it is not clear enough how it differs.\n- The proposed linear interpolation approach appears related to the Perceptual Path Length (PPL) regularization proposed in [1]. Please add this to the related work and discuss how PPL differs from the proposed linear interpolation.\n- The terminology of ‘noise’ and ‘smoothing’ is confusing. E.g, L.136 the *noisy* validation loss corresponding to the *smoothed* labels., L.179 “label smoothing noise”. L.181, increasing the ‘label smoothing frequency’ corresponds to adding higher frequencies, i.e. noise.\n- The key insight, that the learned function is high-frequency between classes and lower frequency within classes, is not very surprising considering that it is measured by linearly interpolating between samples of different, randomly chosen classes. For randomly chosen classes, a path very likely traverses multiple class boundaries which would explain higher frequencies. I suggest rather formulating this more conservatively, i.e. as a ‘proof of concept’ by confirming an intuitional result, as opposed to ‘a key insight’. Nonetheless, the proposed tools allow to quantify and support this intuition.\n\nMissing related work:\n- [1] for PPL, mentioned above.\n- Model sensitivity to image frequency: [2] finds that discriminative models are not sensitive to the frequency range but rather to the magnitude of the frequencies. \nPlease also consider adding a brief discussion on how their finding aligns with your findings on image frequency. Sensitivity to magnitude might also provide another explanation for the higher sensitivity to natural image directions (4.3). \n\n[1] T. Karrs, S. Laine, M. Aittala, J. Hellsten, J. Lethinen, T. Aila. Analyzing and Improving the Image Quality of StyleGAN. CVPR 2020\n\n[2] K. Schwarz, Y. Liao, A. Geiger. On the Frequency Bias of Generative Models. Neurips 2021\n\n\nMisc:\n- In many references the journal/conference is missing.\n- Consider renaming S from ‘target function’ to ‘noise function’ to i) be consistent with Rahaman et al and ii) make the name more intuitive.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nrMKJSGobCc", "reviewer": "Reviewer_ap1u", "summary": "This work seeks to evaluate how training design decisions affects the spectral bias of neural networks. Specifically, they first extend the label noise procedure of Rahaman et. al to the multi-class setting and analyze when the deep learning model fits the noise. They also propose a second method which uses linear interpolation between two images, and use this to distinguish between the spectral content for \"within classes\" or \"between classes\". The authors consider results on CIFAR10 and on ImageNet, and consider factors such as accuracy, model size, training set size, and data augmentation. ", "questions": "- Are there better ways of interpolating between two images (for example, optimal transport methods) rather than simply in pixel space. I could imagine that interpolating in a naive way could cause images to be relatively out of distribution.\n\n- What do the spectral biases look like for non-convolutional networks (such as ViTs). Do they look similar to CNNs?\n\n- In Section 5.2, you found a difference in spectral bias for diverse vs homogeneous classes? Do you see any differences between \"hard\" (i.e., low confidence) or easy examples. ", "limitations": "The authors discuss limitations of the label smoothing method in Section 3.1. It would be useful to move up the discussion of the interpolation method from appendix A.2. The authors do not discuss any negative societal impacts of their work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n*Quality*: I thought that the experiments were quite thorough, and was particularly impressed that they covered so many design decisions (such as data augmentation, training set size, etc). \n\n*Clarity*: I thought the paper was relatively easy to follow. \n\n*Originality/Significance*: This paper seems like an important step for making analysis of spectral biases practical. The paper is thus relevant to the community.\n\n## Weaknesses\n\n*Significance:* Please include some intuition on what these spectral biases translate to for real applications. What kind of spectral bias is desirable (and why)? Should we prioritize design decisions that are higher frequency between classes and lower frequency within classes (or does this just happen to be true for well-generalizing models)?  A final impact section, discussing what the concrete implications of this work are, would be helpful. \n\n*Clarity*: The section on the relationship between the interpolation and label smoothing classes could be more clearly written. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gbBa_PXmQi8", "reviewer": "Reviewer_7SRK", "summary": "This paper presents a study of trained neural networks in the frequency domain. Two methodological contributions are proposed:\n 1) an extension to multiclass classification of the technique in Rahaman et al for probing the ability of a model to learn signals of varying frequency \n 2) a method for estimating the Fourier components of trained networks along linear interpolation paths between examples\nThe empirical contributions of this paper are:\n 3) the spectral bias also applies to CIFAR10/Imagenet ResNets (whereas in Rahaman et al. only simpler models were studied)\n 4) trained networks are on average lower frequency on within-class paths than on between-class paths\n     1) higher disparity between within-class and between-class examples correlates with better generalization performance\n     2) a correlation between more diverse classes (measured by examples with low mean C-scores) and high within-class frequency and vice versa.\n 5) a relationship between image frequency and function frequency where are low image frequency components are learned faster than high image frequency components", "questions": "**Technical flow in 1/** (l129): \"retain a valid probability distribution\". Assume that S(X) = -1 (as can be the case when using the sine function mentioned l146), then for the true class, \\bar{y} = 2-1/M is greater than 1, which does not define a valid probability distribution. How are you then able to train a model using these labels ? Are you using the cross-entropy loss or any other loss function?\n\n**Technical flaw in 2/ thus 4/, 4.1/ and 4/2** (figure 3 right) If I understand the experiment correctly then within class I want to learn a function y(lambda=0) = y(lambda=1) = onehot(true class) and between class a function y(lambda=0) = onehot(class of example 0) \\neq y(lambda=1) = onehot(class of example 1). Then I am not surprised that within class the Fourier coefficients are equally spread (in fact, they probably converge to 0 as you increase the number of examples), whereas between class you basically learn a function that grows continuously from 0 to 1. As you average over many examples, I would not be surprised if this resembled a sigmoid function. Can you maybe directly plot the (averaged) function ? If I am correct this then renders your findings somewhat trivial, but I might be proven wrong.\n\nWhat exactly are you plotting in all barplots with low/high frequency components? What threshold are you using for splitting between low and high frequencies?\n\nIn figure 2 left: why is ||y_lambda - y_0|| > 0 when lambda = 0? when lambda = 1 (images come from the same class)?\n\n**Other major comments**\n - l183: Can you make more precise what \"model output\" means? From the context I think it is the probability, but it could also be the logit output.\n - in figure 5, can you motivate the use of frequency=0.038? How does the experiment vary when using other values?\n\n**Minor comments**\n - l170-175: I find the choice of a linear path and l2 (not precised in the text?) distance between images questionable. As a failure case, imagine 2 images sharing the same background B1 (e.g. grass) but with 2 different foreground objects C1 and C2, then the l2 distance between this images will likely be smaller than two images with the same foreground object class C1 but 2 different backgrounds B1 and B2.\n - figure 3 right: can you make precise whether this plot uses training set examples or test set examples?\n", "limitations": "I don't think that such a paper should discuss negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**: This paper is in the line of other papers studying the so-called *spectral bias*: the tendency of neural networks to learn low-frequency functions first. All points listed in the summary are  original as far as I can tell.\n\n**Quality**: The paper lacks some details in experiments and notation, and I think (but I might be proven wrong in the rebuttal) that there are some technical flaws in 1/, 2/ and 4/. \n\n**Clarity**: Due to (potential) technical flaws, and missing details regarding experiments, I found it difficult to really judge on the significance of the experiments and the validity of the claimed results.\n\n**Significance**: I feel that the methodological contributions are only minor improvements upon existing techniques (Rahaman et al for 1/ and Zhang et al. for 2/). Empirical observation 3/ is an extension of the findings of Rahaman et al. which can also be considered a minor improvement. I have the feeling that points 4/, 4.1/ and 4.2/ have a technical flaw/are trivial. I did not fully grasp empirical result 5/ so I am unable to judge of its significance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SZeEsNOrurj", "reviewer": "Reviewer_orks", "summary": "This paper aims to improve how the spectral bias of neural networks is measured. To this end, the authors introduce two techniques: first based on label smoothing, and the second based on probing the network outputs along a linear path between two data samples. The end result is a number of tricks that make spectral bias a more useful diagnostic tool for analyzing neural nets (pre-trained or otherwise), as empirically demonstrated in the experiments. ", "questions": "- Regarding the experiment described in Figure 5: do the authors have an intuition as to what would happen if the analysis were to be performed on earlier layers of the network?\n- Can the technique described in Section 3.2 be related to the Projection-slice Theorem?\n- How exactly is “high-frequency fraction” defined? Is it the sum of the magnitudes of the DFT for f larger than some threshold? If so, what is this threshold?", "limitations": "The authors have been upfront about the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Positive\n+ The use of label smoothing (instead of adding sinusoids to binary targets as done in Rahaman et al) is sound. The purpose of adding sinusoids to the target in Rahaman et al. was to expose the model to spatially correlated noise, which this method gracefully extends to the multiclass setting.\n+ The idea of using the vector V to locally probe spectral bias is neat. I imagine that setting V to the principle component vectors of the data can also be insightful.\n+ The idea of using projections of the function also makes sense, and I appreciate the sanity check in Figure 2 right.\n+ The experiments are well crafted do a good job at teasing out interesting aspects of spectral bias.\n\n-------\n### Not Positive\n+ The presentation can be improved. For instance, it would make for a more pleasant reading if the authors were to clearly (mathematically or pictorially) express what they mean by “noise fitting” and “min noise fitting” (even when the caption of Figure 1 explains what they are). The y-axis of Figure 2 left can also be confusing. In general, I feel the paper would read better if the experiments are bundled with their description (instead of being split in sections 3 and 4), but this is a matter of taste.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept