Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes two methods for estimating smoothness of the prediction function learned by a deep network : 
 1/ an extension of Rahaman et al. to multiclass classification, by adding a sine of desired frequency to the target vector, then training on the new target vectors. The "effective noise fitting" is then the difference between the validation loss on clean targets and the validation loss on modified targets.
 2/ a measure of change of the logits in l2 norm between 2 training examples, averaged over many examples

Using their methods, they show the effect on the smoothness of the predicted function of:
 - varying the number of parameters
 - adding explicit regularization (weight decay/mixup)
 - distillation
main_review: While the topic is certainly relevant and interesting, I think that the proposed methods are a bit hazy and I found some important missing details in the empirical exploration.

### Smoothness estimation methods:

1/ is a straight forward extension of Rahaman et al, applied to multiclass.
 - I don't really understand why you had to introduce "effective noise fitting" instead of just looking at the validation loss as in Rahaman et al?
 - What is the purpose of the 1/k added to the target vector ?
 - How did you choose the range of frequencies. E.g. in the exp in fig 3.a the frequency ranges from 0.035 to 0.04 which seems a rather tiny range compared to the experiments in Rahaman et al where the high frequency is 5x the low frequency
 - How can you tell from fig 3.a e.g. that the high frequencies are learned later than the low frequencies ? I see that the 0.035 yellow curve takes its minimum at around 50 epochs whereas the "high" frequency 0.04 dark blue one takes its minimum at around 30 epochs. Similarly in fig 3.b left where exactly is the minimum for wrn_160 ?
 
2/ is difficult to interpret precisely:
 - The role of delta is not defined, for instance what does it mean that "delta = 1" in section 3.2 ?
 - Why is it called *cumulative* difference norm? What is cumulative in this measurement?
 - In fig 2. if both images of the same class, why is ||y_lambda - y_0|| high at lambda=1 (i.e. exactly the other image)
 - How precisely do you see the smoothness in these plots? In some cases it is rather ambiguous e.g. in fig 6.c bottom which experiment in the range [0.1; 1.6] is the smoothest one?


### Empirical evaluation

 - The paper is lacking experimental details, such as choice of optimizer and hyperparameters. This is especially true as I would expect the learning rate to play an important role in how fast each frequency component is learned.
 - It would be interesting to include an ablation study when varying the learning rate.


You can also take inspiration from the method in [Zhang et al 2021] who study the spectral bias in the context of epoch-wise double descent, also using training examples, but by performing a full Fourier decomposition.

References:
Zhang, Xiong and Wu, Rethink the Connections among Generalization, Memorization, and the Spectral Bias of DNNs, IJCAI-21
summary_of_the_review: Overall, I find the paper in its current state not ready for publication. I think that the methodologies for measuring smoothness can be improved, and the experiments should include experimental details as well as an ablation study of the role of different hyperparameters on the smoothness of the function obtained after training.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents an empirical study of how different training aspects affect the spectral bias of neural networks in practice. To that end, the authors propose to inject label noise of different frequencies to the CIFAR10 dataset and suggest that the time a neural network takes to start overfitting this noise is a good metric of its spectral bias. They also propose to measure the variability of the loss landscape in the linear interpolation path between two images as proxy for spectral bias. Experiments on the effect of model architecture, explicit regularization, and data augmentation suggest that deep neural networks exhibit a strong spectral bias, in practice, which can be modulated by different design factors.
main_review: ## Strengths

1. **Research question**: I find the proposed research question of great interest to the community. The spectral bias of deep learning is mentioned very often as a justification of many practical tricks in the literature, but we are lacking a proper evaluation of this bias in practice.
2. **Analysis of many factors**: I admire the effort made by the authors in studying the effect that many factors of variability have in the spectral bias of SOTA convolutional neural networks. In particular, I find the study of the role of architecture and network capacity in the strength of the spectral bias very compelling.
3. **Connection between directional bias and spectral bias**: The directional inductive bias in the input space and the spectral bias in the function space are easily confused in the literature. In this sense, I find the analysis bridging the gap between these two inductive biases very interesting. In particular, I honestly believe that moving forward, the research community should strive to understand the interplay between the different types inductive biases present in neural networks, so I find that some of the observations in this work are a good step in the right direction.

## Weaknesses

1. **Evaluation metrics**: My main concern with this work is the fact that the choice of metrics used to evaluate the spectral bias of a neural network are highly subjective and difficult to interpret. Specifically, both the effective noise fitting curve and the cumulative difference norm in the linear path require to inspect a noisy curve to measure the spectral bias. This makes the final assessment of the results very qualitative. In my opinion, this seriously diminishes the strength of the presented results, and this has greatly affected my final score.

Nonetheless, I do not think this problem is unsolvable and I am willing to increase my score if this problem is addressed during the rebuttal. In particular, I would highly encourage the authors to look for alternative metrics to quantify the spectral bias, and perform some experiments to validate their qualitative results with a more numerically precise metric. As a suggestion, I propose the following two metrics:

   - **Test accuracy when training to predict sinusoidal labels of different frequency**: One way to precisely quantify the spectral bias would be by training multiple times on the same dataset with target labels of different frequency. In particular, I propose to generate a benchmark consisting on CIFAR10 samples and different target labels $S(X)$, or similar. The test performance obtained when training to predict different frequency targets should be a good indication of the spectral bias of that network.
   - **Roughness of loss landscape wrt the input**: One could use a similar metric as Mehmeti-Gopel et al. 2021 but in the input space to measure the frequency content of the final loss landscape of a trained neural network.
In any case, I encourage the authors to defend their choice of metric during the rebuttal if they believe there is no better metric to capture this phenomenon.

2. **Inconclusive results**: Another source of concern for me has been the fact that some of the presented analysis are rather speculative and not based on facts. In particular, when discussing the role of different types of regularization in the spectral bias, the authors refer multiple times to the connection between their observed spectral bias and the accuracy of the different models: "*This experiment reinforces the nuanced relationship between function complexity and performance...*", "*we find that [..] some smoothness [...] is beneficial, [but] too much causes detrimental underfitting*" or "*Neither complexity nor smoothness is, in itself, purely beneficial or detrimental: the trick is to balance them appropriately for the dataset.*" All these statements, however, are not supported by the data, as nowhere in this manuscript is the spectral bias of the different training runs compared to the performance of these models. Please, note that I am not implying that these statements are not true, I am just merely stating that these are not supported by the results presented in this manuscript. Reaching those conclusions could be possible if one juxtaposed some quantitative metric of the spectral bias to the final performance reached by these models, but I am afraid that with the current qualitative metrics, this is not possible.

## Other comments and additional questions
1. **Missing citations**: I believe some parts of this work could benefit from the inclusion of a few missing citations. In particular, prior to Ortiz-Jimenez et al. 2020, Yin et al. 2019 had also previously studied the robustness of computer vision classifiers to different frequency components. Also, on the implications of the spectral bias, Shah et al. 2020 argued that the simplicity bias was detrimental for robustness. The directional inductive bias presented in Sec. 4.2. was extensively studied in Ortiz-Jimenez et al. 2020. And finally, Mehmeti-Gopel et al. 2021 recently studied the roughness of the loss landscape of neural networks in an empirical campaign analogous to the one in parts of this work, although with respect to the weight space, rather than the input space.
   - Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin Gilmer. "A Fourier Perspective on Model Robustness in Computer Vision". NeurIPS 2019
   - Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain and Praneeth Netrapalli. "The Pitfalls of Simplicity Bias in Neural Networks." NeurIPS 2020
   - Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard. "Neural Anisotropy Directions". NeurIPS 2020
   - Christian H.X. Ali Mehmeti-Göpel, David Hartmann and Michael Wand. "Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks". ICLR 2021

2. **Possible methodological issues in linear interpolation**: I have two questions regarding the experiments measuring complexity of the loss landscape:
   - Why do the authors choose to work with segments of the same size, and do not fix the number of segments to query in the linear path? Do the results change significantly if one changes this element of the evaluation?
   - Doesn't the exclusion of the incorrectly labeled samples bias the final metric towards smoother values? Specifically, if one repeats this experiment without removing those samples, is the order of complexity of the different methods retained?

3. **Choice of architectures**: Could the authors elaborate on the reasons behind their choice of model architectures? Specifically, why did the authors decide to go with shake-shake models and very wide-resnets and did not provide also results for more manageable ResNets?
4. **Role of data distribution**: The authors blame the data distribution for the interplay between directional inductive bias and spectral bias in Sec. 4.2. However, this statement is also not supported by any evidence. In this sense, I wonder if the authors have tested whether the differences in the spectral bias for different directions are not present in isotropically distributed data, e.g., a normal distribution. If so, this statement would be better supported. Nevertheless, note that Ortiz-Jimenez et al. 2020 found that the directional inductive bias was mostly due to the flow of information in the architectures, so I would not expect that to be the case. In general, it would be good if the authors could comment on what do they expect if instead of using Fourier basis functions, they repeated their experiments using Neural Anisotropy Directions, instead.


summary_of_the_review: Although I believe the proposed research question is of great interest to the community, and I admire the effort made by the authors in studying the effect of many training factors in the variability of the spectral bias in CNNs; I think that the overreliance on qualitative metrics and lack of quantitative results makes most of the observations in this work inconclusive. I am open to increase my score if the authors address these issues during the rebuttal, but at this stage a lean towards rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a set of tools to probe spectral bias of deep neural networks, that is their tendency to learn low frequency, simpler functions earlier in training, whereas high frequencies are fit later. Authors propose adding noise to the labels through a target function via label smoothing, where the frequency and direction of the target function in image space can be varied. Moreover, they introduce a linear interpolation technique on validation data to probe the smoothness of the learned function along paths connecting natural images. Authors perform extensive experiments to demonstrate how the proposed tools can be used to investigate the spectral effect of training parameters such as model size and different forms of explicit and implicit regularization.
main_review: Strengths:
Even though similar methodology has been used before (e.g. in Rahaman et al.), the proposed label smoothing technique seems to be novel in being applicable to multi-class classification. The experiments are extensive and well thought-out. The effects of training parameters on spectral bias (or its proxy) is investigated from various different directions. The paper is well-written and easy to follow. Overview of related work is sufficient.

Weaknesses:
It is not clear to me what tangible novel insights we learned about generalization beyond what has already been investigated and observed in the literature (such as Oymak et al. and similar works), namely that over-parameterized networks learn structure from data first, but can overfit to any noise over time. In other words, the paper makes observations on the effect of different regularization techniques on function frequencies but the *role* of function frequency in generalization is not investigated directly.
Moreover, the significance of the work is somewhat limited in my opinion because it is not clear how it can aid training choices. The key conclusion seems to be that 'an ideal function should include high enough frequencies to fit the data but avoid unnecessary high frequencies that can harm generalization', which is somewhat evident and it is not clear to me what consequences it has. For instance, can we use this methodology to inform us how to choose implicit/explicit regularization or network size?

Oymak, S., et al. "Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian." arXiv preprint arXiv:1906.05392 (2019).

Rahaman, N., et al. "On the spectral bias of neural networks." International Conference on Machine Learning. PMLR, 2019.
summary_of_the_review: Overall, I think the paper has merit and the proposed tools have potential to reveal novel insights on generalization properties of deep neural networks. On the other hand, the core novelty and significance of the findings are somewhat limited in my opinion, therefore I recommend marginal acceptance. I am however open to change my score either way after hearing back from the authors and after further discussion with other reviewers.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on the spectral bias of neural networks, i.e., their tendency to learn first the low-frequency information. The goal of this work is to extend the spectral bias into practical image recognition networks, i.e., beyond the fully-connected nets and the NTK regime that it was previously studied. To that end, the paper conducts a number of studies in both explicit and implicit regularization schemes used in practice, and provides links for the success of model distillation. 
main_review: # Strengths:

-  The text is mostly readable (apart from a few parts as denoted below). 
-  The links to model distillation are interesting, but not explored enough. 
-  The motivation is clear.

# Clarification questions: 

-  How is sec. 3.1 (fig. 1) different to classic overfitting? 
-  In the first paragraph of sec. 3.2 the paper mentions ‘grouping of images in pairs’, however it is not mentioned in that paragraph how the grouping is done.
-  What is the ‘effective amount of interpolation’ mentioned in sec. 3.2?
-  One of the most popular networks (and the almost default baseline) is ResNet, so it is surprising that ResNet is not used. In addition, what are the clean accuracy of the networks used? 
-  Why are ablation studies on different regularization schemes (sec. 4.3) conducted in different networks? Why are wide-residual nets preferred for weight decay and shake-shake for the training set size?
-  Why does RandAugment add ‘necessary complexity’? How does RandAugment differ from Mixup or other training schemes that demonstrate low-frequency preference?
-  Frequently, when training image recognition networks on CIFAR10 data augmentation is used, however it is not explicitly mentioned (other than in sec. 4.3). Was there any data augmentation used in sec. 4.1 and 4.2? 
-  In sec. 4.2, is there some intuition for stopping at k=21 (fig.4)? 


# Improvements:

-  The paper includes empirical results and decisions without mentioning any motivation or theoretical grounding for those. For instance, see the comment about different networks in the ablation studies, or why those networks were used over state-of-the-art networks or the popular ResNet. In addition, neither an algorithm or the source code is provided, which makes it harder to assess the decision choices on the code and the plots. 
-  I would recommend sharing further information on the function S (sec. 3.1); sinusoidal might not be ideal for image domain applications, which is again not mentioned or discussed in the paper. 
-  The paper focuses on a single experiment, i.e. image classification on CIFAR10, which makes it hard to appreciate whether it generalizes beyond this setting. For instance, do the same results emerge in self-supervised learning? 
-  In a similar vein to the previous comment, it would be more convincing if the results on different datasets (e.g. STL10, Imagenet) were presented along with additional training algorithms to demonstrate that the observations indeed generalize well. 
-  One implicit assumption of the paper is that CIFAR10 shares the same high-frequency statistics as the datasets used in practice. Do higher-dimensional datasets (e.g. Imagenet, CelebA-HQ, COCO, iNaturalist) share the same statistics and do the properties presented here generalize to those?
-  The sec. 3.2 (which is probably the most critical in the method) could be improved in terms of writing. Currently, a number of questions are not answered. For instance: why do the path lengths vary? What do the different colors in Fig. 2 depict? Is y_lambda (fig. 2) and y_t (sec. 3.2) different? 
-  Is there some theoretical foundation on how the rapid changes in the cumulative difference of the norm (sec. 3.2) are related to the high-frequency components? 


# Minor comments: 

-  The ‘shake-shake’ (sec. 3.1) is mentioned without defining that it is a network or providing any citation; the reader needs to go all the way to the experiments to understand what this refers to. 
-  Lambda is not a formula (sec. 3.2). 
-  The legends are currently broken, e.g. in Fig. 3 they are blocking a critical part of the figure (i.e. > 200 epochs). 
-  Is the lambda mentioned in sec. 4.3 (Mixup strength) the same as in sec. 3.2? 

summary_of_the_review: In my opinion, the spectral bias can be a useful tool in analyzing the inductive bias of networks, but the current format of the manuscript is not ready for acceptance. There are a number of questions on the methodology and how this relates to the spectral bias, that is the main contribution of this work. In addition, the experiments are not considering several cases (e.g. other datasets, or alternative learning algorithms) that might have correlations with their results. Furthermore, previous results about the manifold of the data (ie. that it impacts the frequencies learnt) are not discussed in the paper. If the revised version addresses the improvement points, I can reconsider my rating.
review_title: null

Meta-review: This paper expands the spectral bias, which has been studied in a constrained situation such as the fully-connected network, to a more practical situation of a multi-class classification situation, and proposes a novel technique that can measure the smoothness through linear interpolation of test examples.

Two reviewers highly evaluated the importance of the research question considered in this study and the value of diverse experiments applying the proposed method in various directions, and suggested acceptance. On the other hand, two other reviewers suggested rejection due to the lack of rigor in writing and experiments. I strongly agree with the reviewer's concern that  the method was only verified  on CIFAR10 and the rigor of the experiment was lacking. Unlike the spectral bias paper, which is the basis of this study, this submission is not a theoretical paper, but rather an experimental paper. I admit that it is impossible to verify in various domains as mentioned by the author. However, I believe that verification on more diverse, especially larger-scale datasets is essential at least focusing on the image classification task.
Decision: Reject