Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel spatial frequency regularization technique that improves the robustness of training neural networks against superficial fourier statistics in a dataset.  In the loss function, It adds a regularization term that is based on the Fourier-transformed input-Jacobian.  This term could be customized such that the trained model could ignore (be insensitive to) features with specific frequency in the dataset.   The authors define spatial frequency sensitivity using input-Jacobian in Fourier space. 
Although this paper focuses on datasets with images, the method is extendable to any n-dimensional data.  Empirically, the method is evaluated for its robustness against fourier filtering, corruptions, and image patches shuffling in CIFAR10 & CIFAR100 datasets.  It shows better results than other baselines in maintaining the classification accuracy of a model against fourier filtering and image patches shuffling.  There do seem to be slight performance gaps from the SOTA AugMix method in image corruptions.  However, the method is considered to be simpler than AugMix and still outperforms other baselines in many cases.   Experiments demonstrate that the proposed method could learn global features present in a dataset.
main_review: Paper strengths: 
Compared with training on fourier-filtered data or applying complex data augmentations, the proposed method is a simpler (and principled) approach to prevent neural networks to bias towards superficial features of the training data.  The motivation and the method sections are well written.  These experiments indicate its effectiveness in learning global structures of the data.  The results in the robustness against the fourier filtering are convincing. 

My concerns:
My major concern is about the significance of the experimental results for data corruptions.  This looks to be the main experiment section and AugMix [1] outperforms standard training in all cases but the proposed method does not.  I would agree that the proposed method could be simpler and extendable to other domains but additional experiments were not provided. (only cifar10 & cifar100)
In addition, I’m not sure about the significance of outperforming “medium/high pass filtered” methods when there are a few data corruption cases the standard training (lambda=0) outperforms the regularized training. (6/24 cases with LSF in CIFAR100, for example).  While the regularization term is present, the original cross entropy loss still operates on raw images so the model might still be picking up high frequency local features under LSF regularization.  The sensitivity to this weighting hyperparameter between the two loss terms is not discussed.    If I’m not mistaken, baseline methods that are trained on filtered data cannot take advantage of those local features.    

A minor note: 
I suspect AugMix [1] won’t perform as well as SFS in the patch-shuffle experiment but it was not provided.  Is AugMix not applicable in this setting?

[1] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. arXiv:1912.02781 [cs, stat], February 2020. URL http://arxiv.org/abs/1912. 02781. arXiv: 1912.02781.
summary_of_the_review: As the extendibility to any n-dimensional data is its big advantage over those complex data augmentation methods, I am interested in seeing some results in other domains.  Given how well AugMix performs against the standard training procedure in data corruptions, I may increase my score if additional experiments are provided to reflect its ease of use/extendibility over methods such as AugMix.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a measure of “spatial frequency sensitivity”, where a model’s input-jacobian is fourier-transformed and frequency bands are aggregated to define how sensitive a trained model is to specific fourier frequencies. The authors show this can also be used to explicitly encourage specific frequency-sensitivities, which they call “spatial frequency regularization”.

main_review: Major comments

The paper claims that “networks trained with our proposed regularizers obtain
significantly improved classification accuracy while maintaining high accuracy on
in-distribution clean test images.” However, all the regularization methods presented have a clean-accuracy drop of >=7.3% which is very significant, especially on a small dataset like CIFAR. This makes it hard to believe the papers claims that its method is competitive to sota-trained methods (it performs worse than AugMix, for instance). Even training with gaussian augmentation can similarly-high robustness if allowed to perform such a huge IID accuracy drop [2].

Additionally, the paper mainly tests on small datasets (CIFAR-10) and only uses Imagenet when measuring model sensitivity (never training with the regularizer). Even in this constrained setting, Fig 1 does not show gaussian-trained [1] (or patch gaussian [2] trained) network for imagenet. This lowers my confidence that the method has practical applicability.

This would not be a concern if the method was a useful tool for drawing even more insights into how models work, but this is not fleshed out in the experiments.

[1] https://papers.nips.cc/paper/2019/file/b05b57f6add810d3b7490866d74c0053-Paper.pdf
[2] https://arxiv.org/abs/1906.02611

Minor comments

It would be useful if the authors could validate the sensitivity-measuring methodology by checking that models that have been Low/Medium/High-pass filtered indeed display the sensitivities being measured.

summary_of_the_review: The usage of input-jacobian-fourier (which is highest input sensitivity) to measure spatial frequency sensitivity is intuitive and well explained. However, the paper neither justifies its practical applicability, nor does it develop the methods into a fleshed out model analysis to yield insightful conclusions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is motivated by the challenge of the model's uncontrolled behavior in learning different frequency components of the image data and introduces a new regularization to improve the empirical performances. The new idea, especially the discussion where chain rule can be directly applied, is very interesting. 
main_review: **Strengths**

This paper presents several interesting discussions, in particular, I think the idea of chain rule can be directly applied to the input-Jacobian is a very neat observation. 


**Weakness**

On the other hand, I think the paper can be improved on several perspectives, mostly related to how the contents can be presented and discussed. 

  - Most importantly, the empirical results are presented in a non-standard way. The bold numbers are used to report performances higher than baseline in this paper, while bold numbers are mostly used to denote the highest performances. 
     
    - By reading the numbers, I feel like the new method introduced by this method actually shows a fair strength in comparison to other methods except for AugMix. In this case, the authors might be able to argue that their method is good enough since AugMix is a very computation-heavy method specially designed on the benchmarks used in the paper (with evidence that AugMix is not that good for Table 1), but the authors seem to prefer to present the paper in a way that they seemingly want to compare to AugMix where AugMix is designed with their own method inspired by something else. As the result, the empirical results seem weak. 

 - The presentation of the main idea can also be improved. For example, some illustrations on how P corresponds to the elements in Figure 1 could help understand the paper better. 


summary_of_the_review: This paper introduces a new idea and demonstrates the empirical strength of the method, while the method has some empirical virtue, the way the results are presented makes it very hard to appreciate the results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a measure for a model’s spatial frequency sensitivity (SFS) based on the input-Jacobian in the Fourier basis. With this measure, the authors observe standard CNN training biases towards certain particular spatial frequencies consistently across samples. Based on this measure, the authors propose a family of spatial frequency regularization techniques to suppress the model’s sensitivities to certain spatial frequencies. 

main_review: Strength 

1) This paper is clearly written and easy to follow.
2) The proposed SFS regularizer is clearly motivated. It is more interpretable than existing sensitivity regularization techniques (Section 2.1). It is also generic to be applied to any differentiable model. 
3) The experiments show that the proposed SFS regularizer is robust to Fourier filtering and corruptions, while maintains high accuracy on clean images.

Weakness

1) The experiment datasets and models look insufficient to me. The authors only show results on CIFAR10/100, but other baseline methods (e.g., AugMix) show at least some ImageNet results. 
2) The presented results are bit unconvincing to conclude the method’s effectiveness. In Table 2 and 3, the SFS regularization methods only match with the performance of the “Low-pass filtered" method, and their performances highly correlate through all types of corruptions. Although the authors point out their technical difference (Section 4.6), I still doubt the advantage of SFS regularization over the simple "Low-pass filtered" method regarding the current performance.

Questions

1) What is the computational cost compared with other baseline methods, considering the regularization term involves Jacobian?
summary_of_the_review: The proposed SFS measures are clearly motivated, and provide interesting insights from the frequency-perspective on how existing methods improves model’s robustness. My major concern lies in the insufficient experiments and unconvincing results as stated in the weakness section.

**Disclaimer**: I am unfamiliar with robustness in CV, it's possible I have some misunderstandings. My rating is highly subject to change based on other reviewers’ comments.
review_title: null

Meta-review: The reviewers all generally appreciated the idea in the paper. However, the nature of this contribution necessitates an empirical evaluation, and the reviewers generally found this to not be sufficient convincing. My assessment is that this idea can likely result in a successful publication, but will require additional empirical evaluation and analysis as suggested by reviewers. While the authors did add some additional results during the response period, they do not seem to be sufficient to fully address reviewer concerns.
Decision: Reject