Spatial Frequency Sensitivity Regularization for Robustness
Authors: Kiran Chari
Chuan-Sheng Foo
See-Kiong Ng
Venue: ICLR 2022 Submitted (2022)

Abstract: The ability to generalize to out-of-distribution data is a major challenge for modern deep neural networks. Recent work has shown that deep neural networks latch on to superficial Fourier statistics of the training data and fail to generalize when these statistics change, such as when images are subject to common corruptions. In this paper, we study the frequency characteristics of deep neural networks in order to improve their robustness. We first propose a general measure of a model's $\textit{\textbf{spatial frequency sensitivity}}$ based on its input-Jacobian represented in the Fourier-basis. When applied to deep neural networks, we find that standard minibatch training consistently leads to increased sensitivity towards particular spatial frequencies independent of network architecture. We further propose a family of $\textit{\textbf{spatial frequency regularizers}}$ based on our proposed measure to induce specific spatial frequency sensitivities in a model. In experiments on datasets with out-of-distribution test images arising from various common image corruptions, we find that deep neural networks trained with our proposed regularizers obtain significantly improved classification accuracy while maintaining high accuracy on in-distribution clean test images.

Under review as a conference paper at ICLR 2022
FOURIER SENSITIVITY AND REGULARIZATION OF COMPUTER VISION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks for computer vision are highly accurate but their out-of- distribution generalization performance remains major challenge, understand- ing their failure modes is an important step towards improving their robustness. Intriguingly, recent work has shown that deep neural networks latch on to the Fourier statistics of training data, show increased sensitivity to certain Fourier- basis directions in the input and are not robust when Fourier-statistics shift un- favorably at test time. Understanding and modifying this Nourier-sensitivin of computer vision models may help improve their robustness, hence, in this paper we study the frequency sensitivity characteristics of deep neural networks using principled approach. We first propose general measure of a computer vision model's spatial frequency sensitivity based on its input-Jacobian represented in the Fourier-basis. When applied to deep neural networks, we find that standard minibatch training consistently leads to increased sensitivity towards particular spatial frequencies independent of network architecture. We further propose a family of frequency regularizers based on our proposed measure to in- duce specific spatial frequency sensitivities in a model. In various generalization tests involving shifts in the Fourier-statistics of the data, we find that deep neu- ral networks trained with our proposed regularizers obtain significantly improved classification accuracy while maintaining high accuracy on in-distribution test im- ages.
INTRODUCTION
While deep neural networks (DNN) achieve remarkable performance on many challenging image classification tasks, they can suffer significant drops in performance when evaluated on out-of- distribution (o.o.d.) data. Intriguingly, this lack of robustness has been partially attributed to the frequency characteristics of data shifts at test time in relation to the frequency sensitivity charac- teristics of the model (Yin et al. 2019 Jo & Bengio 2017). Distinct spatial frequencies in images contain features at different spatial scales; low spatial frequencies (LSF) carry global structure and shape information in a scene whereas high spatial frequencies (HSF) carry local information such edges and borders of objects in are differentially processed in distinct channels of the visual cortex in the brain to learn features at different scales (Appendix B). When information is destroyed or corrupted in frequency bands that a model relies on, performance suffers. Hence, understanding the spatial frequency sensitivity of DNN can help us characterise the features relies on to make predictions.
(Kauffmann et al. 2014); in fact, spatial frequencies
DNNs have been demonstrated to be sensitive to Fourier-basis directions in the input Tsuzuuu & Sato 2019 Yin al.) 2019) both empirically and using theoretical analysis of linear convolutional networks (Tsuzuku & Sato 2019). In fact, the existence of so-called "universal adversarial pertur- bations" (Moosavi-Dezfooli et al. (201)), simple gemantics-preserving distortions that can degrade models' accuracy across inputs and architectures, is attributed to this structural sensitivity and the use of convolution operations. [Yin et al. (2019) also showed that many natural and digital image corruptions that degrade model performance may also be targeting this vulnerability. The Fourier- characteristics of adversarial examples are also known to closely match the sourier-sensitivity of models (Yin et al. 2019). Hence, understanding and modifying Fourier-sensitivity can aid efforts to improve model robustness. While this sourier-sensitivee has been studied empirically, the precise definition and measurement of computer vision model's 's spatial frequency sensitivity still lacks

---

Under review as a conference paper at ICLR 2022
rigorous approach across studies. In addition, there has been no principled method to specifically modify the spatial frequency sensitivity of a model. Existing works have heuristically applied filters on convolution layer parameters (Wang al. 2020 Saikia et al. 2021 and data augmentations (in et al. 2019) to modify a model's frequency sensitivity.
In this work, we propose novel and rigorous measure of deep neural network's ' spatial fre- quency sensitivity using the input-Jacobian represented in the Fourier-basis and show that deep neu- ral networks demonstrate consistent spatial frequency sensitivities across samples, an observation that suggests DNNs are more likely to consistently use some frequencies more than others, and has implications for robustness. In addition, using our proposed measure, which is differentiable with respect to model parameters, we propose novel family of spatial frequency regularizers to directly induce specific frequency sensitivities ations that spatial frequency regularization can modify the frequency sensitivity characteristics of computer vision models and can significantly improve the generalization performance of models on o.o.d. datasets where the Fourier-statistics are unfavorably shifted.
model. We hypothesize and show in empirical evalu-
In summary, the main contributions of this work are as follows:
1. We propose a novel and rigorous measure of a model's spatial frequency sensitivity based its JacobaJacobian represented in the Fourier-basis
2. We propose novel family of spatial frequency regularizers directly induce specific spatial frequency sensitivities
3. We demonstrate that spatial frequency regularization can significantly improve general- ization performance on out-of-distribution data where Fourier-statistics are unfavorably shifted
2 RELATED WORK
Characterising frequency sensitivity: Yin et al. (2019); Tsuzuku & Sato (2019) characterised the Fourier characteristics of trained CNNs using perturbation analysis of their test error under Fourier- basis noise. They showed that a naturally trained model is most sensitive to all but low frequencies whereas models adversarially trained (Madry et al., 2018) models are sensitive to low- low-frequency noise. They further showed that these Fourier characteristics relate to model robustness on corrup- tions and noise, with models biased towards low frequencies performing better under high frequency noise and vice versa. Abello et al. (02T) took a different approach by measuring the impact on accuracy of removing individual frequency components from the input using filters whereas Ortiz- Jimenez et al. (002)) computed the margin in input space along basis directions of the discrete cosine transform (DCT). Wang et al. (2020) made observations about the Fourier characteristics of CNNs in different training regimes including standard and adversarial training by evaluating accuracy on band-past filtered data. In contrast these disparate approaches, in this work, we propose rigorous measure of a model's l' spatial frequency sensitivity.
Regularizing frequency sensitivity: Yin al. (2019) proposed adversarial training (Madry al., 2018) and gaussian noise augmentations as methods that induce a low-frequency sensitivity. Waan al. (2200) proposed smoothing convolution filter parameters induce low-frequency sensitivity in models. We note that, in general, techniques that apply filters on convolutional parameters to affect model's frequency sensitivity do not take into account complex operations such as non- linearities, pooling and other transformations that often follow convolutional layers that can modify as well as undo the effects of such filters. In addition, data augmentations do not provide precise control over the Fourier-sensitivig of a model. In this work, we propose a family of spatial frequency regularizers that can precisely modify the overall spatial frequency sensitivity of any differentiable model.
Jacobian regularization: Methods that regularize the Jacobian of a model's output-logits or loss with respect to its input can broadly be divided into two types; one regularizes the norm of the input-Jacobian and the other regularizes its direction or directional derivatives. Drucker Le Cun [1991) proposed method that penalized the norm of the input-Jacobian to improve generalization, more recently this has been explored to improve robustness to adversarial perturbations (Ross & Doshi 2018 Jakubovitz & Giryes 2018 Hoffmmn et al. 2019). Simard et al. (1992) proposed
2

---

Under review as a conference paper at ICLR 2022
"Tangent Prop", which minimized directional derivatives of classifiers in the direction of local input- transformations (e.g. rotations, translations; called "tangent vectors") to reduce model sensitivity to such transformations. Czarnecki et al. (2017) proposed Sobolev training of neural networks to improve model distillation by matching the input-Jacobian of the original model. Regularizing the direction of the input-Jacobian has also been used improve adversarial robustness (Chan et al.) 2020). In the present work, we regularize Fourier-components in the input-Jacobian to modify the spatial frequency sensitivity of models. As such, we are directly modifying the direction of the input-Jacobian instead of its norm.
METHODS
Preliminaries: We introduce all relevant definitions and notations before describing the proposed methods. Consider an image classification task with input images z, target labels y, and the stan- dard cross-entropy loss function LCE. Let f denote any differentiable model, F(.) the unitary 2D discrete Fourier transform (DFT), F F.) its inverse, and F-) the adjoint of the inverse-Fouricr transform, and let y denote the Fourier space representation of the input, i.e. 2, F(z). We denote the input-Jacobian in the standard input basis 8CCK and &Ccg as the input-Jacobian in the dzz suurier-basis. Let N denote the height of the input images z (although not necessary, all images used in this work are square). The zero-shifted 2D-DFT of the input-Jacobin
0x
denoted F = F( (&CC)) Since the input-Jacobian typically has three color channels, they are averaged before computing the 2D-DFT. Fourier coefficients in F are complex numbers with real and imaginary components; F(u, v) Real(u. (u, v) Imag(u u. v), where (u, v) are indices of coefficients. The power in efficient is its squared amplitude, P(u, v) = [F(u, u)12 = Real(u, u)² + Imag(u u, u)² and the matrix of powers is denoted P (power-matrix), Each coefficient has radial distance r(u, u) from the centre of the matrix, r ruu, u) = d((u, (e, v), (Cu. C.)). where (Cu, G.) denotes the centroid of P and d(;s') ) is Euclidean distance. Distinct radial distances, rounded to the nearest integer, of coefficients in the matrix are the set of integers (I,...,N) N....,N/V. NN/V2] and correspond to low to high spatial frequency bands, the highest spatial frequency being limited by the Nyquist frequency. We denote Protal as the total power in P. excluding the eero-frequency coefficient, Protal Er(ure>==1 P(u, v).
Similarly, define Protat the total power in P excluding the eero-freyuency coefficient as well as coefficients with radial distance r(u,v) > N/2, i.e. outside the largest circle inscribed in the power-matrix P; Protal-
Σ
P(u, v) (please see Figure in Appendix A.1
)<=r(u.)<=N/2
for an illustration). We denote Ps as the total power radial distance & normalized by Protal P(u, v) and Ps total power radial distance k normalized by Protal in-
Pk Protat
Σ
r(u,o)=k
stead, Ps. Protai r(u,))=k
Σ P(u,v).
3.1 SPATIAL FREQUENCY SENSITIVITY OF A MODEL
In this section, we define the proposed spatial frequency sensitivity (SFS) of any differentiable model using its input-Jacobian represented in the Fourier-basis. As the input-Jacobian provides the direction of highest input sensitivity, we show below that its Fourier transform, which is simply a change of basis, provides the model's sensitivities to spatial frequency components in the input. The input-Jacobian in the Fourier-basis, aCeg, comprises the sensitivities of the model with respect to diy
individual frequencies in the input and we obtain this by simply computing the Fourier transform of the input-Jacobian &Cce. In order to justify this, consider the computation graph where the input 8r
is mapped to scalar loss via a model f and loss function LCE (Figure 2 We introduce an implicit operation (shown in red) that maps the Fourier space representation of the input, xy, to
the standard input through the inverse Fourier Fourier-transfome transform; z - x. so in order to compute the must differentiate through this implicit operation in the forward graph. Since the inverse-Fourier transform is unitary operator, its adjoint is also its inverse, i.e. F-1* (F-I)-1= F. Hence, following the chain rule for complex operators, dCeg Ozy
input-Jacobian in the Fourier-basis &CCr de,
is simply the Fourier transform of the input-Jacobian 8CcB 8x
3

---

Under review as a conference paper at ICLR 2022
SPS=proportion of power circular frequency bands
Input image
Input-Jacobian of model
21D DFT input-lacobian
Figure 1: Illustration of steps compute the spatial frequency sensitivity with respect to single input image. The input-Jacobian the model is Fourier-trande tranfforreansf to obtain sensitivities with respect to frequency components followed by computing the proportion of power in low high (small to large radius) frequency bands.
f
LCE
&Coc day
OLCE Bx
backprop
Figure 2: Computation graph to obtain the input-Jacobian in Fourier-space. The symbols in red represent an implicit map of the input from Fourier-space to the standard input space. The dashed arrows denote the backward graph operations to compute the input-Jacobian Fourier space.
OCCE dzj
OCCE =- OCCE F( Ōr
Hence, even though we do explicitly the Fourier representation of the input, this shows that the Fourier transform of the input-Jacobian provides us the sensitivity of the model with respect to spatial frequencies of the input. In fact, similar results hold for any unitary operation on the input- Jacobian, such the discrete cosine transform (DCT). Interested readers can replace the DFT (F) with other such transforms if they would like to understand the sensitivity of a model with respect to other components in the input. We now define the spatial frequency sensitivity fsFs(x, y) with respect to an individual input (I, y) as,
[sss(æ, (Pi,...,PN//5)
where Pke is the proportion of total power in Fourier coefficients at radial distance l in the power input-Jacobian. The overall spatial frequency sensitivity of a model is defined as the expectation of (x 3y) over the data distribution, i.e. [sFs(=;0)
matrix P of the
B(a.)>l/sFFs(r, w)] (please see Figure for illustration and Algorithm [ in Appendix A. We note that although we use the Jacobian of the loss function to estimate The SFS a model, this formulation is valid for other output functions and tasks well. An alternative approach that does not require image labels is to compute the Jacobian of the model's output logits or softmax probabilities. In addition, while the spatial frequency sensitivity assumes the input spatial 2D data (e.g. images), we can extend this approach to any n-dimensional data by using the n-dimensional Fourier-transform well other computer vision tasks.
3.2 SPATIAL FREQUENCY REGULARIZATION
In this section, we propose a novel family of spatial frequency regularizers that can modify the spatial frequency sensitivity, as defined Section 3.1, of model directly. As all computations involved in the SFS of a model (Algorithm D in Appendix A.I) are differentiable with respect to its parameters, we compute a loss function, L(x, y) = Lce y) + Asrs be used to induce specific SFS and Agrs hyperparameter. As
(ars (z,8) y), where Lars
is the proposed loss that Lsrs is defined on the SFS, which is a function of the input-Jacobian, optimizing it requires an additional backpropagation step compute its derivatives with respect to model parameters, similar
4

---

Under review as a conference paper at ICLR 2022
to other Jacobian-regularizzil methods. We now define Laes for three different SFS regularizers; SFS E {LSF, MSF,ASF) LSF regularization trains a model to be insensitive to medium and high spatial frequencies and MSF regularization trains a model to be insensitive to low and high spatial frequencies. We achieve this by penalizing the proportion of total power, Ps. in frequency bands. ASF regularization trains model to be equally sensitive to all spatial frequency (ASF) bands. The motivation behind ASF regularization model is to encourage model to be sensitive multiple frequency bands instead of being concentrated in a single frequeney range. Hence, the ASF-regularizer loss is the negative entropy of the distribution of power over spatial frequency bands.
Σ Pk. k>///6
SFS=LSSF
E
Pk: ifSFS=MSF
(ops(a,))
N/N/2 Σ
log PR. ifSFS=ASF
The definitions of frequency ranges are based on equally dividing the largest circle inscribed in the power-matrix P' into equal parts (Figure D in Appendix [A.I). For ASF-regulariiatioon, very high frequency bands (r(u, v) > N/2 are excluded. Interested readers may modify these loss functions as they see for their particular application.
EXPERIMENTS
In this section we explore the SFS of models as well as explore the effects of spatial frequency regularization by evaluating datasets where the Fourier-statisties of predictive features are nat- urally or artificially altered (Sections 4.2 4.4) We also make some observations about the differ- ences between spatial frequency regularization and training Fourier-filtered training data (Section D.3). We benchmark against methods that have been proposed to modify the frequency sensitivity of models such as adversarial training and Gaussian Noise augmentation (Yin et al. 2019) in- duce low-frequency sensitivity. AugMix (Hendrycks et al.] 2020 is also benchmarked as was shown to achieve state-of-the performance common corruptions. We also trained models on band-pass filtered datasets to demonstrate that SFS regularization is not equivalent to training on Fourier-filtered data (diseussion relegated to Appendix D.3).
We first describe the experimental settings for all experiments that fol- low. We trained the ResNet50 architecture, unless stated otherwise. For CIFARIO and CIFARI00 (Kriheessky & Hinton] 2009), we trained all models for 150 epochs using stochastic gradient de- scent (SGD) with momentum (0.9), an initial learning rate 0.1 decayed by a factor of 10 every 50 epochs, weight decay parameter equal to 5e-4 and batch size equal to 128. For SVHN (Netzer) jet al. 2011). we trained models for 40 epochs using Nesterov momentum with an initial learning rate of 0.01 and momentum parameter 0.9. The training batch size was 128, the L2 regulariza- tion parameter was 0.0005 and we decayed the learning rate at epochs 15 and 30 by dividing by 10. Standard data augmentations random-croo ceoop were used during training. We trained spatial frequency regularized models using Algorithm 2 2 in Appendix A. and λsrs for all models. We observed a simple trade-off between this hyperpa- rameter and clean accuracy (higher values decreased clean accuracy more). Values less than did not always achieve the desired SFS in the evaluated datasets. For adversarial training (AT), we used PGD la attacks (e = 1, attack-steps augmentation, we added i.i.d. Gaussian noise (0, o²) to each pixel in all the images in the training set (a 0.1). We used the robustness (Engstrom et al. 2019) library to train models as well as re- trieve pre-trained ImageNet models of various architectures. We trained AugMix models using the original code shared by the authors (httpss: //gi thub. com reseaaererreaaaaaaaaaae augmix Please see Section D.2 in Appendix for details about band-pass filters.
Experimental Setup:
-fip.
and color-jitter
attack-Ir = 1/7). To train models with Gaussian noise data
5

---

Under review as a conference paper at ICLR 2022
CIRARIO (ReeNee50)
CBIARIO NRoa
ImageNet (ResNee55)
9.31
4.3
Spataa Frequency (nyloes/nago)
Spotial Frequezcy tedieatazgee
Spottal Prequency teyclesteaagn
Figure 3: SFS of ResNet50 models trained by methods on CIFARIO and ImageNet. Computed on 1K randomly selected samples from respective validation sets. The shaded region represents two standard deviations.
4.1 CNNS DISPLAY CONSISTENT SPATIAL FREQUENCY SENSITIVITY ACROSS SAMPLES
We now visualize the spatial frequency frequenyy sensitivity (SFS), as defined in Section 3.1 of models trained on CIFARIO and ImageNet (Figure We note that training biases models to be sensitive to some frequencies more than others and this bias is consistent across samples (the error bars in Figure 3 represent deviation across samples). This consistency also suggests that, once trained, CNNs are more likely to use some frequencies more than others and may explain their tendency to suffer performance drops when evaluated on data with different Fourier-statistics CIFAR100 models have very similar SFS to those of CIFARIO (plots relegated to Appendix (C2) and also most sensitive to mid-to-high spatial frequencies. In contrast, standard training on SVHN leads to a low-frequency sensitivity, which suggests a strong dataset dependence of spatial frequency sensitivity. Please refer to Appendix C.3 for SFS plots of models trained on SVHN.
wide range of the frequency spectrum with peak sensitivity to mid-range frequencies. We also observed consistency in the spatial frequency sensitivity of vari- ous convolutional architectures trained on ImageNet (Appendix C.T. Adversarially trained models (Madry et al. 2018) are most sensitive to low spatial frequencies across datasets and architectures which suggests they rely on coarse global features, in agreement with observations made in prior work. Training on mifiiieed-maeeeet,, proposed by [Geirhos et al. (2019) to train shape-biased in models, induces a lower peak-spatial-te seaatall-freuuec--sesniiiii frequencies compared to training ImageNet (Appendix C.2) which reflects the increased shape- training data corre-
ImageNet models sensitive
and as well as lower sensitivity high
bias of such models. Similarly, a model trained on low-pass Fourier-filtert spondingly has high sensitivity to www-frequencies in comparison to the baseline model (Appendix C.2). Gaussian noise augmentation (000 o²: a = 0.1), decreases the
sensitivity of
the model (Figure 3 compared to baseline. The SFS of the AugMix (Hendrycks et al. (020) trained model was not significantly different from baseline, which suggests it finds more robust features in similar frequency bands using data augmentations.
Interestingly, we also note that models trained on corrupted versions of the CIFAR10 training set (c.g. blurred or (PEG-compressed) can display different spatial frequency sensitivities model trained on clean CIFARI6 10 images (please see Appendix (rneendix(C.4[iee [C.4 for plots). For example, models trained on images with severe noise corruptions (gaussian, shot and speckle) display increased sensitiv- blauseb JPEG-
ity to lower spatial frequencies. Models trained on highly gaussian-blure compressed and pixelated images also display higher sensitivity lower spatial frequencies. These changes reflect the shift in the Fourier-statisties of predictive features in corrupted images.
4.2 PERFORMANCE UNDER FOURIER FILTERING
Jo & Bengio 2017) showed that DNNs have a tendency to rely superficial Fourier-statistics of their training data. In the vein of generalization evaluations they performed, we generate Fourier- filtered CIFAR 1O/CIFARIOOO 100 validation images using radial masking in frequency space (please Appendix D.I for examples). A mask radius r determines Fourier components that are preserved with larger radii preserving more components. We use (Cu, Cv) to denote the centre of the mask and

---

Under review as a conference paper at ICLR 2022
Original
Fourier-filtered
Patch-shuffled
Original
High-Freq
Med-Freq
Low-Freg
4 (d)
1 (c)
4 S
(a)
(c)
(g)
Figure 4: Examples (b): CIFARIO Fourier-filtered (Section 4.2
(C):
CIFARIO Patch-shuffled
(Section 4.5 (e) (g): Fourier-noise distortions on SVHN (Section More in Appendix.
d(ss.) to denote Euclidean distance. The radial mask M, defined as:
M, 11 ifd((u, v), (Cu, Co)) S
(0, otherwise
(1)
The mask is applied on the zero-shifted output of the Fourier transform of each image, denoted X, followed by the inverse transform, i.e. Xfittered = (FX( + M,), where is the element- performed each color channel independently. A standard CNN
wise product. Fourier-filtering suffers up to a 75% drop in accuracy on low-pass filtered CIFAR data as it relies on high frequency information that is no longer present (Table On the other hand, the LSF-REGULARIIED model performs well on both clean images as well severely low-pass filtered images. This shows that regularized CNNs able exploit very low frequency features (r S 5) not easily recognizable even to humans, which reflects the limited spatial peeeial-feequeney-ssssiitiiiii range of the human eye.
CIFARIO
CIFARI00
Method
clean
r=
r=
7
==
clean
r= 11
r= 7
==
Std. Train
94.9
78.1
24.9
18.6
76.2
49.7
14.1
6.6
ESP-REGULARIZED
86.3
86.2
84.4
78.3
61.4
61.5
58.0
46.8
AES-RREGUADD
87.2
86.3
71.5
46.2
62.4
62.2
46.4
18.6
ASF-REGULARIZED
87.6
85.0
69.3
45.0
67.0
62.1
41.1
19.8
AT (PGD f2. 1)
81.6
80.2
76.1
67.5
58.8
56.8
50.0
40.2
Gaussian Noise
94.5
84.4
32.4
19.5
73.1
61.9
27.7
11.6
AugMix
95.8
92.0
79.4
50.2
77.8
64.0
45.6
24.6
Table 1: Accuracy on couricr-filtered CIFARIO/CIFAR 100 test sets.
The MSF-REGULARIZER model performs better for mild Fourier-filtrrne filtering (r 11 and r= 7) but suffers a big drop in accuracy (71.5% to 46.2% CIFARIO and 46. to 18.6% in CIFAR 100) for strong low-pass filtering = 5). We also observed that the ASF-REGULARIZED model is also significantly more robust than the baseline model although not as robust as the REGULARRID model to fourier-filtering This shows that the ASF-REGULLEEE REGULARIZED model pratial-frequence features in addition to medium and high spatial frequencies. The low- fweeuerquncy sensitivity of adversarially trained models makes it more robust than the baseline model but less so than the L.SF-REGULARIZED model, while the Gaussian noise augmented model suffers large drops accuracy. AugMix tends be robust mild foutiir-filtering but suffers large drops in accuracy under more severe filtering (r = 5) due to its reliance on mid-to-bigh-frequenyy information, as suggested by its SFS (Figure 3 comparison the SF-REGULARRZED model.
coarse low-
4.3 PERFORMANCE UNDER FOURIER- CORRUPTIONS
Recently, data-agnostic corruptions along Fourier- basis directions have been identified threat to model robustness and security (Yin al. 2019 Tsuzuku Sato 2019). A Fourier-noise corruption is additive noise containing single Fourier-mode (frequency). These corruptions semantics- preserving and significantly affect model performance while being recognizable to humans, please refer to Appendix [E] for examples in each dataset. We generated corruptions for each of 1,024 fourier-moder in SVHN/CIFAR 1w/.//O////ARROOO 100 and report the overall mean accuracy across all cor- ruptions in Table 2 We note that the baseline model suffers significant drops in accuracy across all

---

Under review as a conference paper at ICLR 2022
datasets. The LSF-regularized model achieves the highest mean accuracy in SVHN while Gaus- sian augmentation also improves over the baseline. AugMix (Hendrycks et al. 2020). a data- augmentation method proposed for robustness, performs worse than baseline. In CIFAR10 and IIFARI00, the MSF-regularized model achieves the highest mean accuracies. Detailed heat maps containing error rates for each Fourier-mode can be found in Appendix E
SVIIN
CIFARIO
CIFAR100
Method
clean
<=3
c=4
clean
3=
e=4
clean
e=3
c=4
Std. Train
96.4
81.9
77.4
94.9
40.8
31.5
76.2
22.3
14.9
LSF- DEFERRGULARDEED
95.1
92.1
91.0
86.3
52.4
47.5
61.4
33.9
30
AIS-REGULARIZZED
93.1
77.1
70.9
87.2
62.4
54.3
62.4
42.6
37.3
ASF-REGULARIZED
96.1
78.3
71.1
87.6
60.8
48.7
67
21.6
14.6
AugMix
96.6
75.7
70.1
95.8
53.0
41.0
77.8
29.1
22.7
Gaussian Noise
96.4
89
85.1
94.5
57.2
40.8
73.1
33.7
20.9
Table 2:
Mean accuracy across all Fourier-noise corruptions averaged across 1,024 randomly lected test samples for each corruption. Eg norms of the additive Fourier-noise are c E {3,4).
4.4 PERFORMANCE UNDER CORRUPTIONS (HENDRYCKS DIETTERICH 2019)
We evaluated the robustness of spatial frequency regularized models to corruptions (please see Appendix T for examples) in CIFARIO-C (Hendrycks & Dietterich 2019). In addition, we ap- plied these corruptions to SVHN to create SVHN-C to evaluate the performance methods across datasets. Overall, SFS regularization provided consistent improvements across both SVHN-C (Ta- ble 3 and CIFAR10-C (Table 4 due to its general and principled approach. We observed that AugMix's 's (Hendryeks et formance on SVHN-C. Similarly, Gaussian augmentation's 's high performance in SVHN-C did not translate to CIFARIO-C. Specifically, on SVHN-C, LeF-regularizztion, low-pass filtering and Gaus- sian augmentation performed very well under noise corruptions whereas AugMix did not improve over the baseline. In CIFARIO-C, we observed that the LSF-REGULARIZED model was signifi- cantly more robust than the baseline model to all blurring corruptions as well as pixelate, elastic, snow and frost corruptions. The baseline model suffered up to 45% drop in accuracy on these cor- ruptions, whereas the LSF-REGULARIIZE model suffered at most 5% drop in accuracy. This is likely due to the robustness of low-spatial-frequencc features to corruptions such as blurring. We also observed that the MSF-REGULARZEE noise corruptions as well as JPEG compression (Table 4). which suggests multiple spatial frequency bands maybe needed to do well under such corruptions. We observed similar results in CIFARI00-4 (Appendix F22.. Overall, SFS regularization can provide consistent improvements in robustness to these corruptions across datasets due to its general and principled approach. On the other hand, data augmentation based methods invariably perform better on some datasets than others depending on the statistics and features of each dataset.
2020) high performance CIFAR 10-C does not translate to high per-
and ASF-REGULARIZED models were more robust to
Naise
Blur
Weather
Digital
Method
Clean Gauss. Shot Impalse
Defocus Glass Motion 7oom
Snow Frrst
Fog Bright
Contrast Flastic Pixel JPEG
Sid. Train
96.4
81.3
84.4
72.7
95.9
92.4
94.9
96.1
75.5
84.0
48.9
94.5
89.4
90.0
95.8
95.9
SE--REGULARIZET
95,1
90.8
91.6
84.7
94.6
92.9
93.5
95.2
82.4
88.4
42.1
93.2
83.6
91.1
94.8
94.9
REFFRREGULARZZE
93.1
68.3
71.8
52.6
92.7
82.8
90.9
93.6
75.2
76.6
40.7
90.7
62.8
84.1
91.8
92.2
BEF.REUULA ARIZED
96.1
80.9
83.3
73.2
95.5
91.7
94.4
96.0
79.2
84.3
45.5
94.3
85.7
91.0
95.2
95.7
Low pass liltered
90.5
91.4
85.1
94.7
93.0
93.7
95.1
88.1
40.8
92.7
90.5
94.8
AT (PGD foo. 8/255)
96.9
85.6
88.1
73.8
96.0
94,2
95.0
96.6
80.1
88.8
43,9
95.0
72.2
93.2
96.5
96.6
Gaussian Noise
96.4
92.8
93.8
84.4
95.5
94.4
94.4
96.2
80.7
89.3
42.6
94.4
74.4
92.4
96.0
96.1
AugMix
96.6
79.1
82.5
74.7
96.4
92.1
95.4
96.4
73.4
83.1
50.0
95.2
93.5
90.5
95.2
96.2
Table 3: Accuracies on SVHN-C corruptions of highest severity level. applying corruptions in CIFARIO-C to SVHN test samples.
We created SVHN-C by
8

---

Under review as a conference paper at ICLR 2022
Noise
Blur
Weather
Digital
Method
Clean Gauss. Shot Impulse
Defocus Glass Motion 7rom
Snow
Frrst
Fog Bright
Contrast Flastic Pixel JPEG
94,9
73.2
LSS-RRGULARRZED
86.3
29.4
32.5
20.0
8399
81.8
80.4
83.9
79.6
79.2
60.6
82.0
28.3
82.5
85.5
65.3
MSF-REGE REGULARIZED
87.2
55.6
59.2
34.9
78.0
71.2
67.7
77.5
77.8
78.4
60.4
82.8
19.5
76.1
83.5
81.4
ASE ESF-REUUT REGULARIZED
87.6
62.3
64.6
44.6
71.3
70.2
69.7
75.3
75.9
74.6
47.3
83.6
26.5
76.1
79.7
82.6
Luw pass liltered
86.6
58.0
60.4
43.5
86.0
84.0
82.5
85.5
81.0
80.3
62.7
82.5
35.2
83.5
86.0
71.6
AT fg. 1)
81.6
75.5
74.9
74.2
70.5
75.8
Gaussian Noise
94.5
44.5
53.6
29.4
53.6
55.4
62.6
66.3
80.6
78.0
68.6
91.6
27.4
73.8
54.5
82.2
AugMix
95.8
66.1
72.7
75.6
92.6
75.0
90.8
91.6
86.6
85.4
83.2
93.6
86.6
85.4
83.2
93.6
Table 4: Accuracies on CIFARIO-C corruptions of highest severity level.
4.5 EVALUATION OF GLOBAL FEATURE LEARNING
Here wish to evaluate the extent which models use global features by measuring their per- formance on patch-shuffee images, which have previously been used by Mummadi et al. (2021); Zhang & Zhu (2019); [Wang et al. (2019). Patch-shufflig involves splitting an image into squares and randomly swapping the positions of these squares. This is intended to destroy global features and retain local features with larger values of retaining less global structure the image (please see Appendix G for examples). As such, models that rely more global rather than lo- cal structure suffer more from patch- satch-shuffling. Hence, in this benchmark, lower accuracy suggests increased reliance and use of global structure. We observed that spatial frequency regularized models, adversarially trained models suffered larger drops in accuracy compared to the standard trained, AugMix and Gaussian augmentation models, which suggests their reliance global struc- ture images contributes to their robustness (Table
CIFARIO
CIFARI00
Method
clean
2
3
clean
k
2
A
3
Std. Train AugMix Gaussian Noise
94.9
66.5
45.8
76.2
39.9
21.4
67.3
48.7
77.8
40.7
23.8
62.9
44.5
73.1
34.4
18
LSF-REGUL REGULARIZED
86.3
43.2
30.6
61.4
23.4
13
MSF- ESS-REGULARIZED
87.2
46.8
33.1
62.4
24.1
13.0
ASF-REGULARIZER
87.6
46.8
32.6
67
29.0
15
Low-pass filtered
86.6
43.4
30.0
60
20.4
10.4
AT (PGD &2, 1)
81.6
45.2
35.0
58.8
19.1
11.1
Table 5: Accuracy on patch-shuffled CIFARIO/CIFARION test sets.
CONCLUSION
We proposed novel and rigorous measure of the spatial frequency sensitivity (SFS) of any differ- entiable computer model using its input-Jacobian represented in the Fourier-basis. We showed that models display consistent SFS across samples as well across architectures for given task and when there is shift in the Pourier-statisties of features relative to a model's frequency sensitivity, performance can suffer. We also proposed novel method of spatial frequency regularization that can be used to improve robustness to many test-time corruptions. Extending our method to other computer vision tasks (e.g. detection) as well as domains (n-diimessiona) are interesting directions for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Antonio A. Abello, Roberto Hirata, and Zhangyang Wang. Dissecting the High-Freqqencyy Bias in Convolutional Neural Networks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 863-871, 2021. URL www.theteeccccc ctt httpap/www.o.oooooeet//////////n
https::/openaccess wtwpeeeeeeeeaaaaaaaaoooooooooaaa Dissecting the Bas___the_ihhhhhee Networks CVPRW MWW.orrk___CRRWW__ 2021 paper.ppppp.... .html
F W Campbell and J G Robson. Application of Fourier analysis to the visibility of gratings. The
Journal of physiology, 197(3):511-566, August 1968. ISSN 0022-3751. doi: 10. 11.phhsl/p 1968.50008574. URL httptst:/ppbbbbedd..ttte
r.m. nih. wtt......h..ov/666511..... gov
FW Campbell and JG Robson. Application of fourier analysis to modulation response of eye. In Journal of the Optical Society of America, volume 54, pp. 581, 1964. Issue: 4.
Alvin Chan, Yi Tay, Yew Soon Ong, and Jie Fu. Jacobian Adversarially Regularized Networks for Robustness. arXiv: 1912. 10185 [cs]. January 2020. URL 10185 arXiv: 1912.10185.
Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev Training for Neural Networks. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL /ttpsts///rrreeeeiiggg neur.pss......p..p.ee........ cc/
paper/2017 file/////////// (ee55e00ee6116888000eeeeeeeeeee 2tt.///88550...00000033333222222
H. Drucker and Y. Le Cun. Double backpropagation increasing generalization performance. In WCCNN-9I-Seatle International Joint Conference Neural Networks, volume ii, pp. 145-150 vol.2, 1991. doi: 10.1 11110///NNN.11911115555.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (Python Library), 2019. URL https: :t.p::///ii. //ggttgt.h/.///// com/
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv:1811.1223 [cs, q-bio, stat], January 2019. URL
htttp://arxiv.org//
12231
arXiv: 1811.12231.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Com-
mon Corruptions and Perturbations. arXiv: /226/ [es, stat), March 2019. URL http:
//axxiv.org/abll 1903. 12261
arXiv: 1203.12261.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A Simple Data Processing Method to Improve Robustness and Uncer- tainty. arXiv:1912.0278) (cs, stat), February 2020. URL http::::/wax...o////aaaaaae///e/
02781
arXiv: 1912.02781.
Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust Learning with Jacobian Regularization. arXiv: 1908. 02729 [cs, stat], August 2019. arXiv: 1908.02729.
Aapo Hyvärinen, Janmo Hurri, and Patrick O. Hoyer. Natural Image Statistics: A Probabilistic Ap- proach to Early Computational Vision. Springer Publishing Company, Incorporated, 1st edition, 2009. ISBN 1-84882-490-4.
Daniel Jakubovitz and Raja Giryes. Improving DNN Robustness to Adversarial Attacks using Ja- cobian Regularizatinn. In Proceedings of the European Conference Computer Vision (ECCV), September 2018.
Jason Jo and Yoshua Bengio. Measuring the tendency of CNNs to Learn Surface Statistical Regular- ities. arXiv:1711.1155 [cs. stat], November 2017. URL http::/ http///axxvvv/.a http:://aaaxxi..oo//////////1//
11561 arXiv: 1711.11561.
10

---

Under review as a conference paper at ICLR 2022
B. Julesz and T. V. Papathomas. On opatial-frequeney channels and attention. Perception & psy- chophysics, 36(4):398- -998, October 1984. ISSN 0031-5117. doi: 10.3758/bf03202799. Place: United States.
Louise Kauffimann, Stephen Ramanoël, and Carole Peyrin. The neural bases of spatial frequency processing during scene perception. Frontiers in inlegrative neuroscience, 8:37, 2014. ISSN 1662-5145. doi: 10.3389/mmint22244.0033...
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report 0, Technical report, University of Toronto, Toronto, Ontario, 2009. Backup Pub- lisher: University of Toronto.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Confer- ence on Learning Representations, 2018. URL
IJZIBfZAb
Seyed-Mohsen Moosavi-Dezfoon Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Univer- sal Adversarial Perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Chaithanya Kumar Mummadi, Ranjitha Subramaniam, Robin Hutmacher, Julien Vitay, Volker Fis- cher, and Jan Hendrik Metzen. Does enhanced shape bias improve neural network robustness to common corruption?? In International Conference on Learning Representations, 2021. URL
lttps::/openreview net http//ooommmmmmdddee htt.e//eneenooooouoooddoooooo/o.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 201 2011. URL WRLLhttp:///ffdde stanford.
adt/hhousennumbrrr//nisssss httpasss211shhhsuuuuusseeeee housenumbers.... pdff
Guillermo Ortiz-Jimenez. Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Hold me tight! Influence of discriminative features on deep network boundaries.
arXiv:2002.068 06349 [es, stat), October 2020. URL ULL.nttts///axxx..og///////222.
arXiv: 2002.06349.
Andrew Slavin Ross and Finale Doshi Velez. Improving the Adversarial Robustness and Inter- pretability of Deep Neural Networks by Regularizing their Input Gradients. In AAAI, 2018.
Pia Rotshtein, Andrew Schofield, María Funes, and Glyn W. Humphreys. Effects of spatial fre- quency bands on perceptual decision: it is not the stimuli but the comparison. Journal of vision, 10(10):25, August 2010. ISSN 1534-7362. doi: 10. 10.1167/10.0..55. Place: United States.
Tonmoy Saikia, Cordelia Schmid, and Thomas Brox. Improving robustness against common cor- ruptions with frequency biased models. arXiv:2103. 16241 [cs], March 2021. URL http:
7/ //arxiv.org///aaae...aaaaa wttp.//.wrr//bbbbb.1.... 2213.b/2111111111111111
arXiv: 2103.16241.
P. G. Schyns and A. Oliva. Dr. Angry and Mr. Smile: when categorization flexibly modifies the perception faces in rapid visual presentations. Cognition, 69(3))243-265, January 1999. ISSN 0010-0277. doi: 10.10166001000077799800800
Place: Netherlands.
Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent Prop - A formal- ism for specifying selected invariances in adaptive network. In J. Moody, S. Hanson, and R. P. Lippmann (eds.), Advances in Neural Information Processing Systems, volume 4. Morgan- Kaufmann, 1992. URL https: http::://cccccceeeeeeeee neurips.......s................. http:///ww....n.eeeeeeeeeeeeee/e wttppe/pp////9//iiee.... Paper
Yusuke Tsuzuku and Issei Sato. On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions. In 2019 IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 51-60, Long Beach, CA. USA, June 2019. IEEE. ISBN 978-1-72813-293-- doi: 10. 10.1109/CVPR.20199000 htte//documentt/9944555
URL /ttl:tsss/eeeeeeeeeee.ee ieee.
11

---

Under review as a conference paper at ICLR 2022
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Representa- tions by Penalizing Local Predictive Power. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL ht httpttpspspppppppeeeeeceooooeoee ngs neur ps cc/ paper/2019/F htt.1eeeeeeeeeeeeeeeeeeeee (ww.:/88877744889922998 ott.r//cc.........33333333626333
Haohan Wang, Xindi Wu, Zeyi Huang. and Eric P. Xing. High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks. In 2020 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 8681-869 Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817-168-5.
URL doi: 10. (G.11099CCP44420000000000 http::://eeepllre.ieeess ieee ort///ccmmmeett/15542555
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, and Justin Gilmer. A Fourier Perspective on Model Robustness in Computer Vision. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, 2019.
Tianyuan Zhang and Zhanxing Zhu. Interpreting Adversarially Trained Convolutional Neural Net- works. arXiv: arXiv:/905.0979) [cs, stat], May 2019. URL http:///arx...orr/abbss 1905. 09797
arXiv: 1905.09797.
12

---

Under review as a conference paper at ICLR 2022
APPENDICES
A SPATIAL FREQUENCY SENSITIVITY AND REGULARIZER
A.1 PSEUDO-CODES
Algorithm 1:
Spatial Frequency Sensitivity (SFS)
Input: Labeled samples L = {( M=) model f with trained parameters
Output: Estimated Spatial Frequency Sensitivity of model, fsrs(::0)
for i 1, I...., do
compute loss (I;() 1R) // forward pass
backpropagate LCE to obtain dCen // jacutajacobian
ds.
8CCn F( @CCR) // 2D DFT of input - jociti-jjcbiann (averaged across 8z,;
color channels)
fsrs( M) = [Ps; for to Ps is proportion of total power (excluding DC) in coefficients at radial distance end
fsrs(:6) (sr((e // estimated SFS model
Algorithm 2:
Spatial Frequency Regularized Minibatch Training
Input: Labeled samples = {(r;, 1=))=11 model / with parameters 0; SFS {LSF, MSF, JSF); hyperparameter Asrs
Output: Regularized model with updated parameters 0
for i 1,...., n do
compute loss (f v.) // forward pass
[(x,, wi) = LCE(#, (Zi, vi) + AsFs vs) // backpropagate to update
end
P(u,v)
P(u,v)
a)
b)
N/v2
N=
zero-trequency (DC)
N
N=3
N/
N
(c
P Total power inside entire square
P Protal power inside circle
N
Figure 5: a) Illustration of the power-matrix P of the Fourfee-transformedd input-Jacobian. Protats the power in all components excluding the zero-frequency component used to normalize the pro- posed SFS of the model at a particular input. b) LSF (purple). MSF (blue) and ASF (purple, blue and green) frequency bands used for spatial frequency regularization. Best viewed in color.
13

---

Under review as a conference paper at ICLR 2022
B SPATIAL FREQUENCY CHANNELS IN THE BRAIN
To further motivate the spatial frequency perspective of visual representation learning, we briefly describe some relevant findings from neuroscience and vision research. While the brain does not strictly perform a Fourier analysis of visual scenes, there has been mounting evidence over decades for multiple spatial frequency channels in the early human visual system that are physiologically independent and selectively responsive to distinct spatial frequency bands (Campbell & Robson 1964 [968). It has been posited that the use of different spatial frequency channels are determined by the demands of given visual task through an attention mechanism (Schyns & Oliva [ 1999,
Rotshtein et al. 2010 Julesz & Papathomas 1984). Spatial frequency channels enable us attend to different spatial scales in a scene at fixed viewing distance, similar to the focus lens in camera (Figure 6). On the other hand, once trained, a CNN's spatial frequency sensitivity (Section 3.1 is inflexible and does not vary much across samples (both in and out of its training data distribution).
EFFFEF TFAFF2ooonnnooooooooooooooooooo
EFEFEFEFEE DFA.
EFEFEFEF EF www.aoowwwwoo....ooooooooooooooa
EFEFEF F wrw.://ooooooooooooroooroooooor.
EFEFEF WFW. TFFF
EFEF EF TF F BF FF
EFFFEFEFEF TF F TO
WFOOEESSFEFFFFFOOOOOOOEEEEEEEEES EFEFEFEFE TTFODHHFFOOOOOOOOOOOFFFEEEEEEEEO B FFTT XXHHWH/HHHHHHHHHHHOOOOOOOOOOOOEE XPT.
EFEFEFEFERE
EF EFRUWEECOEEESSEEEEEEEEEESSSSSEEE DB BIUUUUGOEEEEEEEEEEEEESTTTEETEETD EXPROEEREERRRREEEEEEEEEEEEEEEEEE EF EEROOEEOEEEEERRRREEEEEESSSEEEEEE EFEF BO
Figure 6: Letters at multiple spatial scales. This image comprises the letters o, n, and at a small spatial scale (HSF). The letter T' is also visible at a larger spatial scale (LSF) formed by the specific arrangement of the letters o, n amidst letters E, F. Identifying these letters requires processing fea- tures at multiple scales, enabled by distinct spatial frequency channels in the early visual cortex. Our ability to recognize only one of these scales at a time evidence for the physiological independence of spatial frequency channels in the brain. Image based on Julesz& & Papathomas (1984).
14

---

Under review as a conference paper at ICLR 2022
C
SUPPLEMENTARY PLOTS
C.1 SPATIAL FREQUENCY SENSITIVITIES OF POPULAR IMAGENET MODELS
Spetal Trequeccy igycinttrgge
Spoitsl Preguancy
Speril Theepancy toydiattagee
ShattoNin
VOGT6
RecNent00
DE Spalial (ncllorrang))
Sparal toyliatinipe
Figure 7: Spatial frequency sensitivities of popular ImageNet architectures after standard training and adversarial training using PGD-&2 (e = 3) attacks. The SFS is consistent across architectures for both standard and adversarially trained models although VGG16 shows increased sensitivity to high frequencies. The shaded region represents two standard deviations.
C.2 SPATIAL FREQUENCY SENSITIVITIES MODELS TRAINED ON CIFARIO AND CIFARI00
CIFARIO (ResNe150)
CIFAR100 (ResNet50)
1.40
144
Jn Miterd eadepas Staerad
Total Dower Taalo
Doower Ratio
2:18
2:15
0.00
6.16
Spatial Prequency (cyclestimage)
Spatial Prequency (cyclestimage)
Figure 8: Various methods trained CIFARI0 (left) and CIFAR100 (right). The shaded region represents two standard deviations.
15

---

Under review as a conference paper at ICLR 2022
C.3
SPATIAL FREQUENCY SENSITIVITIES MODELS TRAINED ON SVHN
SVHN (ResNet50)
SVIN (ResNet50)
Admrronn Trankig
Power
Spatial Frequency (cyclestimage)
Spatial Frequency (cycles/image))
Figure 9: Various methods trained on SVHN. The shaded region represents two standard deviations.
C.4 TRAINING ON CIFAR10 CORRUPTIONS
dast ectios thr
Clao
Total Power Ratio
0.00
bit
Total Powerr Retio
0.30
8.25
0.35
6.08
1.6
Spattal Prequency (cycles/tmage)
Spatial Prequency (cyclestimage)
SAM
peg rompecatio wito
Totel Tower Ratio
0.00
Total Poweer Retio
0.34
k.2
0.35
Spatial Prequency (cyclestimage)
Spatial Prequency (cyclestimage)
Figure 10: Spatial frequency sensitivities of ResNet50 models trained on the CIFARIO training set distorted by corruptions derived from the CIFARIO-C (severity 5) dataset. The shaded region represents two standard deviations.
16

---

Under review as a conference paper at ICLR 2022
D FOURIER FILTERING
D.1 RADIAL FILTERING
unfiltered
1=1l
1
Figure 11: First image in each is the mask in Fourier space (lowest frequency centre). White pixels preserve and black pixels set Fourier components to zero. Top row are original CIFARIO images, other Fourier-filtered with different radial masks.
D.2 BAND-PASS FILTERING
unfiltered
low-pass
high-pass med-pass
Figure 12: For band-past fourier-filterin CIFAR10 training images, we filtered fourier-coefficienr each color channel separately. For low-pass filtering, Fourier-coefficients with radial distance
r(u, v) > 5 were set to zero. For medium-pass filtering, Fourier-coefficienn with r(u, v) < and r(u, v) 10 were set to zero. In high-pass filtering, Fourier-coefffcients with r(u, v) 10 were set to zero. Images contast-maximins for viewing.
17

---

Under review as a conference paper at ICLR 2022
D.3 SPATIAL FREQUENCY REGULARIZATION NOT EQUIVALENT TRAINING ON HOURIER-FILTEED DATA
We note that spatial frequency regularization is equivalent standard training corresponding Fourier-filtered data. In natural images, the amount of energy in frequency bands falls off rapidly as frequency increases [Hväärmenn al. 2009), hence, medium and high-pass filtered natural im- ages typically appear completely empty to the human eye without additional contrast maximisation. Even still, such images are not easily recognizable to the human eye due to its and reliance on low frequency information to recognize objects optimally (Appendix D.2 We observed that standard training CNNs on medium or high-pass Fourier- data did not achieve high clean accuracies (Tables 6. A model trained medium-pan Fouteee- CIFARIO achieved a clean accuracy of only 33% on clean test samples whereas the MSF-RES RE--RELLLLREEEDD model's clean accuracy is ~87, and a model trained on high-pass filtered CIFARIO training samples achieved only 155% accuracy on clean test samples. Due to the energy statistics over frequency bands in natural images, training Fourier-filtered data is not successful for all but the lowest frequency bands, where most of their energy resides. In addition, we note that regularizers such as ASFrregulaiization cannot be recreated by Fourier-filtering the training data due to the tendency of standard training be sensitive to certain frequencies.
18

---

Under review as a conference paper at ICLR 2022
E BLACK BOX FOURIER-NOISE CORRUPTIONS
E.1 CIFARIO FOURIER-NOISE CORRUPTIONS
AugMix
Gauesian sugmeniation
04
LSFZEUULAREED
MSF REGULARZED
Figure 13: (CIFARI0) Heat map of error rates for each Fourier-mode corruption. Each pixel in the heat map is the error of the model when the corresponding Fourier-mode noise (e = 4) is added to the inputs. The bottom row displays example images containing the corresponding corrier-oorruttion.
E.2 SVHN FOURIER-NOISE CORRUPTIONS
Baseline
Aughit
Gaussian augmentstion
24
LSFAEEGULRRIEDD
ASF-REOULRREED
4 4
4
4
4
4
4
Figure 14: (SVHN) Heat map of error rates for each Fourier-mode corruption. Fach pixel in the heat map is the error the model when the corresponding Fourier-mode noise (c = 4) is added to the inputs. The bottom row displays example images containing the corresponding Fourier-corrupions..
19

---

Under review as a conference paper at ICLR 2022
F
CORRUPTIONS IN (HENDRYCKS & DIETTERICH, 2019)
F.1 EXAMPLES
Gaussian Noise
Shot Noise
Impuise Noise
Defocus Blur
Frosted Glass Blu
Motion Blur
Zoom Blur
Snow
Frost
Fog
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure 15: Examples of image corruptions curated by Hendrycks & Dietterich 20019 evaluate robustness of vision models.
F.2 CIFAR100-C BENCHMARK
Noise
Blur
Weather
Digital
Method
Clean Gauss, Shot Impulse
Defocus Glass Motion Zoom
Snow Frost Fog Bright
Contrast Elastic Pixel JPEG
Std. Train
76.2
10.9
12.5
8.7
30.5
18.3
41.1
37.7
46.0
30.6
40.0 69.9
29.8
49.6
21.4
48.9
REGULARZED
MSF REGUL ARIZED 62.4
31.8
34.6
16.2
49.6
50.1
44.6
52.9
51.6
48.7
28.0
54.3
9.2
53.4
58.7
54.5
ASF-REGULAY EE--REULLARZED
67.0
18.4
41.4
47.0
26.9
52.6
61.0
Low pass filtered
60.1
32.7
34.9
18.9
59.3
56.6
54.8
58.1
50.2
49.4
27.0
54.1
13.4
57.0
59.9
43.8
AT
1)
Gaussian moise
73.1
25.8
30.8
12.8
32.9
33.5
36.1
39.9
55.4
51.4
30.7
66.6
11.9
$2.3
41.2
59.0
AugMix
41.9
71.8
70.3
47.7
50.0
Table 6: Accuracies for CIFAR100-C corruptions of highest severity level. We note that the LSF- REGULARIZED model significantly more robust than the baseline for blurring corruptions. For noise corruptions, the as the LSF-REGULARIZED model. Similar to CIFARIO-C low-pass filtering and AT also lead to models robust blurring and noise. MSF-REGULARIZED and ASF-REGULARIZED models again more robust to noise and JPEG noise, compared to the ASF-REGULARIZED model.
model is more robust compared to the baseline well
20

---

Under review as a conference paper at ICLR 2022
G
PATCH-SHUFFLING IMAGES
original
2x2
3x3
Figure 16: Patch-shuflling: Images partitioned into squares whose positions are randomly
changed. This operation destroys global structure in the image and is used to evaluate the extent to which model relies global information.
21


Reviews:
[{"review_id": "eH7bCO-lFz", "reviewer": "Reviewer_fjLz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength \n\n1) This paper is clearly written and easy to follow.\n2) The proposed SFS regularizer is clearly motivated. It is more interpretable than existing sensitivity regularization techniques (Section 2.1). It is also generic to be applied to any differentiable model. \n3) The experiments show that the proposed SFS regularizer is robust to Fourier filtering and corruptions, while maintains high accuracy on clean images.\n\nWeakness\n\n1) The experiment datasets and models look insufficient to me. The authors only show results on CIFAR10/100, but other baseline methods (e.g., AugMix) show at least some ImageNet results. \n2) The presented results are bit unconvincing to conclude the method’s effectiveness. In Table 2 and 3, the SFS regularization methods only match with the performance of the “Low-pass filtered\" method, and their performances highly correlate through all types of corruptions. Although the authors point out their technical difference (Section 4.6), I still doubt the advantage of SFS regularization over the simple \"Low-pass filtered\" method regarding the current performance.\n\nQuestions\n\n1) What is the computational cost compared with other baseline methods, considering the regularization term involves Jacobian?", "summary_of_the_paper": "This paper proposes a measure for a model’s spatial frequency sensitivity (SFS) based on the input-Jacobian in the Fourier basis. With this measure, the authors observe standard CNN training biases towards certain particular spatial frequencies consistently across samples. Based on this measure, the authors propose a family of spatial frequency regularization techniques to suppress the model’s sensitivities to certain spatial frequencies. \n", "summary_of_the_review": "The proposed SFS measures are clearly motivated, and provide interesting insights from the frequency-perspective on how existing methods improves model’s robustness. My major concern lies in the insufficient experiments and unconvincing results as stated in the weakness section.\n\n**Disclaimer**: I am unfamiliar with robustness in CV, it's possible I have some misunderstandings. My rating is highly subject to change based on other reviewers’ comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WqCjDxuE4kA", "reviewer": "Reviewer_GEyu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major comments\n\nThe paper claims that “networks trained with our proposed regularizers obtain\nsignificantly improved classification accuracy while maintaining high accuracy on\nin-distribution clean test images.” However, all the regularization methods presented have a clean-accuracy drop of >=7.3% which is very significant, especially on a small dataset like CIFAR. This makes it hard to believe the papers claims that its method is competitive to sota-trained methods (it performs worse than AugMix, for instance). Even training with gaussian augmentation can similarly-high robustness if allowed to perform such a huge IID accuracy drop [2].\n\nAdditionally, the paper mainly tests on small datasets (CIFAR-10) and only uses Imagenet when measuring model sensitivity (never training with the regularizer). Even in this constrained setting, Fig 1 does not show gaussian-trained [1] (or patch gaussian [2] trained) network for imagenet. This lowers my confidence that the method has practical applicability.\n\nThis would not be a concern if the method was a useful tool for drawing even more insights into how models work, but this is not fleshed out in the experiments.\n\n[1] https://papers.nips.cc/paper/2019/file/b05b57f6add810d3b7490866d74c0053-Paper.pdf\n[2] https://arxiv.org/abs/1906.02611\n\nMinor comments\n\nIt would be useful if the authors could validate the sensitivity-measuring methodology by checking that models that have been Low/Medium/High-pass filtered indeed display the sensitivities being measured.\n", "summary_of_the_paper": "This paper proposes a measure of “spatial frequency sensitivity”, where a model’s input-jacobian is fourier-transformed and frequency bands are aggregated to define how sensitive a trained model is to specific fourier frequencies. The authors show this can also be used to explicitly encourage specific frequency-sensitivities, which they call “spatial frequency regularization”.\n", "summary_of_the_review": "The usage of input-jacobian-fourier (which is highest input sensitivity) to measure spatial frequency sensitivity is intuitive and well explained. However, the paper neither justifies its practical applicability, nor does it develop the methods into a fleshed out model analysis to yield insightful conclusions.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "TIzMLkmnH-", "reviewer": "Reviewer_womq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThis paper presents several interesting discussions, in particular, I think the idea of chain rule can be directly applied to the input-Jacobian is a very neat observation. \n\n\n**Weakness**\n\nOn the other hand, I think the paper can be improved on several perspectives, mostly related to how the contents can be presented and discussed. \n\n  - Most importantly, the empirical results are presented in a non-standard way. The bold numbers are used to report performances higher than baseline in this paper, while bold numbers are mostly used to denote the highest performances. \n     \n    - By reading the numbers, I feel like the new method introduced by this method actually shows a fair strength in comparison to other methods except for AugMix. In this case, the authors might be able to argue that their method is good enough since AugMix is a very computation-heavy method specially designed on the benchmarks used in the paper (with evidence that AugMix is not that good for Table 1), but the authors seem to prefer to present the paper in a way that they seemingly want to compare to AugMix where AugMix is designed with their own method inspired by something else. As the result, the empirical results seem weak. \n\n - The presentation of the main idea can also be improved. For example, some illustrations on how P corresponds to the elements in Figure 1 could help understand the paper better. \n\n", "summary_of_the_paper": "The paper is motivated by the challenge of the model's uncontrolled behavior in learning different frequency components of the image data and introduces a new regularization to improve the empirical performances. The new idea, especially the discussion where chain rule can be directly applied, is very interesting. ", "summary_of_the_review": "This paper introduces a new idea and demonstrates the empirical strength of the method, while the method has some empirical virtue, the way the results are presented makes it very hard to appreciate the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6zKyR2PoCna", "reviewer": "Reviewer_pK3z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper strengths: \nCompared with training on fourier-filtered data or applying complex data augmentations, the proposed method is a simpler (and principled) approach to prevent neural networks to bias towards superficial features of the training data.  The motivation and the method sections are well written.  These experiments indicate its effectiveness in learning global structures of the data.  The results in the robustness against the fourier filtering are convincing. \n\nMy concerns:\nMy major concern is about the significance of the experimental results for data corruptions.  This looks to be the main experiment section and AugMix [1] outperforms standard training in all cases but the proposed method does not.  I would agree that the proposed method could be simpler and extendable to other domains but additional experiments were not provided. (only cifar10 & cifar100)\nIn addition, I’m not sure about the significance of outperforming “medium/high pass filtered” methods when there are a few data corruption cases the standard training (lambda=0) outperforms the regularized training. (6/24 cases with LSF in CIFAR100, for example).  While the regularization term is present, the original cross entropy loss still operates on raw images so the model might still be picking up high frequency local features under LSF regularization.  The sensitivity to this weighting hyperparameter between the two loss terms is not discussed.    If I’m not mistaken, baseline methods that are trained on filtered data cannot take advantage of those local features.    \n\nA minor note: \nI suspect AugMix [1] won’t perform as well as SFS in the patch-shuffle experiment but it was not provided.  Is AugMix not applicable in this setting?\n\n[1] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. arXiv:1912.02781 [cs, stat], February 2020. URL http://arxiv.org/abs/1912. 02781. arXiv: 1912.02781.", "summary_of_the_paper": "This paper proposes a novel spatial frequency regularization technique that improves the robustness of training neural networks against superficial fourier statistics in a dataset.  In the loss function, It adds a regularization term that is based on the Fourier-transformed input-Jacobian.  This term could be customized such that the trained model could ignore (be insensitive to) features with specific frequency in the dataset.   The authors define spatial frequency sensitivity using input-Jacobian in Fourier space. \nAlthough this paper focuses on datasets with images, the method is extendable to any n-dimensional data.  Empirically, the method is evaluated for its robustness against fourier filtering, corruptions, and image patches shuffling in CIFAR10 & CIFAR100 datasets.  It shows better results than other baselines in maintaining the classification accuracy of a model against fourier filtering and image patches shuffling.  There do seem to be slight performance gaps from the SOTA AugMix method in image corruptions.  However, the method is considered to be simpler than AugMix and still outperforms other baselines in many cases.   Experiments demonstrate that the proposed method could learn global features present in a dataset.", "summary_of_the_review": "As the extendibility to any n-dimensional data is its big advantage over those complex data augmentation methods, I am interested in seeing some results in other domains.  Given how well AugMix performs against the standard training procedure in data corruptions, I may increase my score if additional experiments are provided to reflect its ease of use/extendibility over methods such as AugMix.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all generally appreciated the idea in the paper. However, the nature of this contribution necessitates an empirical evaluation, and the reviewers generally found this to not be sufficient convincing. My assessment is that this idea can likely result in a successful publication, but will require additional empirical evaluation and analysis as suggested by reviewers. While the authors did add some additional results during the response period, they do not seem to be sufficient to fully address reviewer concerns.
Decision: Reject