Fourier Sensitivity and Regularization of Computer Vision Models
Authors: Kiran Krishnamachari
See-Kiong Ng
Chuan-Sheng Foo
Venue: Accepted by TMLR ()

Abstract: Recent work has empirically shown that deep neural networks latch on to the Fourier statistics of training data and show increased sensitivity to Fourier-basis directions in the input. Understanding and modifying this Fourier-sensitivity of computer vision models may help improve their robustness, hence, in this paper we study the frequency sensitivity characteristics of deep neural networks using a principled approach. We first propose a $\textbf{\textit{basis trick}}$, proving that unitary transformations of the input-gradient of a function can be used to compute its gradient in the basis induced by the transformation. Using this result, we propose a general measure of any differentiable computer vision model's $\textit{\textbf{Fourier-sensitivity}}$ using the unitary Fourier-transform of its input-gradient. When applied to deep neural networks, we find that computer vision models are consistently sensitive to particular frequencies dependent on the dataset, training method and architecture. Based on this measure, we further propose a $\textit{\textbf{Fourier-regularization}}$ framework to modify the Fourier-sensitivities and frequency bias of models. Using our proposed regularizer-family, we demonstrate that deep neural networks obtain improved classification accuracy on robustness evaluations.

Published in Transactions on Machine Learning Research (12/2022)
Fourier Sensitivity and Regularization of Computer Vision Models
Kiran Krishnamachari
kinankGa. nus.edduuuuuu
Institute for Infocomm Research, A A,SAR, Singapore School of Computing, National Universily of Singapore, Singopore
Sco-Kiong Ng
seckiong www.ggguuuuuuu....u...
Institute of Data Science, National University of Singapore, Singapore School of Computing, National University of Singopore, Singopore
Chuan-Sheng Foo
foo__hhuue chuan sheng@ilr.as a-strrrrssstt..rrr edu.
Institute for Infocomam Research, A A,TA,, Singapore Centre for Frontier AI Research, A SSAAR, Singopore
Reviewed on OpenReview: /ttp:e/v/opereeveeet net/ forum? VmTYgjYloM
Abstract
Recent work has empirically shown that deep neural networks latch on to the Fourier statis- lies of training data and show increased sensitivity Fourier-basis directions in the input. Understanding and modifying this Fourier-sensitivie of computer vision models may help improve their robustness. Hence, in this paper study the frequency sensitivity character- istics of deep neural networks using a principled approach. We first propose a basis trick, proving that unitary transformations of the input-gradient of function can be used to compute its gradient in the basis induced by the transformation. Using this result, we pro- pose general mcasure of any diffcrentiable model's of its input- gradiet... When applied to deep neural networks, find that computer vision models consistently sensitive to particular frequencies dependent the dataset, training method and architecture. Based on this measure, further propose Fourier-regularizztion framework to modify the seniii-sensitivities and frequency bias of models. Using our proposed regularizerffamly, we demonstrate that deep neural nerwsrks obtain improved classification accuracy on robustness evaluations.
using the unitary
1 Introduction
While deep neural networks (DNN) achieve remarkable performance many challenging image classification tasks, they can suffer significant drops in performance when evaluated on out-of-distribution (o.o.d.) data. Intriguingly, this lack of robustness has been partially attributed to the frequency characteristics of data shifts at test time relation the frequency sensitivity characteristics of the model (Yin et al.] 2019 Jo & Bengio 2017). It known that distinct spatial frequencics images contain features different spatial scales: low spatial frequencies (LSF) carry global structure and shape information whereas high spatial frequencies (HSF) carry local information such edges and borders of objects (Kauffmann et al. 2014). Morcover, spatial frequencies may also differentially processed in the brain's visual cortex to learn features at different scales (Appendix A We find that when information in frequencies that model relies on is corrupted or destroyed, performance can suffer. Hence, understanding the frequency sensitivity DNN can help us characterise and improve them.
DNNs have been demonstrated be sensitive Fourier-basis directions in the input Tsuzuku & Sato 2019 Yin et al. 2019) both empirically and using theoretical analysis of linear convolutional networks

---

Published in Transactions on Machine Learning Research (12/2022)
Tsuzuku & Sato 2019 In fact, the existence of so-called "universal adversarial perturbations" (Moosavi- distortions that can degrade models' accuracy across inputs and architectures, is attributed to this structural sensitivity. Yin et al. (2019) also showed that many natural and digital image corruptions that degrade model performance may also be targeting this vulnerability. Hence, understanding and modifying Fourier-sensitiiity a promising approach to improve model robustness. While this problem has been studied empirically, the precise definition and measurement of computer vision model's Fourier--enssilvvily still lacks a rigorous approach across studies. In addition, principled method has been proposed to study and modify the Fourier-sensitivie sensitivits of model. Existing works have applied heuristic filters convolution layer parameters (Wang et al. 2020 Saikia et al. 2021 and input data augmentations Yin et al. 2019) to modify a model's frequency sensitivity.
Dexfooli et al.] 2017], simple
In this work, we first propose a novel basis trick, proving that unitary transformations of a function's gradient can be used to compute its gradient the basis induced by the transformation. Using this result, we propose a novel and rigorous mcasure of DNN's Fourier-scnse sensiiivsnsttiiity using its input-gradient represented in the Fourier-basis. We demonstrate that DNNs consistently sensitive to particular frequencies that are dependent on dataset, training method and architecture. This observation confirms that DNNs tend to rely some froquencies more than others, which has implications for robustness when Fourier-statisties change at test time. Further, using our proposed measure, which is differentiable with respect to model parameters, we propose a framework of Fourier-regularizate
to directly modify the Fourier-sensitivittes and frequency bias of a model. We show in extensive empirical evaluations that rouriee-regullarizatoon can indeed modify frequency characteristics of computer vision models, and can improve the generalization performance of models o.o.d. datasets where the Fourier-statisties shifted. In summary, our main contributions are:
1. We propose basis trick, proving that unitary transformations of the input-gradient of any function can be used to compute its gradient in the basis induced by the transformation
We proposc a novel and rigorous measure of model's Fourier-sensitivity basod on the unitary Fourier-transform of its input-gradient. We empirically show that Fourier-sensitivity of model is dependent the dataset, training method and architecture
3. We propose a framework of Fourier-reeularization to directly induce specifie Fourrer--ensitivittes in a computer-vision model, which modifies the frequency bias of models and improves generalization performance on out-of-distribee data where Fourier-statistic are shifted
2 Related work
2.1
Frequency perspectives of robustness
Yin et al. [019 Tsuzuku & Sato 2019) characterised the Fourier characteristics of trained CNNs using perturbation analysis of their test error under Fouricr-basis noise. They showed that a naturally trained model most sensitive to all but low frequencies whereas adversarially trained (Madry et al. 2018) models are sensitive to low-frequency noise. They further showed that these Fourier characteristies relate to model robustness corruptions and noise, with models biased towards low frequencies performing better under high frequency noise and vice versa. Abello et al. (2021) took different approach by measuring the impact of removing individual frequency components from the input using filters on accuracy, whereas OOtiz-Jimenez et al. 2020 computed the margin in input space along basis directions of the discrete cosine transform (DCT). Wang et al.) (2020) made observations about the Fourier characteristics of CNNs in different train- ing regimes including standard and adversarial training by evaluating accuracy on band-pass filtered data, Contrary to these empirical approaches, we propose rigorous measure of a model's Fourier-sensitiviity
2.2 Modifying frequency sensitivity of models
Yin et al. (2019) obscrved that adversarial training (Madry et al. 2018) and Gaussian noise augmentation can induce a low-frequency sensitivity on some datasets. Wang et al. (2020) proposed smoothing convolution filter parameters to induce low-frequency sensitivity in models. We note that such techniques can, in
2

---

Published in Transactions on Machine Learning Research (12/2022)
principle, be undone by subsequent layers of a network. Shhi et al. (2022 proposed similar techniques in the context of deep image priors applied to generative tasks. In addition, data augmentations such as Gaussian noise do not provide precise control over the Fourier-sensitiiity of a model. In this work, we propose a Fourien-regulaiiatton framework to preciscly modify the sourier-sensitivity of any differentiable model.
2.3 Jacobian regularization
Methods that regularize the input-Jacobian of model can be broadly classified into two categories: meth- ods that minimize the norm of the input-Jacobian, and those that regularize its direction or directional derivatives at the input. Drucker & Le Cun (1991) proposed method that penalized the norm of the input-Jacobian to improve generalization; more recently, this has been explored to improve robustness to ad- versarial perturbations (Ross Doshi Velez, 2018 Jakubovitz Giryes 2018 Hoffman et al. 2019). Simard ct al. (1922) proposed "Tangent Prop", which minimized directional derivatives of classifiers in the direction of local input-transformatioos (e.g. rotations, translations; called "tangent vectors") reduce sensitivity to such transformations. Czarnecki et al. (2017) proposed Sobolev training of neural networks to improve model distillation by matching the input-Jacobian of the original model. Regularizing the direction of the imput- Jacobian has also been used to improve adversarial robustness (Chan et al. 2020). In the present work, we regularize frequency components in the input-gradient, to improve performance on out-of-distribution tasks. As such, we are interested modifying the
along certain directions instead its total norm.
3 Proposed methods
Consider an image classification task with input z, labels y, and standard cross- entropy loss function LCE. Let f denote any differ- entiable model that outputs a scalar loss, F()) the unitary discrete Fourier transform (DFT), F-)(() its inverse, and (-) the adjoint of the inverse- Fourier transform, and let zy denote the Fourier- space representation of the inpul, iee. Iy F(x). We denote the input-gradient in the standard basis as Jy(x), and Jy(ī,) as the gradiegradient with re- spect to the imput in the caurier-basis. Let N be the height. of input images (although not necessary, all images used in this work are square).
Preliminaries:
P(n.v)
P(n.v)
(x)
(b)
www-frequencyy (DC)
Protal power square
Tuael power inside circle
Figure 1: dower-matrix P of input-gradien
The zero-shifted (rearrange DC component to centre and high frequencies further from center) 2D-DFT of the input-gradient is denoted F. Since the input-gradient typically has three color channels, they are averaged before computing the 2D-DFT. Fourier coefficients in F are complex numbers with real and imaginary components; F(u, v) = Real(u, v) + i x Imag(u, v), where (u, v) are indices of cocfficients. The power a coefficient is its squared amplitude i.c. P(u, = [F(u, v) Real(u, w)2 +Img(u, 8²2 and the matrix of powers is denoted P (power-matrix). Each coelficient has radial distance r(u,v) v) from the centre of the matrix, r(u, v) d((u, v), (Cu, co)), where (c) denotes the center of P and -(-,-) is Euclidean distance rounded to the nearest integer. Distinct radial distances of coefficients in the matrix are the set of integers f1...., N.....,N//ee N/ and correspond to low to high spatial frequencies, the highest frequeney being limited by the Nyquist-frequency We denote Protal as the total power in P, excluding the zero-frequene frr-ffreuuency coefficient i.e. Protal = En(,,))>= P(u, v). Similarly, we define Protat as the total power in P excluding the zero- zero-frequee pero-frequency coefficient and coefficients with radial distance r(u, v) N/2, i.e. coefficients for illustration). We
DFT notation:
outside the largest circle inscribed in P; Protal = Σ P(u, u) (see Figure
I< <<=(,,))<=N//2
Σ denote P& as the power at radial distance & normalized by Prozats Ps Protal r(u,))=
P(u, v) and Pr as the
power at radial distance & normalized by Protat Pr. Protel r(u,))=k
Σ
u(u,u).
3

---

Published in Transactions on Machine Learning Research (12/2022)
3.1 Basis Trick: Unitary transformations of the input-gradient
In this section, we prove that unilary transformations of the input-gradient of function provide its gradient in the new basis induced by the transformation. We term this the basis trick and use it to compute the Fourier-sensitivity of a model using the Founier-transform of its input-gradicnt. To illustrate the basis trick, consider the computation graph in @egure@where the imput x in the standard basis is mapped to an output via a function f. We introduce an implicit operation (shown red) that maps the Fourier-space representation
of the input to the standard basis via the inverse Fourien-transfom,, i.e. Iy - x. In order to compute the input-gradicnt with respect to input in the Fourier-basis, (z)),, we must differentiate through this implicit operation in the forward graph. Since the inverse-Fourier transform is a unitary operator, we have that below). Hence, even though we do not explicitly
F(){ = ( due to the chain rule (see Corollary [] compute the Fourier-space representation of the input, this shows that the Fourier transform of the input- gradient provides the gradient of the model with respect to the input in Fourier-space. Analogous results can be obtained for other unitary operators such as the discrete cosine transform (DCT) and discrete wavelet transform (DWT) (see Proposition | below). In addition, this approach can be extended to n-dimensional input, e.g. time-serics or 3D signals, by using the n-dimensional Founier-transform. We formalize the basis trick below as a proposition and its corollary when the unitary operator is the Fourier-transfrm..
Definition 1 (Unitary Operators). A bounded lineor operator U H + H on a Hilbert space H is said to be unitary if U is bijective and its adjoint U U-- Morcover, if unitary, U-1 is also a bounded and unitary linear operator.
Lemma 1 (Generalized Chain Rule). Let I be a scalur valued function of vector x, and be a bijective liner operator such that x Ata. Then, (J)(()) is the gradient of f with respect Ia i.e. Jy(Ea) = A" (z)),,) where A is the adjoint of A.
Proposition 1 (Basis Trick). Let f be scalor valued function of vector x, and A be bijective linear operator such that Ara. Then, the gradient vector of f w.r.t Fas Jy(x,) = =A='(J(z))) iff A is unitary.
Proof. Since I = Aza, ()( = (z)) due to Lemma Since A = A iff A is unitary (Definition we have that Jy(x,) A (J)(z)) iff A is unitary.
Corollary 1 (Fourier Basis Trick). If A = F-1, the unitary inverse-Fourier operator such that x F-lny with B being the Fourier-basis representation of x, we have Jz(x,) = Fx))(x)) where F 1)-
Fourier-space
Input-space
Output-space
Xj
x
f(x)
(x,)
(x)
F
Figure Fourier-transfom of input-gradient is the gradient with respect to input in Fourier-space, i.e., Symbols in red represent the input in Fourier-space and need not explicitly computed.
- =
3.2 Fourier-sensitivis of computer vision models
In this section, we define the Fourier-sensitivvity of any differentiable model using its input-gradient represented in the Fourier-basis. Fourier-sensitivitty a mcasure of the relative magnitudes of model's input- gradient with respect to different frequency bands in the input spectrum. As shown in Section 3.1] the input- gradient of function with respect to the Fourier-basis can be computed by the unitary Fourier-transfor of (z). To enable interpretation of the complete input-gradient in the Pourier-basis (see Appendix C.5 for examples), we summarize the information over frequency bands as shown in Figure 3 The Fourier-sensitivity [srs(a, y) of a model with respect to an individual input (I,, y) is defined as
(srs(r. y) = [PI,... PN/val
(1)
4

---

Published in Transactions on Machine Learning Research (12/2022)
Input
Imput-Graiee Gradient
DFT Power Matrix
Compute fraction total power in circular bands
DC Nc-comppoent
X-0tir rodlux circular bund
Figure 3: Computing Fourier-sensitiiity. The input-gradient of the model is Fourier-transfommed to obtain sensitivities with respect to Irequencies. Fourier-sensliiiity is then the vector with components being the proportion of total power in each circular frequency band.
where Ps is the proportion of total power in Fourier coefficients at radial distance l in the power matrix P of Jy(z)). The overall spatial frequency sensitivity (SFS), simply Fourier-sensitivity, of model is defined as the expectation of fsFs(r. (r. y) over the data distribution p(r, v), i.c. fsFs(;;) = (tex)~pl/sps(t..)) (Algorithm Appondix B.1
3.3 Feuuiii-reg of computer vision models
this section, we propose a framework of Fourier-reguuarizatiinn roureierregllaaiiatooon enables control over the Fourier-sensitivity of model by modifying the relative magnitude of a model's sensitivity different frequency bands in the input spectrum. Fouuier-regulaaizattoo can modify the natural frequency scnsitivity of neural networks well their gencralization behavior. Our Fourier-rognlarizer augments the usual cross entropy loss: for single example the new loss is L(x, = Lcr(x, u) + Csps v), where Cars is the proposed regularizer and λsFS is a hyperparameter. Our regularizer penalizes the proportion of power in frequency bands based on the target Yourier-sensitivity As Lsfs is a function of the input- gradient, optimizing requires an additional backpropagation step to compute derivatives with respect to parameters, similar to other graddint-reguulaiizatio methods.
We now define LSPs for four instances of this regularizer: SFS (LSF, MSF, USF, ASF). Low-spatial- frequency (LSF) regularization trains model to be insensitive to medium and high spatial frequencies, seaiiumm-spatial-freqeeeee
(MSF) regularization trains model to be insensitive to low and high spatial frequencies, and tatt-spatial-frquueeeeee (ISF) regularization trains model to be insensitive to low and medium spatial frequencies. These are achieved by penalizing the proportion of power, Pk, in the frequencies we wish the model to be insensitive to. All-spatial-frequenn (ASF) regularization trains a model to be equally sensitive to all frequency bands. The motivation behind ASF regularization model to encourage a model be sensitive to multiple frequency bands instead of being concentrated in small frequency range. Hence, the reg-ragulr loss is defined as the negative entropy of the distribution of power over frequency bands. The definitions of low, medium and high frequency ranges based equally dividing the radius of the largest circle inscribed in the power-matrix P into three equal parts (Figure [p). For ASF- roguaaauuai,,ttion, very high frequency bands, ie.. r(u, v) > N/2 are excluded, which is reflected in the PR terms. Pk. is the proportion of power in frequency bands within the largest circle inscribed in the power power-matrix, P. Concretely, Cars is defined for each of these three cases as follows:
LSF
MSF
HSF
ASF
N/2
Csps
Σ Ps
Σ
Ps
Σ P
Σ Ps log Ps
1
N/6
N/8
kCN/3
k=1
5

---

Published in Transactions on Machine Learning Research (12/2022)
ImagoNet (ReNNet30)
CIFARIO (ReaNwi50)
SVHN (Ro
64.
Stescaed Tritteg
Stuodard Trasting
Wandend Transing
Micessnna Trasung
Aeeersoil Trarring
Adveradaa Transing
Tobs Power Rato
Tobo Thover Retto
6.3
Tobal
tsyckssimmge)
230
90
60
Spatial Frequency (cydessimage
Special Frequescy (cycisslimage))
ImageNee, (RoaNett0)
CIFARIO (RoeNe450)
SVIN (Ro)Ne:50)
Soundand Traitin Ceeoadn Nota, loed
Sandert Tratchng
Stundart Thasing
Gassiaa Noii (0e0.1
Cacodan Koioo
T0ts. Power Etato
Total Power Raatn
-20
190
0800
Spatial Frequency (cycles/image)
THE 222
30
Égotial Frequenny (nyclestmagee
brequency tcyclestimage)
Figure 4: Fourier-sensitivity of (a).(b),(c) standard and adversarial training, (d),(e)((f) standard and Gaus- sian noise augmented training on ImageNet, CIFAR10 and SVHN with ResNet50 backbone.
4 Experiments
We first study below the Fourier-sensilivity of various architectures and training methods across dalasets (Section 4.2p. We found that both training method and architecture can have significant impact on Fourier- sensitivity. We then identify an interesting connection between adversarial attacks and Fouricr-sonsitivity. Further, we study the effects of Fourier-regularizttion on representation learning (Section 4.3) as well as real o.o.d. benchmarks (Section
4.1 Experimental setup
Fourier-sensitivity analysis: Fourier-sensitivie was computed by averaging across 1000 randomly se- lected validation samples for all datasets and shaded areas in plots represent two standard-deviatiooss We computed the Fouricr-sensitivity of pre-trained ImageNct architectures obtained from Pyrorc Image Mod- els (Wightman) 2019). On CIFARIO and CIFAR100 (Krizhessky & Hinton 2009), we trained all models for 150 epochs using stochastic gradient descent (SGD) with momentum (0.9), an initial learning rate of 0.1 decayed by factor of 10 every 50 epochs, weight decay parameter equal to 5 x 10-4 and batch size equal to 128. On SVHN (Netzer et al. 2011 we trained models for 40 epochs using Nesterov momentum with an initial learning rate of 0.01 and momentum parameter 0.9. The training batch size was 128, T.2 regularization parameter was x 10-4 and learning rate was decayed at epochs 15 and 30 by a factor of 10. The following standard data augmentations random-crop, random- coaaan-oooozzzaa-fiiii random-rotation, and color-jitter were used during training.
Fourier-regularization experiments: We demonstrate Fourier-regullarzztioon using ResNet50, Efficient- NetBO, MobileNetV2 and DenseNet architectures. To evaluate the proposed regularizer on high-resolution images, also trained models with images resized to 224 x 224 (ImagcNet-suusset). They were trained with SGD till convergence (Ir=0.1, weight decay=1 x 10-4; Ir mutliplied by 0.1 every 50 epochs); all models converged within 200 epochs. We trained Fourier- regularzee models using ÀSFS = 0.5, which was set as the smallest value that achieved the target Fourier-scnsitivity computed on validation samples independently of performance on target distribu- tion data. We benchmarked against methods that have been proposed to modify the frequency sensitivity of models such as adversarial training and Gaussian noise augmentation to induce low- orequreyynccy sensitivity
subset of ImageNet derived from twenty five randomly chosen classes

---

Published in Transactions on Machine Learning Research (12/2022)
ImageNNe (Standard Traino))
ImngoNol (Stancledd Triaad)
IungoNei (Ativasarial Trainnd)
Biy Trancfer (IT):
DevseNet
Dezasbet
Yaiso Thaadamar 19I1
Net
5455
MariiNNer
Incepcsong
ResNenttA
ResNectt
Total Poweer Rabio
Beckedti
Shutteliee
Total Power Stato
Stullkckct
Mishes
VCcIe
60000
0.0
Total
2013
6.09
699
uo 343 Spatial Proquency icycleshmage)
0.90
369
43
Spatial Proquency (cycles/maage)
Spatial Frequencc (cyelessimage)
Figure Fourrer-senssttivities of multiple architectures after (a).(b) standard training and (c) adversarial training (PGD-L2 (e = 3)) on ImageNet.
Yin ct al. 2019). For these methods, we used hyperparameter values most popular for training robust models in previous works. For adversarial training (AT), we used standard PGD & attacks (e = 1 for CIPAR1O/CIFARI
and 6= for ImageNet-subset, attack-steps = 7 attack-Ir = =/7). For Gaussian noise training, we added i..d. Gaussian noise drawn from N/0 o²) to each pixel during training (a = 0.1). We used the robustness (Engstrom al. 2019) library for training.
4.2 Fourier-sensitivity analysis
4.2.1 Fourier-sensitivity is dependent on dataset, training and architecture
We visualized the Fourier-sensitivity of models trained ImageNet, SVHN and CIFARIO (Figure 4 axes vary across datasets due to different image sizes). We observed that models are sensitive to some frequencies more than others and that this bias is consistent across samples (shaded areas represent two standard deviations across samples). Standard trained ImageNet models are in general sensitive to a wide range of the frequency spectrum with peak sensitivity to mid-range frequencies. The InceptionV33 (Szegedy et al. 2016) architecture is more sensitive to low oow-fruueecciss while Vision Transformer (ViT) (Dosovitskiy et al. 2021 displays sensitivity to mid-range as well as high frequencies (Figure (pguure5p). In contrast, Big Transfer (BiT) [olesnikov et al., 2020 MixNet (Tan & Le 2019b) and ResNet18 (He et al., 2016) models are sensitive to frequencies across the spectrum, with sensitivity lapering off at the high-frequencies (Figure 5p). These results suggest that model architecture can affect Dourier-sensitivin due to their different inductive biases. We further obscrved consistency of Fourier-sensitivity across popular convolutional architectures trained on ImageNet (Figure 5 p). Adversarially trained models (Madry et al., 2018) are most sensitive to low spatial frequencies across datasets and architectures, which suggests that they rely on coarse global features as observed in prior work (Figures 4o. 1o, 5b). Gaussian noise augmented training slightly biases the model towards lower frequencics (Figures
10p) compared to bascline. Training Stylized-Imageeee. proposed by Geirhos et. al. (2019) to train shape-biased models, induces sensitivity to lower frequencies
(Figure II in Appendix C.3 which reflects the increased shape-bias of these models.
Standard trained CIFAR10 models are most sensitive to high frequencies (Figure 4p), similar to CIFAR100 models (Figure
Appendix C.1 In contrast, standard training on SVHN leads to a low-frequency
sensitivity (Figure ( which suggests datasct dependence of Fourior-ccnsitiiity Interestingly, we observed that models trained on common corruptions of CIFAR10 borrowed from (Hendrycks & Dietterich, 2019) display different Fourier-sensitivittes to a model trained on clean CIFAR10 images (Appendix C.2 For example, models trained on images with severe noise corruptions (Gaussian, shot and speckle noise) display increased sensitivity to lower frequencies (Figure 10p), as did models trained on highly Gaussian-blurred,
Glass-blurred, JPFG-compressel and pixelated images (Figure 10a, 10 the Fourier-statistics of these corrupted images.
These changes reflect the shift in
modifies the Fourier-sensitivity of models across datasets (Figure 6 LSF- REGULARIZED models are most sensitive to low-frequencies, AIIIRRD models most sensitive
Finally,
7

---

Published in Transactions on Machine Learning Research (12/2022)
LmageNNtt aubsct (ResSeet50)
CIPARIO (ReeNe15))
SVHN (RooNee50)
[4]
Stendkaed Truntry
Standard Trustng
ES8 Regulariodd
155 Regularized
MST-Segsslrttss
MSS-Segralattes
Total Ratio
AST Citt-Rauuiiioonn lond
Total Power Itetes
ASP Repuioaloo
1122-logsaatees
Titaa Power Rato
3122 Soggssatts
ASB Ree alariedd
6-2
601
Spatial Frequenny (cycles/linge))
Spotial Prequency (cyciosiinags)
Spacial Friquency (cyclneimage)
Figure 6: Fourier-sensttivtty of models trained on (a) ImageNet-subset, (b) CIFARIO, (c) SVHN.
to the mid-frequency range, HSF-REGULARIZED models are most sensitive to high-frequenciis, and ASF- REGULARIZED models are sensitive to a wide frequency range. Further, Fourier-regallaization is demon- strated to be effective across architectures (Figure 12 in Appendix C.4).
4.2.2 Fourier-sensitivity and adversarial attacks
Adversarial attacks are imperceptible perturbations that can drastically reduce the classification performance of computer vision models. Many methods have been proposed to defend against well analyse the prop- erties of such perturbations, including frequency-based approaches. Contrary to opinions that adversarial attacks are strictly a low-frequency or high-frequency phenomenon, we observed that adversarial perturba- tions closely resemble the target models' Fourier-sensitivity, which varies with dataset, training method and architecture as we have shown. This connection naturally ariscs from the fact that common gradient-basod advesariial allack procedures such as PGD (Projected Gradient Descent) (Madry et al. 2018) lypically use the direction of the auput-gradient to generate perturbations. Plotting models' sensitvviss along with the Fourier power-spectra of adversarial perturbations shows this connection (Figure 14 in Appendix C.6 Consistent with observations made by Sharma et al. 2019) that adversarially trained ImageNet models are still vulnerable to low-frequency constrained perturbations in some settings, the Fourier-sensitivity of ad- versarially trained ImageNet models is concentrated in low-frequencier (Figures 41, 4 5p.) Sharma et al. (2099) also observed that low-frequency constrained attacks cannot casily fool standard trained ImageNet models, for which the Fourier-sensilivityy has its peak at medium and high frequencies (Figure 4a. 4p) and are hence less vulnerable to low-frequency constrained attacks. We also observed that adversarial attacks against Fourier-regularized models have matching power-spectra (Figure 14p in Appendix [.6). For example, PGD adversarial perturbations against ASF-REGULARIZED models have power-spectra concentrated in mid-range frequencies. This suggests that adversarial perturbations are not a low or high-frequency phenomena but depend on the Fourier-sensitvvity characteristics of a model.
4.3
Feuuiiz-ne modifies the frequency bias of models
Here we demonstrate that Fourierrregulariiztioon modifies the input frequencies that model relies on using evaluations in different settings.
4.3.1 Validating Fourier-regularizatio using frequency-specific noise
We investigate the sensitivity of Fourier-regularizeed models to Fourier-basis directions in the input using data- agnostic corruptions, which have also been identified threat to model security (Yin et al. 2019 Tsuzuku & Sato 2019 A Fourier-noise corruption is additive noise containing single Fourier-mode (frequency). but affect model performance and can be used to evaluate the sensitivily of a model to individual frequencies (see Figure 7 and Appendix E for examples). We added noise at all frequencies to the respective test sets of SVHN and CIFAR10 to evaluate the sensitivity of models. On CIFAR10, the standard trained model has the highest error at modium-to-high frequencies (Figure 17p. in Appendix The standard trained SVHN model makes the most errors when low-to-medium frequency noise is added to the input (Figure 18 in Appendix F This is in agreement with their respective
These corruptions are monantics-preservvnn
8

---

Published in Transactions on Machine Learning Research (12/2022)
Original
Fourier-Fillees Filtered
Patch-Shuffed
Original
High-Frequency
Med-Frequency
Low-Prequency
4
1 (c)
1 (l)
(a)
(b)
(c)
(d)
(g)
Figure Examples (b): CIFARIO Fourier-filtered (Section
4.3.3
(c):
CIFARIO Patch-shuffled (Section
4.3.2 (e) -gg: Fourier-noii corruptions on SVHN (Section 4.3.1 More examples in Appendix.
Fourier-sensitivittes (Figure ! Similarly, the LSF-REGULARIZED model is most sensitive to low-frequency perlurbations and less so to medium and high-frequency distortions, across both CIFARIO (Figure 17b) and
SVHN (Figure 18p) The ASF-REGULARIZER models are most sensitive to mid-range frequencies (F (Figures
17 18
and ASF-REGULARIZE) models are sensitive to frequencics across the spectrum (Figures 17
18
Detailed heat maps of error rates for noise across the frequencies reflect the modified Fourier-sensitiiilies of Fourien-regularizedd models (Appendix F.. This validates that the Fourier-regulaaiiztioo framework can indeed modify the sensitivity of models frequencies in the input spectrum.
4.3.2 Learning global image features
As low frequency features correspond to large spatial scales while high frequency features are local in nature, Fourier-regularization allows us to bias the scale of features used by model. Here we explore the extent to which Fourier-regularize models use global features by measuring their classification accuracy on patch- Wang et al.
shuffled images, which have previously been used by Mummadi et al. 22021 Zhang & Zhu (2019) :
(2019). Patch-shuffling involves splitting an image into & x squares and randomly swapping the positions of these squares. This is intended to destroy global features and retain local features; larger values of k retain less global structure in the image (see Figure 7 and Appendix for examples). As such, models that rely more global rather than local structure suffer more from patch-shuffling. Hence, lower accuracy suggests increased reliance on global structure. We observed that LSF-REGULARIZED models, which most sensitive well as adversarially trained models, suffered large drops in accuracy, which suggests they rely on global structure images (Table 1]. This contrasts with standard trained, HSF-REGULARIZED and Gaussian noise augmented models, which retain higher accuracy under patch-shuffling. This reflects their bias towards learning local features instead of global structure on these datasets.
to low-frequencies,
Table 1: Accuracy of ResNet50 patch-shuffld CIFAR10 and CIFAR100 test sets.
CIFAR10
CIFAR100
Method
h == 2
h =
k == 2
k = 3
Std. Train
66.5
45.8
39.9
21.4
Gaussian Noise
62.9
44.5
34.4
18.0
HSF-REGULAARIEED
60.7
38.7
37.3
19.0
ASF-RECULARIZED
43.2
30.6
23.4
13.0
ANS-RREGLAD
46.8
33.1
24.1
13.0
ASF-REGULARIZED
46.8
32.6
29.0
15.0
AT (PGD l,,
45.2
35.0
19.1
11.1

---

Published in Transactions on Machine Learning Research (12/2022)
4.3.3 Robustness to Fourier-filtenie fiiterr-i
Jo & Bengio (2017) showed that DNNs have a tendency to rely on superficial Fouricr-statistics of their training data. In the vein of generalization evaluations they performed, we generated bernantics-preeevvii Fourier-filtered test images using radial masking in frequency space (see Figure 7 and Appendix D.] for examples). A mask radius determincs Fourier components that are preserved with larger radií preserving more components. We use (Cu, Cy) to denote the centre of the mask and d(.:-) to denote Euclidean distance. The mask is applied on the zero-shifted output of the Fourier transform of each image, denoted X, followed by the inverse transform, i.e. X fittered = (F Mr), where @ is the element-wise product. Formally, the
radial mask is (1, v) = 1, ifd((u, ()) Cx) r Fourier-filtering is performed on cach color channel
0, otherwise
independently. On ImageNet--scale images, LSS-RECULARIZATION is robust to significant low-pass filtering = 37), with just ~1% drop in accuracy, whereas the baseline model drops by ~7% (Table 2 A standard trained CIFARIO model suffers up to a 75% drop in accuracy on highly low-pass filtered data as it relies on high frequency information that is no longer present. On the other hand, other Fourier-regularized models perform robustly against Fourier-filtering even on severely low-pass filtered images (r = 5), achieving an accuracy of 78.3% compared to the standard trained model's 18.6%. This shows that LSP-REGULARIZED CNNs are able exploit low frequency features more than other models in the absence of high-frequency features. The adversarially trained (AT) model is significantly more robust than the baseline model due to its low-frequency sensitivity but not robust as the LSS-REEUL model. Gaussian noise augmentation does not provide significant robustness to Fourier-fllterngg. Both MSF- SESS-REGULRRZED and ASF ASF-REGULARIZED models also provide significant robustness to Fourier-filtering while not as much as the comparable to or more robust than the standard trained model. We further observed that other architectures EfficientNetBO (Tan & Le. 2019a). MobileNetV2 (Sandler et al. 2018)) and DenseNet (Huang et al. 2017]- are also significantly vulnerable over baseline methods these architectures well (Table 3 plots in Appendix C.4
On CIFAR10, the ASF-REGULARIZED model performs robustly
model. The HSF-REGULARIZEN model was
courier-filtering couuiee-regularizationn can similarly improve robustness
4.4 reguliz-eg confers robustness to real distr-iiit data shifts
Here we explore the robustness of courier-regularization on real o.o.d. data. Image corruptions in deploy- ments of computer vision models can cause unfavorable shifts in the Fourier-statistics of data (Yin et al., 2019). For example, computer vision models deployed in vehicles may encounter motion blur due to ment, which disrupt high-frequency information in images. Similarly, digital corruptions can cause similar effects on Fourier-statistics, such as JPEG compression artifacts and pixelation in low resolution set- tings. On ImageNet-C (Hendrycks & Dietterich 2019 Fourier- regularzigg confers robustness to multiple corruptions. L.FFREG (A=1) was most robust to blur corruptions, which carry most information in the low frequencies. ASF-REG (A=0.5) provided significant robustness to weather and digital corruptions (Table which suggests that sensilivity to a broad range of frequencies needed to be robust to these corruptions.
Table Accuracy of ResNet50 on Pourier-filterdd ImageNet-subbset, CIFAR10 and CIFAR100 test sets.
ImageNNet-subset
CIFAR10
CIFAR100
Method
clean
37
20
clean
11
7
T
clean
r= 11
7
Std. Train
84.6
77.2
54.2
94.9
78.1
24.9
18.6
76.2
49.7
14.1
6.6
L.F-REGULARIZED
84.4
83.0
67.8
87.1
86.2
84.4
78.3
62.5
61.5
58.0
46.8
MSF-REGUTAARIZM
86.2
74.0
59.0
90.6
86.3
71.5
46.2
70.7
62.2
46.4
18.6
IIS-REGULARIZED
87.3
78.2
52.2
93.5
76.4
34.5
25.1
75.8
50.2
18.2
9.8
ASP-REGULARIZEE
88.5
82.4
65.3
87.9
85.0
69.3
45.0
67.0
62.1
41.1
19.8
AT-PGD
81.8
75.8
54.3
81.6
80.2
76.1
67.5
58.8
56.8
50.0
40.2
Gaussian-noiss
84.8
74.6
36.7
94.5
84.4
32.4
19.5
73.1
61.9
27.7
11.6
10

---

Published in Transactions on Machine Learning Research (12/2022)
The HSF-REG (A=0.5) model was more robust to weather and digital corruptions compared to blurring, which requires a low-frequency bias. Hence, modifying the Fourier-sensitivity can improve robustness under multiple o.o.d. shifts that affect model robustness.
5 Discussion
5.1
Fourier-regularizerr selection
Selecting the regularizer (i.e., LSF, MSE, USF, ASF) that gives the best performance on a (shifted) target data distribution may be done using pross-valdattion if labeled data from the target distribution is available. Otherwise, since model or regularizer selection the absence labeled data from the target distribution is generally a hard and unsolved problem, we suggest choosing the Fourier-regularizer based on prior knowledge about the frequency bias of the learning task the target distribution. For example, have shown that many common image corruptions such various forms of blurring, LSP-RECULARIZED models perform well due to the loss of high-frequency information under blurring (Section 4.4 This agrecs with previous work that has analysed the spectra of corrupled images YYin et al. 2019 As demonstrated in Section 4.3.2 LSS-REGGLARIZED models are also more reliant on global features, which are generally robust to local changes in image texture (Geirhos al., (219)). On high-rcsolution images, we showed that encouraging models to use more frequencies using AST-REGULARIZZTIOON can improve clean accuracy (Table 4
5.1.1 Asrs hyper-parameter selection
Fourier-regullriiztioon requires choosing the frequency bias as well as the hyperparameter Asrs- We note that when ÀSFs = 0, Fouriierregullariation is equivalent to standard training. Hence, very small values may not modify the frequency bias significantly. We found that useful heuristic to set the parameter as small possible to achieve the target frequency bias, which be measured by computing the Fourien-sensitivity of the model using training validation samples independently of performance the target distribution. Values larger than this can unnecessarily decrease clean accuracy further without improving accuracy on the target distribution. For SST-RECULARIZATIOO on CIFAR10, we found that increasing ASFS from 0 to 0.5 gradually mudges the model towards low-frequencies (Figure 20 in Appendix G As we increased Asrs further from 0.5 to 1, clean accuracy decreased further without modifyynn the Fourier-sensitlivil (Table in Appendix Gp. Strietly restricting the model to have a particular frequency bias using large values of Agrs may overly constrain model capacity. This procedure can be performed to identify optimal ÀSFS values even in the absence of labeled target distribution data.
5.2 Fourier- reguliziin and clean accuracy
Fourier-regularizstion can affect the frequencies utilized by models in given dataset. The effect of Fourier- regularization clean accuracy depends the dataset and the chosen frequency range (e.g., LSF, HSF, ASF). On high-resolution ImageNet-scale images (224 x 224) observed that encouraging the model to use wide range of frequencies using AAS-RREGULARIZATIONN improved generalization performance over
Table 3: Evaluating Fourier-filtered CIFAR10 using other architectures.
NfficientNet.B0
MobileNett
DenseNet
Method
clean
11
5
clean
11
clean
11
r=7
Std. Train
89.9
76.1
30.2
24.0
92.6
74.5
27.6
18.3
94.0
69.6
19.3
16.4
L.F-REGULARRIED
84.1
83.5
80.2
68.3
81.7
81.5
78.3
67.4
86.2
85.7
80.7
69.0
ASSEREEGUA
88.7
86.6
55.5
31.2
89.0
87.6
68.7
38.7
90.6
89.6
72.5
38.5
ISF-REGULARIZED
90.5
73.3
28.6
19.2
90.3
83.2
52.2
36.1
92.9
80.5
35.0
23.6
ASF- ASP-RECULARIZED
89.5
77.4
30.6
24.1
79.0
73.0
44.9
28.6
88.2
85.5
69.7
46.7
Gaussian-noise
89.3
78.4
35.4
26.1
91.2
79.3
41.0
26.5
93.2
82.3
30.4
21.1
AT-PGD
72.3
71.5
68.4
63.1
82.0
80.2
75.9
67.0
81.9
80.9
76.3
67.9
11

---

Published in Transactions on Machine Learning Research (12/2022)
Table Accuracy of ResNet50 on clean and corrupted (severity 2) test set of ImageNet-subset.
Blur
Weather
Digital
Method
Clean
Defocus
Glass
Motion
Zoom
Snow
Frost:
Fog
Brightness
Contrast
Flastic
JPEG
Std. Train
84.6
59.2
68.8
71.4
69.8
46.8
51.7
44.6
79.8
40.9
71.8
82.1
74.1
LSF-REG (A=1)
83.8
71.5
77.6
76.6
76.4
54.2
58.6
46.7
79.6
46.6
75.3
83.7
76.5
AISE-REb (A=1) 1)
8G.8
58.3
64.4
70.1
68.2
47.5
51.0
39.6
81.2
37.8
71.1
79.7
76.0
DSF-REG (A=1]
86.9
56.3
65.0
67.6
63.9
48.1
53.7
41.8
80.5
38.1
69.9
81.1
78.2
ASP-REG (A=1)
85.5
62.3
68.6
72.5
70.2
50.3
55.8
40.5
80.7
42.4
72.1
81.9
75.5
LSP-REG (A=0.5)
84.4
63.4
73.0
72.7
70.6
52.1
56.2
43.5
80.5
42.4
74.2
82.2
75.9
MSF-REG (A=0.5)
86.2
61.6
67.5
72.0
71.3
52.0
50.8
53.1
81.6
46.7
72.2
82.1
78.0
REF- (A=0.5)
87.3
60.4
68.9
71.0
70.6
52.6
58.6
51.1
83.3
49.8
72.3
83.6
77.8
ASP-REG (A=0.5)
88.5
65.8
74.3
74.7
73.9
58.2
63.5
59.7
84.6
53.0
74.6
86,8
78.1
Ganssian-noise
84.8
38.8
55.4
61.8
37.6
47.4
51.0
35.0
81.4
27.1
67:2
78.1
71.4
AT-PGD
81.8
58.4
65,5
67.0
66.3
50.3
47.8
13.7
79.0
18.5
66.5
78.3
79.0
the baseline (88.5% vs 84.6%), did HSE-REGULARIZATTON (Table 2 On SVHN, an easier task, Fourier- regularization did not have significant effect as baseline models already achieve high clean accuracies (~96%) (Table G in Appendix E.3 CIFAR10 and CIFAR100 are more challenging small image tasks that require high spatial frequencies (HSF) to maximise clean accuracy. Hence, the HSF-REGULARIZHH model had high clean accuracy while we observed a drop in the clean accuracy of LSF,MSF,A, regularized models, although they performed better on o.o.d. data. In summary, Youriee-regulariiatiin a generic framework that can be used to improve performance in both iidd and o.o.d. settings.
5.3 Feuriee-regullriiztion vs training on Fourier-filtered data
Here we contrast Fourior-regularizttoon and training on Fourier-filtered data to modify the frequency bias of models. We note that Fourier-regularizatio cannot be replicated by training on Fourier-filtered data. Low-pass filtered images completely discard information higher frequencies, which may not be desirable. Morcover, natural images, the amount of energy in frequency bands falls off rapidly high frequencies [Hyvärinen et al., 2009), hence, medium and high-pass filtered natural images typically appear completely emply to the human eye without additional contrast. maximisation and are stiill not easily recognizable (see Figure 16 in Appendix D.2). Hence, training on medium-past pass Fourier-filtered CIFAR10 achieved clean accuracy of only ~33% whereas the MSF-REGULARIZED model's clean accuracy is ~90% (Table 5 in Appendix D.2). Similarly, training on high-pass filtered CIFARIO training samples achieved only ~ 15% accuracy on clean test samples, while HSF-REGULARIZATTON AEF-REGGUAAIIZATIOO can achieve 93.5% (Table 2 Due to the energy statistics across frequency bands in natural images, training on Fourier-filtered data is not successful for all but the lowest frequency bands, where most of their energy resides. On the other hand, the Fourier- regularization framework allows controlling the sensitivity to each frequency band. In addition, note that ASF-REGULARIZATION cannot be realized using Fourier-files filtering alone.
6 Conclusion
We proposed a novel basis trick and proved that unitary transformations of a function's input-gradien can be used to compute its gradient in the basis induced by the transformation. Using this result, we proposed a novel and rigorous measure of the Fourier-sensitivity of any differentiable computer vision model. We explored Fourier-sensitivie of various models and showed that it depends on dataset, training and architecture. We further proposed framework of Fourier-repularizaton that modifies the frequency bias of models and can improve robustness where Fourier-statisties of data have changed. We demonstrated that Fourier-regularizall
is effective on different image resolutions, datasets (Table 2 as well as archilectures (Table 3p. More broadly, Fourier-sensitivity and regularization can also be extended to other data modalities like audio and time-scrics, where Fourier analysis of machine learning models may also be useful. As Fourier- analysis an important and fundamental toolkit, the analysis and control of machine learning models enabled by work may prove to be valuable for learning tasks beyond those explored in this paper.
12

---

Published in Transactions on Machine Learning Research (12/2022)
Acknowledgments
We thank Bryan Hooi and Wenyu Zhang for helpful discussions about the project as well as feedback on an early draft of the paper. Individual fellowship support for Kiran Krishnamachari was provided by the Agency for Science, Technology, and Research (A (A*STAR), Singapore.
References
Antonio A. Abello, Roberto Hirata, and Zhangyang Wang. Dissecting the High- High-Frequency Bias in Convo- lutional Neural Networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 863-871, June 2021.
F W Campbell and J G Robson. Application of Fourier analysis to the visibility of gratings. The Journal of physiology, 197(3):551 566, August 1968. ISSN 0022-3751. doi: 10. 10.1113/iphysiol.998 196.s000877 URL https:///ubmees: ncbi nlm. nih gov/5666169
FW Campbell and JG Robson. Application of fourier analysis to modulation response of eyc. In Journal of the Optical Society of America, volume 54, pp. 581, 1964. Issue: 4.
Alvin Chan, Yi Tay, Yow Soon Ong, and Jie Fu. Jacobian Adversarially Regularizod Networks for Robustness. In International Conference on Learning Represenlations, 2020. URL hstps://openreview.ncer formm?
id=HkeOVIrKPS
Wojeiech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev Training for Neural Networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems, vol- une 30. Curran Associates, Inc., 2017. URL .df.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021. URL
H. Drucker and Y. Le Cun. Double backpropagation increasing generalization performance. In IJCNN- 91-Seattle International Joint Conference on Neural Networks, volume ii, pp. 145-150 vol.2, 1991. doi: 00.110/LJCCNN.191.1555
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (Python Library), 2019. URL https://tt thub. hwt.c///addryyaa//rrorussees
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trainedd CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https: net/forummidee (iw.oorummnd-BBgg990000009
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1109 10.1109/CVPR.C. /CV/CC2VR........
ISSN: 1063-6919.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Cor- ruptions and Perturbations. In International Conference on Learning Representations, 2019. URL
https://openreviiew wwt.://ooruumiiddddbettt...
Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust Learning with Jacobian Regularization. arXiv:1908.0m 0200900027922 fes, stot|, August 2019. URL http:///axxiv.ooraab///ee// 02729 arXiv: 1908.02729.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densoly Connected Convo- lutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognation (CVPR), July 2017.
13

---

Published in Transactions on Machine Learning Research (12/2022)
Aapo Hyvärinen, Jarmo Hurri, and Patrick O. Hoyer. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Springer Publishing Company, Incorporated, 1st edition, 2009. ISBN 1-84882-490-4.
Daniel Jakubovitz and Raja. Giryes. Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Jason Jo and Yoshua Bengio. Measuring the tendency of CNNs to Learn Surface Statistical Regulari- ties. arXic:1/11.11561 fes, stat), November 2017. URL http:///arav..orr//a///111111 11561 arXiv: 1711.11561.
B. Julesz and T. V. Papathomas. On cuntial-frequence channels and attention. Perception & psychophysics, (6(4):398-390 October 1984. ISSN 0031-5117. doi: (..7758//603202795
Place: United States.
Louise Kauffmann, Stephen Ramanoël, and Carole Peyrin. The neural bases of spatial frequency processing during scene perception. Frontiers in integrative neuroscience, 8:37, 2014. ISSN 1662-5145. doi: 10.3389/ f0in0.2014.00037.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. In Audrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12350, pp. 491 507. Springer International Publishing, Cham, 2020. ISBN 978-3-030-58857-J 978 978-3-030-58558-7. doi: 10.1007 177-3-030058858-7 29. URL
Series Title: Lecture Notes in Computer Science.
Alex Krizhevsky and GeofTrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, Technical report, University of Toronto, Toronto, Ontario, 2009. Backup Publisher: University of Toronto.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Rosistant to Adversarial Attacks. In International Conference on Learning Repre- senlalions, 2018. URL hattps://openreviieet wwt.ee/ee..eeoomoo//r/rddo//.aa
Seyed-Mohsen Goosavi-Dezfooli: Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal Adversarial Perturbations. In Proceedings of the IEFE Conference on Computer Vision and Pallern Recognation (CVPR), July 2017.
Chaithanya Kumar Mummadi, Ranjitha Subramaniam, Robin Hutmacher, Julien Vitay, Volker Fischer, and Jan Hendrik Metzen. Does enhanced shape bias improve neural network robustness to common corruptions? In International Conference Learning Representations, 2021. URL httpps://openrevies het//forumiid-yylxllaa33333
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Un- supervised Feature Learning 2011, 2011. URL housenumbers. pdff
Guillerno Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi, and Pascal Frossard. Hold me tight! Influence of discrininative features on deep network boundaries. In Advances in Newral Information Pro- cessing Systems, volume 33, pp. 2935 2946. Curran Associates, Inc., 2020. URL httpp:://proceedings. http:////00000/////..ooo//.c6660 Ats.aaa. Abstract.html
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing Network Design Spaces. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition (CVPR), June 2020.
Andrew Slavin Ross and Finale Doshi-Velez. Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients. Tn AAAT, 2018
14

---

Published in Transactions on Machine Learning Research (12/2022)
Pia Rotshtein, Andrew Schofield, María J. Funes, and Glyn W. Humphreys. Effects of spatial frequency bands on perceptual decision: it is not the stimuli but the comparison. Journal of vision, 10(10):25, August 2010. ISSN 1534-7362. doi: 10.1167/10.10.25, Place: United States.
Tonmoy Saikia, Cordelia Schmid, and Thomas Brox. Improving robustness against common corruptions with frequency biased models. In 2021 IEEE/CVI International Conference on Computer Vision (ICCV), pp. 10191-10200, 2021. doi: 10.1109. AL119//ICC49999 201.92222.220000.....
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. Tn 2018 TEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018. doi: 10.1109 CBB.0//CVV..228..001101
P. G. Schyns and A. Oliva. Dr. Angry and Mr. Smile: when categorization flexibly modifics the perception of faces in rapid visual presentations. Cognition, 69(3)2243 265, January 1999. ISSN 0010-0277. doi: 10.1016/s0010-00 02770(27777700))
Place: Netherlands.
Yash Sharma, Gavin Weiguang Ding, and Marcus A. Brubaker. On the Effectienness of Low Fre- quency Perturbations. In Proceedings of the Twenty-Bighth International Joint Conference Artifi- cial Intelligence, pp. 3389-3396, Macao, China, August 2019. International Joint Conferences on Arti- ficial Intelligence Organization. ISBN 978-0-1992411---11
doi: 10.24963/ijcai.. 2019//47 URL https:
//ww.. wwwai.
Zenglin Shi, Pascal Mettes, Subhransu Maji, and Cees G. M. Snoek. On Measuring and Controlling the Spectral Bias of the Deep Image Prior. Int. J. Comput. Vision, 130(4):885 908, April 2022. ISSN 0920- 5691. doi: 0000//s12233021-011777 USA Publisher: Kluwer Academic Publishers.
URL https://doi.
Place:
Patrice Simard, Bernard Victorri, Yann LeCun, and John Denker. Tangent Prop A formalism for speci- fying selected invariances in adaptive network. In J. Moody, S. Hanson, and R. P. Lippmann (eds.), Advances in Neural Information Processing Systems, volume 4. Morgan-Kaufimam 1992. URL https: //proceedings. neurips. iss cc wwwpeeeddinnnnnnniii.eeeeeee
.dff
Christian Szegedy, Vincent Vanhoucke, Sergey Iolle, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2822 June 2016.
doi: 10.1109 10.1109/CVPR..C 201..2006.300...
ISSN: 1063-6919.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Kamalika Chandhuri and Ruslan Salakhutdinov (ods.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, June 2019a. URL https::/ppoceeeeeeeee
mlr, wwwpppeessre... 1wt.97/tan19aanttem.
Mingxing Tan and Quoc V. Le. MixConv: Mixed Depthwise Convolutional Kernels. In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff UK, September 9-12, 2019, pp. 74. BMVA Press, 2019b. URL https::/bmmcc/0:: org/wp ww- wontent//ploodd/////paeeeee/eeee
Yusuke Tsuzuku and Issci Sato. On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 51-60, Long Beach, CA, USA, June 2019. IEEE. ISBN 978-1-72813-293- doi: (A1109/CCPPR.201.00000 CRL https://ieeexplore ieee. org/ http//dcmmnntt8844000
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Representa- tions by Penalizing Local Predictive Power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceeddnng: www.sseeeeiiiiiiceeeeeeoee htt.rppsss.cpppppp////////ee pdd..
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks. In 2020 IEFE/CVP Conference on Computer Vision
15

---

Published in Transactions on Machine Learning Research (12/2022)
and Pattern Recognition (CVPR), pp. 8681-8691, Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817- 168-5. doi:
URT httpss://eeeeepplrrrrss ieee www..ee.eee...ooo.......eeeo... (tteee.rrg/cccmmme/95566622//
Ross Wightman. PyTorch Image Models, 2019.
URL https://githubbccm/// htt.://o//wwwhaaa...
pytorch- image- models Publication Title: GitHub repository.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Fkin D. Cubuk, and Justin Gilmer. A Fourier Perspec- tive on Model Robustness Computer Vision. In Proceedings of the 33rd International Conference on Newral Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, 2019.
Tianyuan Zhang and Zhanxing Zhu. Interpreting Adversarially Trained Convolutional Neural Networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7502-755 -7-11, PMLR, June 2019. URL https://proceeiing..mtt www.ss..rsssvvveeeeeee v9/ssss////// 2ta/s///aaag9..... html
16

---

Published in Transactions on Machine Learning Research (12/2022)
A Spatial frequency channels in the brain
To further motivate the spatial frequency perspective of visual learning, we briefly describe some relevant findings from neuroscience and vision research. While the brain does not strictly perform a Fourier-analysis of visual scenes, there has been mounting evidence over decades for spatial frequency channels in the human visual system that are physiologically independent and selectively responsive to distinct spatial frequency bands (Campbell & Robson, 1964
It has been posited that the use of different spatial frequency channels are determined by the demands of a given visual task through attention mechanism (Schyns & Oliva 1999: Rotshtein et al. 2010 Julesz & Papathomas 1984). Spatial frequency channels enable to attend to different spatial scales in scene at fixed viewing distance, similar to the focus lens in camera
1968).
(Figure 8 Similarly, computer vision models develop a Sourien-sensitivig (Section 3.2) that is dependent
on the dataset, architecture and method used to train them.
EEREEEEREEEEEEEEEEEEEEEEEEEEEEEE EREEEEEEEEEEEEEEEEEEEEEEEEEEEEE FEFREEEEEEEEEEEEEEEEEEEEEEEEEEEE FFFEEEEEEEEEEEEEEEEEEEEEEEEEEEEE EFR EFREEFER TTT EF EE KPEF EFEFFT EF EP EF EE EX
RF WWW:
EF / EF FFFEEEEEEEEEEEEEOOOOOOOOOEEEEEEE /
EP FFFEE//SSSSSTTTAPP00oooooa000eA0 D EF TFF. F EP EE T 
EF T EF TAT.TTTAAATTTAATTTTTTTTTTTTTTTTT T EP WFw EF EF FEFFEEEFFFFFFFEEEEEEEEEEEEEEEEEE EFFFE PEFFEEEEEEEEEEEEEEEEEEEEEEEEEEEE EFR
Figure 8: Letters multiple spatial scales. This image comprises the letters 'o', 'n', 'E' and 'F' at small spatial scale (corresponds to high-frequency). The letter "T" is also visible at larger spatial scale (LSF) formed by the specific arrangement of the letters 'o', 'n'. Identifying these letters requires processing features at multiple scales, enabled by distinct spatial frequency channels in our visual cortex. Our ability to recognize only one of these scales at a time is evidence for the physiological independence of spatial frequency channels in the brain. Image based on Julesz Papathomas (184)).
17

---

Published in Transactions on Machine Learning Research (12/2022)
B
seusivt-sen
B.1
Pseudo-code
Algorithm sourivr-sensitiiity of model
Input: Labeled samples L = ((1, y)) model with trained parameters 0 Output: Estimated Fourivr-sensitvvity of model, (sfs(;:)
for i= 1 to n do
Compute loss (c;(f(x;); 34) {forward pass}
Backpropagate LCE to obtain 8CCB (input-gradient, averaged across color channels]
dz;
0CCr F( (fCCn) {unitary 2D DFT of input-gradient]
Say,
Uz.
fars( Vi) = [P& for k-1 to N/v2] {soc Equation excludes DC component)
end for
fsrs(:0) 1/ Ei= fsrs(E;1 (1) festimated Fourier sensitivity of model}
18

---

Published in Transactions on Machine Learning Research (12/2022)
C Supplementary plots
C.1
Fourier-sensitivities of ResNet50 models trained on CIFAR10 and CIFAR100
CIFARIO
CIFARI00
Standard Training
Standard Training
0.35
Reguaagll
9.25
LSF-Regularized
MSF-Regularized
MST-Regularized
Total Power Ratio
0.30
Regulariid ASF-Regularize
Total Power Ratio
0.30
HSF-Ragularized ASP Regulagardd
0.25
0.26
0.20
0.20
0.15
0.15
0.10
0.25
10 11 20
Spatial Frequency (cycles/image)) 20
10
Spatial Frequency (cycles/image)
Figure 9: Fourier-sensitivity of models trained with various regularizers on CIFAR10 (left) and CIFAR100
(right). The shaded region represents two standard deviations.
C.2 Training on corrupted CIFAR10
(a)
Original
(b)
0.45
Original
Gausaian Blar
Glass Blur
Shoi Noise
6.91
Motion Blar
Total Power Batio
0.33
Trpuusnn Naise
Defocus Blur
Spockle Noise
Zoom Blur
Power
0.33
0.28
0333
0.18
0.00
038
15
Spotial Frequency (cycles/inege)
Spatial Frequency (cycles/imege))
(0)
Original
(d)
Originel
6.33
Pivelute
0.20
Snow
Saturate
Spatter
Ratlo
6.33
JPBG Compression
Total Power Ratio
4.20
Fog
Flastic Transform
Brightness
0.25
8.20
8.15
8.10
8.00
100
Spatial Prequency (cycles/mmage)
Spatial Frequency (cycles/Image)
Figure 10: Fourier-sensitivity of ResNet50 models trained on the CIFAR10 training set distorted by cor- ruptions derived from the CIFAR10-C (severity 5) datasct. The shaded region represents two standard deviations.
19

---

Published in Transactions on Machine Learning Research (12/2022)
C.3 Fourier- sensitivity of model trained Stylized ImageNet
ImageNet (ResNet550)
ImageNet
0.605
Stybized ImageNet
Total Power Ratio
0.032
8.623
0.009
0.015
0.005
8.005
8.000
120
:20
Spatial Frequency (cycles/image)
Figure 11: Fourier-sensitivity of ResNet50 trained on ImageNet and Stylized IeageNet.
C.4 reguaiii of other architectures
EfficientNerB0
MobileNetV2
Std.
LSP Rogulaggldd
15P-Rogularized
Regulatzed
MSF-Regularized
Total Power Ratio
ASF-Ragularize
Total Power Ratio
ASF-Ragulartzed
HSF-Rogulartood
HSF- Regularted
6.
Gaussian Noise
Gaussian Noise
ta
Spatial Frequency (cycles/image)
Spatial Frequency (cycles/image)
DenseNet
RegNet
4.5
15
Std. Trsin
Std. Train
J.F-lagularszed
JSF-Regularized
5.4
MSF- Regularrdd
MSF-Regularized
Total Power Ratio
ASF-Ragularized
Total Power Ratio
ASF-Rogullribed
HSF-Rag Rag--aaglar
HSP-R DSS-Regularized
Gaussian
Gaussian Noise
ACPGD
ALPCD
Spatial Frequency (cycles/image)
Spatial Frequency (cycles/image)
Figure 12: Fourier-sensitivitty plots for other architectures (FffficientNee (Tan 2019a). MobileNet V2
Sandler al. 2018). DenseNet (Huang et al. 2017), RegNet (Radosavovie al. 2020 trained CIFAR1O.
20

---

Published in Transactions on Machine Learning Research (12/2022)
C.5
Imput-gradient in Fourier-basis of models trained on CIFAR10
Std. Train
Gaussian Noise
AT-PGD
15-
15
15
20
20
25
25-
25
30
30
25
30
J.S--Regularized
MISF-Regularieed
ASF-Regularinn
10
15
15
20
20
25
25-
25
30-
30
10
15
Figure 13: Input-gradients of ResNet50 models trained on CIFAR1O, averaged across 1000 randomly chosen validation samples. Low frequencics are close to the center, high frequencies are further from the center.
21

---

Published in Transactions on Machine Learning Research (12/2022)
C.6
Yourier-sensitivee and adversarial attacks
ImageNet (ResNet50)
CIFARIO 10 (ResNer50)
(a)
Buseline
(b)
Beseline
Buseline (PCD uttack)
Beseline (200 attadl
Adv. Trais
LSF- -55-eegabrrred
Adv. Trais (PGD attaik)
ESP Regulerrr (NDD attockt
MSP Regularadd
Total Power Ratio
MSF Regularaze (PCD attolai
Total Power
0.010
0.1
6.005
8.000
0.9
100
100
Spatial Frequency (cycles/image)
Spatial Frequency (cycles/image)
Figure Power-spectra of adversarial perturbations align with Fourier-sensitivity of models. attacks (e 3) were used for each model.
PGD
22

---

Published in Transactions on Machine Learning Research (12/2022)
D
Fourier-filtering
D.1 Radial Tourier-filtering
r=11 unfiltered
Figure 15: First image in each row the mask in Fourier space (lowest frequency at centre). White pixels preserve and black pixels set Fourier components to zero. Top row are original CIFAR10 images, other rows are Fourier-filtere with different radial masks.
23

---

Published in Transactions on Machine Learning Research (12/2022)
D.2 Band- pass Fourier-filtering
unfiltered
high high-pass med-pass low-pass
Figure 16: Band-pass Pourier-filtered CIFAR10 images. We filtered Fourier-coeffiieents in each color channel separately. For low-pass filtering, Fourier-coeffiieens with radial distance r(u, v) > were set zero. For modium-pass filtering, Fourier-cocffiients with r(u, v) and r(u, were zero. For high-pass filtering, Fourier-coeflcieens with r(u,v) < 10 were set zero. Medium-pass and high-pass filtered images were contrast-maxinae for viewing.
Table 5: Comparing clean accuracy of models standard trained on filtered CIFAR10 images vs Fourier- regularization (ResNet50).
Method
Accuracy
Low-pass Filtered
86.6
Medium-pass Filtered
33.8
High-pass Filtered
15.3
-SP-REGULARIZED
87.1
MSF-REGULLARIZEDD
90.6
HSF-REGULARIIED
93.5
ASF-REGULARIZED
87.9
24

---

Published in Transactions on Machine Learning Research (12/2022)
E
Fourier-noise corruptions
E.1
CIFAR10
Standard Trained
LSF-REGULARIZED
MSS-REGULARIZED
ASP-REGULARIZED
(c)
0.0
Figure 17: (CIFAR10) Heat map of error rates for each Fourier-mos mode corruption (low-frequeniies close to the center). Fach pixel in the heat map the error of the model when the corresponding Fourier-mode noise (e 4) added the inputs. The bottom row displays example images containing the corresponding Fourier-noise.
25

---

Published in Transactions on Machine Learning Research (12/2022)
E.2 SVHN
Standard Trained
LSF-REGULARIZED
(
32
MIS-REGGLARIZER
ASF-REGULARIZED
(c)
30
30
44 4 4 4
Figure 18: (SVHN) Heat map of error rates for each Fourier-mode corruption fow-frequencies close to the center). Each pixel in the heat map is the error of the model when the corresponding Fourier-mode noise (e = 4) is added to the inputs. The bottom row displays example images containing the corresponding Fourier-corruptions.
26

---

Published in Transactions on Machine Learning Research (12/2022)
E.3 Accuracy Fourier-noise distortions
Table 6: Mean accuracy across all Fourier-noise corruptions averaged across 1024 randomly selected test samples for each corruption. &2 norms of the additive Fourier-noise E (3, 4).
SVHN
CIFAR10
CIFAR100
Method
clean
<=3
e=4
clean
c=3
c=4
clean
c=4
Std. Train
81.9
77.4
94.9
40.8
31.5
76.2
22.3
14.9
REGULARIZED
95.1
92.1
87.1
52.4
47.5
62.5
33.9
30.0
MSF-REGULARIZED
93.1
77.1
70.9
90.6
62.4
54.3
70.7
42.6
37.3
ASF-REGULARIZED
96.4
78.3
71.1
87.9
60.8
48.7
67.0
21.6
14.6
27

---

Published in Transactions on Machine Learning Research (12/2022)
F
Patch-shuflling images
original
2x2
3x3
Figure 19: Patch-shuffling Images are partitioned into squares whose positions are randomly exchanged.
This operation destroys global structure in the image and is used to evaluate the extent to which model relies on global information.
28

---

Published in Transactions on Machine Learning Research (12/2022)
G
Sensitivity to ASFS
Table Clean accuracy on CIFAR10 (ResNet50) at different AgFs values
Method
Asrs
Asrs
0.1
Aaps
0.2
Asps
0.5
Asrs
ESPRREUULARIZE
94.8
93.8
91.1
87.1
84.6
0.5
LSF-REG (A=0)
LSF-REG (A=0.1)
0.4
LSF-REG (A=0.2)
Total Power Ratio
LSF-REG (2=0.5)
LSF-REG (A=1)
0.3
0.2
Spatial Frequency (cycles/image)
Figure 20: fourier-sensittee of LST-RECULARIZATIO at different ÀSES CIFARIO (ResNet500.
29


Reviews:
[{"review_id": "DSUod2viaT", "reviewer": "Reviewer_dm8R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] Understanding the sensitivity of the models, even through experimental validation, is significant. \n\n[+] The paper conducts a number of experiments, which validates different aspects of the sensitivity, e.g. different architectures, datasets. \n\n[-] The novelty in the technical part is not very clear. The relationship of different frequencies (of the data) and the model sensitivity has been observed numerous times in the past. \n\n[-] The writing could be further polished (see comment below). \n\n* Even though this is not required for publication, it would be recommended to make the source code public. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the relationship between the Fourier spectrum of the images and the sensitivity of the contemporary neural networks. The paper proposes a way to measure the Fourier sensitivity of such models and then conducts a number of experiments to showcase the Fourier sensitivity of different models (e.g. Resnet-based, Transformers, etc). The paper also indicates what is the relationship of this sensitivity with adversarially-trained models. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* There are no limitations of the study at the moment. \n\n* The writing could be refined. Is the \"spectral bias of neural networks\" (sec. 3.4) referring to the spectral bias observed in the literature (e.g. in [1-4])? Because I found no reference of this related work, and I think the analysis here is based on the Fourier analysis of the data not on the neural network level.\n\n* Is the $\\epsilon=3$ used in Fig. 5 standard for PGD-L2 attacks? \n\n\n\n[1] On the Spectral Bias of Neural Networks, ICML'19.\n\n[2] The Spectral Bias of Polynomial Neural Networks, ICLR'22.\n\n[3] On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels, NeurIPS'22. \n\n[4] Towards Understanding the Spectral Bias of Deep Learning.", "broader_impact_concerns": "There is no broader impact statement at the moment. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y334qQ7QZa", "reviewer": "Reviewer_14vs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nFourier analysis is fundamental in the theory of functions. Therefore it is a natural choice to use these techniques in the study of deep vision models. A careful and complete empirical study is a welcome contribution.\n\nThe empirical results in this paper appear to me well-founded. No theoretical explanations are given, but this seems fine to leave for future work.\n\nFigure 4 convinces me that adversarially trained models are more sensitive to low frequency perturbations than standardly trained models, although only for CIFAR10 and Imagenet. It remains open why this is not the case for ~CIFAR-10~ SVHN, or what property of the data exactly leads to this sensitivity.\n\nFigure 5 convinces me that there is indeed a variable Fourier sensitivity across models, e.g.  InceptionV3 models are more sensitive to low frequencies, but Vision Transformer models are more sensitive to middle and high frequencies. Why this occurs is still an open question.\n\nFigure 6 is convinces me that Fourier-regularization has the intended effect of enhancing sensitivity towards a given frequency band.\n\nPatch shuffling is an appropriate empirical test of low-frequency bias.\n\nThe effect of Fourier-regularization on select OOD shifts is interesting (Table 4). It suggests that specific data shifts may be partially addressed by the appropriate Fourier regularization.\n\n**Weaknesses**\n\nThis is primarily an empirical work. It would of course be desirable to have theoretical explanation of the observations made here, but I am okay with the authors leaving this for future work.\n\nI don't find the \"basis trick\" proposed to be a significant theoretical contribution: it is simply the application of the chain rule for multivariate functions. I do think it is an appropriate measure for the empirical investigation conducted.\n\nA possible connection to the neuroscience of spatial frequency processing in the brain is made in the intro and elaborated in Appendix A. This is potentially interesting, but I found the wording in the intro to be overly strong. It suggests that this connection is indeed already established fact: (*\"In fact, spatial frequencies are differentially processed...\"*). I cannot accept this claim of fact without further justification. Moreover these claims already appear soften in Appendix A (*\"it has been posited...\"*).\n\nI am unclear on why the authors use the term \"Jacobian\" instead of gradient in this work. I cannot detect any reason why the term Jacobians should be used. All model functions in this work have scalar outputs, for which the term gradient is usually used. Perhaps the authors can clarify.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work carries out an empirical study of the sensitivity of various deep vision models to perturbations in the Fourier domain of input images, where perturbations are divided by band. The measure of Fourier sensitivity used is the Fourier transform of input gradients. Regularizers towards this sensitivity proposed. A number of empirical studies are conducted. It is found the sensitivity of models varies by dataset, architecture, and training. The proposed regularizers are empirically shown to mitigate this sensitivity in select cases.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The exclusion of high-frequency perturbations (HSF) in this work appears to me to be an omission. The authors note in Section 5.2: *\"as the baseline models for these datasets already have high frequency sensitives..., we did not train HSF regularized models\"*. I am left wondering about the high frequency sensitivity and regularization results for Figure 6 and Tables 1,2,3,4. For completeness I would still like to see these experiments conducted. This point is important to securing my recommendation.\n\nI suggest that the authors drop claims of the neuroscience connection as fact. I don't think this is an appropriate venue to try to justify that claim. I am in favor of including the discussion of this connection in Appendix but with softened claims.\n\nI have doubts that the \"basis trick\" proposed by the authors should be a theorem. I found that the presentation as such reduced the clarity of the paper, and suggest presenting it as an explicit calculation. This point would strengthen the clarity of the work in my opinion.\n\nWhy weren't the results in Figure 5c replicated for the models in Figure 5a? I suspect this is because of computational cost, however it would improve the completeness of the paper to include them.\n\n", "broader_impact_concerns": "I did not find a broader impact statement included, however I do not harbor any concerns about the broader impacts of this work.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oZL9Mdyaol", "reviewer": "Reviewer_h6JT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. Fourier sensitivity is a useful quantity to understand robustness properties of a model. The approach to compute it is neat and easy to adopt as it just depends on the Jacobian of the network.\n2. The experiments show Fourier sensitivity of different models and how this quantity is affected in adversarial training and training with Gaussian noise augmented images. Additionally, the Fourier regularisation shows that the 'sensitive' frequency band can be controlled while training and it yields improved robustness.\n3. The experiments backed discussion in Section 4 is valuable. Overall the paper is well written.\n\n## Weaknesses\nI didn't notice any major weaknesses of the paper. One writing suggestion is to always define the symbols used in the equations and captions. It is hard to dig them from the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an approach to compute 'Fourier sensitivity' which is a measure of what frequency bands the model is sensitive to. Then, they discuss how this can be used to regularise the model while training to control the frequency bias of the model. Such regularised models show improved robustness to adversarial noise as well as OOD examples. Experiments are performed on popular image classification datasets with various architectures.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Eq.1 is not clear. What does the curly bracket represent? It is better to improve clarity.\n2. Section 3.4 table, what is $\\tilde{P}_k$?\n", "broader_impact_concerns": "The paper is of theoretical nature where it provides a tool to inspect model robustness. It may be possible to use this tool with malicious intent.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission addresses Fourier sensitivity of neural network models, and explores effects of model architecture, as well as the relationship to adversarial examples.  These topics are well addressed with experiments appropriately supporting the claims in the paper.  They are also topics of high interest in computer vision and neural network research.  The reviewers were unanimous in their opinion that the paper should be accepted without further revision.
Decision: Accept (poster)