Year: 2021

Abstract:
Deep neural networks for computer vision are highly accurate but their out-ofdistribution generalization performance remains a major challenge, so understanding their failure modes is an important step towards improving their robustness. Intriguingly, recent work has shown that deep neural networks latch on to the Fourier statistics of training data, show increased sensitivity to certain Fourierbasis directions in the input and are not robust when Fourier-statistics shift unfavorably at test time. Understanding and modifying this Fourier-sensitivity of computer vision models may help improve their robustness, hence, in this paper we study the frequency sensitivity characteristics of deep neural networks using a principled approach. We first propose a general measure of a computer vision model’s spatial frequency sensitivity based on its input-Jacobian represented in the Fourier-basis. When applied to deep neural networks, we find that standard minibatch training consistently leads to increased sensitivity towards particular spatial frequencies independent of network architecture. We further propose a family of spatial frequency regularizers based on our proposed measure to induce specific spatial frequency sensitivities in a model. In various generalization tests involving shifts in the Fourier-statistics of the data, we find that deep neural networks trained with our proposed regularizers obtain significantly improved classification accuracy while maintaining high accuracy on in-distribution test images.

heading: 1 INTRODUCTION
text: While deep neural networks (DNN) achieve remarkable performance on many challenging image classification tasks, they can suffer significant drops in performance when evaluated on out-ofdistribution (o.o.d.) data. Intriguingly, this lack of robustness has been partially attributed to the frequency characteristics of data shifts at test time in relation to the frequency sensitivity characteristics of the model (Yin et al., 2019; Jo & Bengio, 2017). Distinct spatial frequencies in images contain features at different spatial scales; low spatial frequencies (LSF) carry global structure and shape information in a scene whereas high spatial frequencies (HSF) carry local information such as edges and borders of objects in a scene (Kauffmann et al., 2014); in fact, spatial frequencies are differentially processed in distinct channels of the visual cortex in the brain to learn features at different scales (Appendix B). When information is destroyed or corrupted in frequency bands that a model relies on, performance suffers. Hence, understanding the spatial frequency sensitivity of a DNN can help us characterise the features it relies on to make predictions.
DNNs have been demonstrated to be sensitive to Fourier-basis directions in the input (Tsuzuku & Sato, 2019; Yin et al., 2019) both empirically and using theoretical analysis of linear convolutional networks (Tsuzuku & Sato, 2019). In fact, the existence of so-called “universal adversarial perturbations” (Moosavi-Dezfooli et al., 2017), simple semantics-preserving distortions that can degrade models’ accuracy across inputs and architectures, is attributed to this structural sensitivity and the use of convolution operations. Yin et al. (2019) also showed that many natural and digital image corruptions that degrade model performance may also be targeting this vulnerability. The Fouriercharacteristics of adversarial examples are also known to closely match the Fourier-sensitivity of models (Yin et al., 2019). Hence, understanding and modifying Fourier-sensitivity can aid efforts to improve model robustness. While this Fourier-sensitivity has been studied empirically, the precise definition and measurement of a computer vision model’s spatial frequency sensitivity still lacks a
rigorous approach across studies. In addition, there has been no principled method to specifically modify the spatial frequency sensitivity of a model. Existing works have heuristically applied filters on convolution layer parameters (Wang et al., 2020; Saikia et al., 2021) and data augmentations (Yin et al., 2019) to modify a model’s frequency sensitivity.
In this work, we propose a novel and rigorous measure of a deep neural network’s spatial frequency sensitivity using the input-Jacobian represented in the Fourier-basis and show that deep neural networks demonstrate consistent spatial frequency sensitivities across samples, an observation that suggests DNNs are more likely to consistently use some frequencies more than others, and has implications for robustness. In addition, using our proposed measure, which is differentiable with respect to model parameters, we propose a novel family of spatial frequency regularizers to directly induce specific frequency sensitivities in a model. We hypothesize and show in empirical evaluations that spatial frequency regularization can modify the frequency sensitivity characteristics of computer vision models and can significantly improve the generalization performance of models on o.o.d. datasets where the Fourier-statistics are unfavorably shifted.
In summary, the main contributions of this work are as follows:
1. We propose a novel and rigorous measure of a model’s spatial frequency sensitivity based on its input-Jacobian represented in the Fourier-basis
2. We propose a novel family of spatial frequency regularizers to directly induce specific spatial frequency sensitivities
3. We demonstrate that spatial frequency regularization can significantly improve generalization performance on out-of-distribution data where Fourier-statistics are unfavorably shifted

heading: 2 RELATED WORK
text: Characterising frequency sensitivity: Yin et al. (2019); Tsuzuku & Sato (2019) characterised the Fourier characteristics of trained CNNs using perturbation analysis of their test error under Fourierbasis noise. They showed that a naturally trained model is most sensitive to all but low frequencies whereas models adversarially trained (Madry et al., 2018) models are sensitive to low-frequency noise. They further showed that these Fourier characteristics relate to model robustness on corruptions and noise, with models biased towards low frequencies performing better under high frequency noise and vice versa. Abello et al. (2021) took a different approach by measuring the impact on accuracy of removing individual frequency components from the input using filters whereas OrtizJimenez et al. (2020) computed the margin in input space along basis directions of the discrete cosine transform (DCT). Wang et al. (2020) made observations about the Fourier characteristics of CNNs in different training regimes including standard and adversarial training by evaluating accuracy on band-pass filtered data. In contrast to these disparate approaches, in this work, we propose a rigorous measure of a model’s spatial frequency sensitivity.
Regularizing frequency sensitivity: Yin et al. (2019) proposed adversarial training (Madry et al., 2018) and gaussian noise augmentations as methods that induce a low-frequency sensitivity. Wang et al. (2020) proposed smoothing convolution filter parameters to induce a low-frequency sensitivity in models. We note that, in general, techniques that apply filters on convolutional parameters to affect a model’s frequency sensitivity do not take into account complex operations such as nonlinearities, pooling and other transformations that often follow convolutional layers that can modify as well as undo the effects of such filters. In addition, data augmentations do not provide precise control over the Fourier-sensitivity of a model. In this work, we propose a family of spatial frequency regularizers that can precisely modify the overall spatial frequency sensitivity of any differentiable model.
Jacobian regularization: Methods that regularize the Jacobian of a model’s output-logits or loss with respect to its input can broadly be divided into two types; one regularizes the norm of the input-Jacobian and the other regularizes its direction or directional derivatives. Drucker & Le Cun (1991) proposed a method that penalized the norm of the input-Jacobian to improve generalization, more recently this has been explored to improve robustness to adversarial perturbations (Ross & Doshi-Velez, 2018; Jakubovitz & Giryes, 2018; Hoffman et al., 2019). Simard et al. (1992) proposed
”Tangent Prop”, which minimized directional derivatives of classifiers in the direction of local inputtransformations (e.g. rotations, translations; called ”tangent vectors”) to reduce model sensitivity to such transformations. Czarnecki et al. (2017) proposed Sobolev training of neural networks to improve model distillation by matching the input-Jacobian of the original model. Regularizing the direction of the input-Jacobian has also been used to improve adversarial robustness (Chan et al., 2020). In the present work, we regularize Fourier-components in the input-Jacobian to modify the spatial frequency sensitivity of models. As such, we are directly modifying the direction of the input-Jacobian instead of its norm.

heading: 3 METHODS
text: Preliminaries: We introduce all relevant definitions and notations before describing the proposed methods. Consider an image classification task with input images x, target labels y, and the standard cross-entropy loss function LCE. Let f denote any differentiable model, F(·) the unitary 2D discrete Fourier transform (DFT), F−1(·) its inverse, and F−1∗(·) the adjoint of the inverse-Fourier transform, and let xf denote the Fourier space representation of the input, i.e. xf = F(x). We denote the input-Jacobian in the standard input basis as ∂LCE∂x , and ∂LCE ∂xf
as the input-Jacobian in the Fourier-basis. Let N denote the height of the input images x (although not necessary, all images used in this work are square). The zero-shifted 2D-DFT of the input-Jacobian is denoted F = F(∂LCE∂x ). Since the input-Jacobian typically has three color channels, they are averaged before computing the 2D-DFT. Fourier coefficients in F are complex numbers with real and imaginary components; F (u, v) = Real(u, v) + Imag(u, v), where (u, v) are indices of coefficients. The power in a coefficient is its squared amplitude, P (u, v) = |F (u, v)|2 = Real(u, v)2 + Imag(u, v)2 and the matrix of powers is denoted P (power-matrix). Each coefficient has a radial distance r(u, v) from the centre of the matrix, r(u, v) = d((u, v), (cu, cv)), where (cu, cv) denotes the centroid of P and d(·, ·) is Euclidean distance. Distinct radial distances, rounded to the nearest integer, of coefficients in the matrix are the set of integers {1, . . . , N/ √ 2} and correspond to low to high spatial frequency bands, the highest spatial frequency being limited by the Nyquist frequency. We denote PTotal as the total power in P , excluding the zero-frequency coefficient, PTotal = ∑ r(u,v)>=1 P (u, v).
Similarly, we define P̃Total as the total power in P excluding the zero-frequency coefficient as well as coefficients with radial distance r(u, v) > N/2, i.e. outside the largest circle inscribed in the power-matrix P ; P̃Total = ∑ 1<=r(u,v)<=N/2 P (u, v) (please see Figure 5 in Appendix A.1 for an illustration). We denote Pk as the total power at radial distance k normalized by PTotal, Pk =
1 PTotal ∑ r(u,v)=k P (u, v) and P̃k as total power at radial distance k normalized by P̃Total in-
stead, P̃k = 1P̃Total ∑
r(u,v)=k
P (u, v).

heading: 3.1 SPATIAL FREQUENCY SENSITIVITY OF A MODEL
text: In this section, we define the proposed spatial frequency sensitivity (SFS) of any differentiable model using its input-Jacobian represented in the Fourier-basis. As the input-Jacobian provides the direction of highest input sensitivity, we show below that its Fourier transform, which is simply a change of basis, provides the model’s sensitivities to spatial frequency components in the input. The input-Jacobian in the Fourier-basis, ∂LCE∂xf , comprises the sensitivities of the model with respect to individual frequencies in the input and we obtain this by simply computing the Fourier transform of the input-Jacobian ∂LCE∂x . In order to justify this, consider the computation graph where the input x is mapped to a scalar loss via a model f and loss function LCE (Figure 2). We introduce an implicit operation (shown in red) that maps the Fourier space representation of the input, xf , to the standard input x through the inverse Fourier-transform; xf F−1−−−→ x, so in order to compute the input-Jacobian in the Fourier-basis, ∂LCE∂xf , we must differentiate through this implicit operation in the forward graph. Since the inverse-Fourier transform is a unitary operator, its adjoint is also its inverse, i.e. F−1∗ = (F−1)−1 = F . Hence, following the chain rule for complex operators, ∂LCE∂xf is simply the Fourier transform of the input-Jacobian ∂LCE∂x .
Input image Input-Jacobian of model 2D DFT input-Jacobian SFS=proportion of power in circular frequency bands
∂LCE ∂xf = F−1∗(∂LCE ∂x ) = F(∂LCE ∂x )
Hence, even though we do not explicitly use the Fourier representation of the input, this shows that the Fourier transform of the input-Jacobian provides us the sensitivity of the model with respect to spatial frequencies of the input. In fact, similar results hold for any unitary operation on the inputJacobian, such as the discrete cosine transform (DCT). Interested readers can replace the DFT (F) with other such transforms if they would like to understand the sensitivity of a model with respect to other components in the input. We now define the spatial frequency sensitivity fSFS(x, y) with respect to an individual input (x, y) as,
fSFS(x, y) = {P1, . . . , PN/√2}
where Pk is the proportion of total power in Fourier coefficients at radial distance k in the power matrix P of the Fourier-transformed input-Jacobian. The overall spatial frequency sensitivity of a model is defined as the expectation of fSFS(x, y) over the data distribution, i.e. fSFS(·; θ) = E(x,y)∼p[fSFS(x, y)] (please see Figure 1 for an illustration and Algorithm 1 in Appendix A.1). We note that although we use the Jacobian of the loss function to estimate the SFS of a model, this formulation is valid for other output functions and tasks as well. An alternative approach that does not require image labels is to compute the Jacobian of the model’s output logits or softmax probabilities. In addition, while the spatial frequency sensitivity assumes the input is spatial 2D data (e.g. images), we can extend this approach to any n-dimensional data by using the n-dimensional Fourier-transform as well as to other computer vision tasks.

heading: 3.2 SPATIAL FREQUENCY REGULARIZATION
text: In this section, we propose a novel family of spatial frequency regularizers that can modify the spatial frequency sensitivity, as defined in Section 3.1, of a model directly. As all computations involved in the SFS of a model (Algorithm 1 in Appendix A.1) are differentiable with respect to its parameters, we compute a loss function, L(x, y) = LCE(x, y) + λSFSLSFS(x, y), where LSFS is the proposed loss that can be used to induce a specific SFS and λSFS is a hyperparameter. As LSFS is defined on the SFS, which is a function of the input-Jacobian, optimizing it requires an additional backpropagation step to compute its derivatives with respect to model parameters, similar
to other Jacobian-regularization methods. We now define LSFS for three different SFS regularizers; SFS ∈ {LSF,MSF,ASF}. LSF regularization trains a model to be insensitive to medium and high spatial frequencies and MSF regularization trains a model to be insensitive to low and high spatial frequencies. We achieve this by penalizing the proportion of total power, Pk, in frequency bands. ASF regularization trains a model to be equally sensitive to all spatial frequency (ASF) bands. The motivation behind ASF regularization model is to encourage a model to be sensitive to multiple frequency bands instead of being concentrated in a single frequency range. Hence, the ASF-regularizer loss is the negative entropy of the distribution of power over spatial frequency bands.
LSFS(x, y) :=  ∑ k>N/6 Pk, if SFS=LSF ∑ k<N/6,k>N/3 Pk, if SFS=MSF
k=N/2∑ k=1 P̃k log P̃k, if SFS=ASF
The definitions of frequency ranges are based on equally dividing the largest circle inscribed in the power-matrix P into equal parts (Figure 5 in Appendix A.1). For ASF-regularization, very high frequency bands (r(u, v) > N/2 are excluded. Interested readers may modify these loss functions as they see fit for their particular application.

heading: 4 EXPERIMENTS
text: In this section we explore the SFS of models as well as explore the effects of spatial frequency regularization by evaluating on datasets where the Fourier-statistics of predictive features are naturally or artificially altered (Sections 4.2, 4.4). We also make some observations about the differences between spatial frequency regularization and training on Fourier-filtered training data (Section D.3). We benchmark against methods that have been proposed to modify the frequency sensitivity of models such as adversarial training and Gaussian Noise augmentation (Yin et al., 2019) to induce low-frequency sensitivity. AugMix (Hendrycks et al., 2020) is also benchmarked as it was shown to achieve state-of-the-art performance on common corruptions. We also trained models on band-pass filtered datasets to demonstrate that SFS regularization is not equivalent to training on Fourier-filtered data (discussion relegated to Appendix D.3).
Experimental Setup: We first describe the experimental settings for all experiments that follow. We trained the ResNet50 architecture, unless stated otherwise. For CIFAR10 and CIFAR100 (Krizhevsky & Hinton, 2009), we trained all models for 150 epochs using stochastic gradient descent (SGD) with momentum (0.9), an initial learning rate of 0.1 decayed by a factor of 10 every 50 epochs, weight decay parameter equal to 5e-4 and batch size equal to 128. For SVHN (Netzer et al., 2011), we trained models for 40 epochs using Nesterov momentum with an initial learning rate of 0.01 and momentum parameter 0.9. The training batch size was 128, the L2 regularization parameter was 0.0005 and we decayed the learning rate at epochs 15 and 30 by dividing by 10. Standard data augmentations random-crop, random-horizontal-flip, random-rotation, and color-jitter were used during training. We trained spatial frequency regularized models using Algorithm 2 2 in Appendix A.1, and λSFS = 1 for all models. We observed a simple trade-off between this hyperparameter and clean accuracy (higher values decreased clean accuracy more). Values less than 1 did not always achieve the desired SFS in the evaluated datasets. For adversarial training (AT), we used PGD `2 attacks ( = 1, attack-steps = 7 attack-lr = 1/7). To train models with Gaussian noise data augmentation, we added i.i.d. Gaussian noiseN (0, σ2) to each pixel in all the images in the training set (σ = 0.1). We used the robustness (Engstrom et al., 2019) library to train models as well as retrieve pre-trained ImageNet models of various architectures. We trained AugMix models using the original code shared by the authors (https://github.com/google-research/augmix). Please see Section D.2 in Appendix D for details about band-pass filters.

heading: 4.1 CNNS DISPLAY CONSISTENT SPATIAL FREQUENCY SENSITIVITY ACROSS SAMPLES
text: We now visualize the spatial frequency sensitivity (SFS), as defined in Section 3.1, of models trained on CIFAR10 and ImageNet (Figure 3). We note that training biases models to be sensitive to some frequencies more than others and this bias is consistent across samples (the error bars in Figure 3 represent deviation across samples). This consistency also suggests that, once trained, CNNs are more likely to use some frequencies more than others and may explain their tendency to suffer performance drops when evaluated on data with different Fourier-statistics. CIFAR100 models have very similar SFS to those of CIFAR10 (plots relegated to Appendix C.2) and are also most sensitive to mid-to-high spatial frequencies. In contrast, standard training on SVHN leads to a low-frequency sensitivity, which suggests a strong dataset dependence of spatial frequency sensitivity. Please refer to Appendix C.3 for SFS plots of models trained on SVHN.
ImageNet models are sensitive to a wide range of the frequency spectrum with peak sensitivity to mid-range frequencies. We also observed consistency in the spatial frequency sensitivity of various convolutional architectures trained on ImageNet (Appendix C.1). Adversarially trained models (Madry et al., 2018) are most sensitive to low spatial frequencies across datasets and architectures which suggests they rely on coarse global features, in agreement with observations made in prior work. Training on Stylized-ImageNet, proposed by Geirhos et al. (2019) to train shape-biased in models, induces a lower peak-spatial-frequency-sensitivity and as well as lower sensitivity to high frequencies compared to training on ImageNet (Appendix C.2), which reflects the increased shapebias of such models. Similarly, a model trained on low-pass Fourier-filtered training data correspondingly has high sensitivity to low-frequencies in comparison to the baseline model (Appendix C.2). Gaussian noise augmentation N (0, σ2;σ = 0.1), decreases the high-frequency sensitivity of the model (Figure 3) compared to baseline. The SFS of the AugMix (Hendrycks et al., 2020) trained model was not significantly different from baseline, which suggests it finds more robust features in similar frequency bands using data augmentations.
Interestingly, we also note that models trained on corrupted versions of the CIFAR10 training set (e.g. blurred or JPEG-compressed) can display different spatial frequency sensitivities to a model trained on clean CIFAR10 images (please see Appendix C.4 for plots). For example, models trained on images with severe noise corruptions (gaussian, shot and speckle) display increased sensitivity to lower spatial frequencies. Models trained on highly gaussian-blurred, glass-blurred, JPEGcompressed and pixelated images also display higher sensitivity to lower spatial frequencies. These changes reflect the shift in the Fourier-statistics of predictive features in corrupted images.

heading: 4.2 PERFORMANCE UNDER FOURIER FILTERING
text: Jo & Bengio (2017) showed that DNNs have a tendency to rely on superficial Fourier-statistics of their training data. In the vein of generalization evaluations they performed, we generate Fourierfiltered CIFAR10/CIFAR100 validation images using radial masking in frequency space (please see Appendix D.1 for examples). A mask radius r determines Fourier components that are preserved with larger radii preserving more components. We use (cu, cv) to denote the centre of the mask and
d(·, ·) to denote Euclidean distance. The radial mask Mr is defined as:
Mr(u, v) := { 1, if d((u, v), (cu, cv)) ≤ r 0, otherwise
(1)
The mask is applied on the zero-shifted output of the Fourier transform of each image, denoted X , followed by the inverse transform, i.e. Xfiltered = F−1(F(X) Mr), where is the elementwise product. Fourier-filtering is performed on each color channel independently. A standard CNN suffers up to a 75% drop in accuracy on low-pass filtered CIFAR10 data as it relies on high frequency information that is no longer present (Table 1). On the other hand, the LSF-REGULARIZED model performs well on both clean images as well severely low-pass filtered images. This shows that regularized CNNs are able to exploit very low frequency features (r ≤ 5) not easily recognizable even to humans, which reflects the limited spatial-frequency-sensitivity range of the human eye.
The MSF-REGULARIZED model performs better for mild Fourier-filtering (r = 11 and r = 7) but suffers a big drop in accuracy (71.5% to 46.2% in CIFAR10 and 46.4% to 18.6% in CIFAR100) for strong low-pass filtering (r = 5). We also observed that the ASF-REGULARIZED model is also significantly more robust than the baseline model although not as robust as the LSF-REGULARIZED model to Fourier-filtering. This shows that the ASF-REGULARIZED model can use coarse lowspatial-frequency features in addition to medium and high spatial frequencies. The low-frequency sensitivity of adversarially trained models makes it more robust than the baseline model but less so than the LSF-REGULARIZED model, while the Gaussian noise augmented model suffers large drops in accuracy. AugMix tends to be robust to mild Fourier-filtering but suffers large drops in accuracy under more severe filtering (r = 5) due to its reliance on mid-to-high-frequency information, as suggested by its SFS (Figure 3), in comparison to the LSF-REGULARIZED model.

heading: 4.3 PERFORMANCE UNDER FOURIER-NOISE CORRUPTIONS
text: Recently, data-agnostic corruptions along Fourier-basis directions have been identified as a threat to model robustness and security (Yin et al., 2019; Tsuzuku & Sato, 2019). A Fourier-noise corruption is additive noise containing a single Fourier-mode (frequency). These corruptions are semanticspreserving and significantly affect model performance while being recognizable to humans, please refer to Appendix E for examples in each dataset. We generated corruptions for each of 1,024 Fourier-modes in SVHN/CIFAR10/CIFAR100 and report the overall mean accuracy across all corruptions in Table 2. We note that the baseline model suffers significant drops in accuracy across all
datasets. The LSF-regularized model achieves the highest mean accuracy in SVHN while Gaussian augmentation also improves over the baseline. AugMix (Hendrycks et al., 2020), a dataaugmentation method proposed for robustness, performs worse than baseline. In CIFAR10 and CIFAR100, the MSF-regularized model achieves the highest mean accuracies. Detailed heat maps containing error rates for each Fourier-mode can be found in Appendix E.

heading: 4.4 PERFORMANCE UNDER CORRUPTIONS IN (HENDRYCKS & DIETTERICH, 2019)
text: We evaluated the robustness of spatial frequency regularized models to corruptions (please see Appendix F for examples) in CIFAR10-C (Hendrycks & Dietterich, 2019). In addition, we applied these corruptions to SVHN to create SVHN-C to evaluate the performance of methods across datasets. Overall, SFS regularization provided consistent improvements across both SVHN-C (Table 3) and CIFAR10-C (Table 4) due to its general and principled approach. We observed that AugMix’s (Hendrycks et al., 2020) high performance on CIFAR10-C does not translate to high performance on SVHN-C. Similarly, Gaussian augmentation’s high performance in SVHN-C did not translate to CIFAR10-C. Specifically, on SVHN-C, LSF-regularization, low-pass filtering and Gaussian augmentation performed very well under noise corruptions whereas AugMix did not improve over the baseline. In CIFAR10-C, we observed that the LSF-REGULARIZED model was significantly more robust than the baseline model to all blurring corruptions as well as pixelate, elastic, snow and frost corruptions. The baseline model suffered up to 45% drop in accuracy on these corruptions, whereas the LSF-REGULARIZED model suffered at most 5% drop in accuracy. This is likely due to the robustness of low-spatial-frequency features to corruptions such as blurring. We also observed that the MSF-REGULARIZED and ASF-REGULARIZED models were more robust to noise corruptions as well as JPEG compression (Table 4), which suggests multiple spatial frequency bands maybe needed to do well under such corruptions. We observed similar results in CIFAR100-C (Appendix F.2). Overall, SFS regularization can provide consistent improvements in robustness to these corruptions across datasets due to its general and principled approach. On the other hand, data augmentation based methods invariably perform better on some datasets than others depending on the statistics and features of each dataset.

heading: 4.5 EVALUATION OF GLOBAL FEATURE LEARNING
text: Here we wish to evaluate the extent to which models use global features by measuring their performance on patch-shuffled images, which have previously been used by Mummadi et al. (2021); Zhang & Zhu (2019); Wang et al. (2019). Patch-shuffling involves splitting an image into k × k squares and randomly swapping the positions of these squares. This is intended to destroy global features and retain local features with larger values of k retaining less global structure in the image (please see Appendix G for examples). As such, models that rely more on global rather than local structure suffer more from patch-shuffling. Hence, in this benchmark, lower accuracy suggests increased reliance on and use of global structure. We observed that spatial frequency regularized models, adversarially trained models suffered larger drops in accuracy compared to the standard trained, AugMix and Gaussian augmentation models, which suggests their reliance on global structure in images contributes to their robustness (Table 5).

heading: 5 CONCLUSION
text: We proposed a novel and rigorous measure of the spatial frequency sensitivity (SFS) of any differentiable computer model using its input-Jacobian represented in the Fourier-basis. We showed that models display consistent SFS across samples as well as across architectures for a given task and when there is a shift in the Fourier-statistics of features relative to a model’s frequency sensitivity, performance can suffer. We also proposed a novel method of spatial frequency regularization that can be used to improve robustness to many test-time corruptions. Extending our method to other computer vision tasks (e.g. detection) as well as domains (n-dimensional) are interesting directions for future work.

heading: APPENDICES

heading: A SPATIAL FREQUENCY SENSITIVITY AND REGULARIZER

heading: A.1 PSEUDO-CODES
text: Algorithm 1: Spatial Frequency Sensitivity (SFS) Input: Labeled samples L = {(xi, yi)}ni=1; a model f with trained parameters θ Output: Estimated Spatial Frequency Sensitivity of model, fSFS(·; θ)
for i = 1, . . . , n do compute loss LCE(f(xi), yi) // forward pass backpropagate LCE to obtain ∂LCE∂xi // input-jacobian ∂LCE ∂xf i
= F(∂LCE∂xi ) // 2D DFT of input-jacobian (averaged across color channels)
fSFS(xi, yi) = [Pk; for k=1 to N/ √ 2] // Pk is proportion of total power
(excluding DC) in coefficients at radial distance k end fSFS(·; θ) = 1n ∑n i=1 fSFS(xi, yi) // estimated SFS of model
Algorithm 2: Spatial Frequency Regularized Minibatch Training Input: Labeled samples L = {(xi, yi)}ni=1; a model f with parameters θ; SFS ∈ {LSF,MSF,ASF}; hyperparameter λSFS Output: Regularized model with updated parameters θ
for i = 1, . . . , n do compute loss LCE(f(xi), yi) // forward pass L(xi, yi) = LCE(xi, yi) + λSFSLSFS(xi, yi) // backpropagate L to update θ end

heading: B SPATIAL FREQUENCY CHANNELS IN THE BRAIN
text: To further motivate the spatial frequency perspective of visual representation learning, we briefly describe some relevant findings from neuroscience and vision research. While the brain does not strictly perform a Fourier analysis of visual scenes, there has been mounting evidence over decades for multiple spatial frequency channels in the early human visual system that are physiologically independent and selectively responsive to distinct spatial frequency bands (Campbell & Robson, 1964; 1968). It has been posited that the use of different spatial frequency channels are determined by the demands of a given visual task through an attention mechanism (Schyns & Oliva, 1999; Rotshtein et al., 2010; Julesz & Papathomas, 1984). Spatial frequency channels enable us to attend to different spatial scales in a scene at a fixed viewing distance, similar to the focus lens in a camera (Figure 6). On the other hand, once trained, a CNN’s spatial frequency sensitivity (Section 3.1) is inflexible and does not vary much across samples (both in and out of its training data distribution).
EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFonononononononononononononoEFEFEFEFEF EFEFEFEFEFonononononononononononononoEFEFEFEFEF EFEFEFEFEFonononononononononononononoEFEFEFEFEF EFEFEFEFEFonononononononononononononoEFEFEFEFEF EFEFEFEFEFonononononononononononononoEFEFEFEFEF EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFE EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF
Figure 6: Letters at multiple spatial scales. This image comprises the letters o, n, E and F at a small spatial scale (HSF). The letter T is also visible at a larger spatial scale (LSF) formed by the specific arrangement of the letters o, n amidst letters E, F. Identifying these letters requires processing features at multiple scales, enabled by distinct spatial frequency channels in the early visual cortex. Our ability to recognize only one of these scales at a time is evidence for the physiological independence of spatial frequency channels in the brain. Image based on Julesz & Papathomas (1984).

heading: C SUPPLEMENTARY PLOTS

heading: C.1 SPATIAL FREQUENCY SENSITIVITIES OF POPULAR IMAGENET MODELS
text: VGG16
MobileNet
ResNext50ShuffleNet
DenseNetWideResNet50

heading: C.2 SPATIAL FREQUENCY SENSITIVITIES OF MODELS TRAINED ON CIFAR10 AND
text: CIFAR100

heading: C.4 TRAINING ON CIFAR10 CORRUPTIONS

heading: C.3 SPATIAL FREQUENCY SENSITIVITIES OF MODELS TRAINED ON SVHN

heading: D FOURIER FILTERING
text: D.1 RADIAL FILTERING
D.2 BAND-PASS FILTERING

heading: D.3 SPATIAL FREQUENCY REGULARIZATION IS NOT EQUIVALENT TO TRAINING ON FOURIER-FILTERED DATA
text: We note that spatial frequency regularization is not equivalent to standard training on corresponding Fourier-filtered data. In natural images, the amount of energy in frequency bands falls off rapidly as frequency increases (Hyvärinen et al., 2009), hence, medium and high-pass filtered natural images typically appear completely empty to the human eye without additional contrast maximisation. Even still, such images are not easily recognizable to the human eye due to its and reliance on low frequency information to recognize objects optimally (Appendix D.2). We observed that standard training of CNNs on medium or high-pass Fourier-filtered data did not achieve high clean accuracies (Tables 4, 6). A model trained on medium-pass Fourier-filtered CIFAR10 achieved a clean accuracy of only ∼33% on clean test samples whereas the MSF-REGULARIZED model’s clean accuracy is ∼87%, and a model trained on high-pass filtered CIFAR10 training samples achieved only ∼15% accuracy on clean test samples. Due to the energy statistics over frequency bands in natural images, training on Fourier-filtered data is not successful for all but the lowest frequency bands, where most of their energy resides. In addition, we note that regularizers such as ASF-regularization cannot be recreated by Fourier-filtering the training data due to the tendency of standard training to be sensitive to certain frequencies.

heading: E.2 SVHN FOURIER-NOISE CORRUPTIONS

heading: E.1 CIFAR10 FOURIER-NOISE CORRUPTIONS

heading: E BLACK BOX FOURIER-NOISE CORRUPTIONS

heading: F.2 CIFAR100-C BENCHMARK

heading: F.1 EXAMPLES

heading: F CORRUPTIONS IN (HENDRYCKS & DIETTERICH, 2019)
text: G PATCH-SHUFFLING IMAGES

Review summaries:
This paper proposes a novel spatial frequency regularization technique that improves the robustness of training neural networks against superficial fourier statistics in a dataset.  In the loss function, It adds a regularization term that is based on the Fourier-transformed input-Jacobian.  This term could be customized such that the trained model could ignore (be insensitive to) features with specific frequency in the dataset.   The authors define spatial frequency sensitivity using input-Jacobian in Fourier space.  Although this paper focuses on datasets with images, the method is extendable to any n-dimensional data.  Empirically, the method is evaluated for its robustness against fourier filtering, corruptions, and image patches shuffling in CIFAR10 & CIFAR100 datasets.  It shows better results than other baselines in maintaining the classification accuracy of a model against fourier filtering and image patches shuffling.  There do seem to be slight performance gaps from the SOTA AugMix method in image corruptions.  However, the method is considered to be simpler than AugMix and still outperforms other baselines in many cases.   Experiments demonstrate that the proposed method could learn global features present in a dataset.

This paper proposes a measure for a model’s spatial frequency sensitivity (SFS) based on the input-Jacobian in the Fourier basis. With this measure, the authors observe standard CNN training biases towards certain particular spatial frequencies consistently across samples. Based on this measure, the authors propose a family of spatial frequency regularization techniques to suppress the model’s sensitivities to certain spatial frequencies.