Spatial Frequency Sensitivity Regularization for Robustness

1 INTRODUCTION . While deep neural networks ( DNN ) achieve remarkable performance on many challenging image classification tasks , they can suffer significant drops in performance when evaluated on out-ofdistribution ( o.o.d . ) data . Intriguingly , this lack of robustness has been partially attributed to the frequency characteristics of data shifts at test time in relation to the frequency sensitivity characteristics of the model ( Yin et al. , 2019 ; Jo & Bengio , 2017 ) . Distinct spatial frequencies in images contain features at different spatial scales ; low spatial frequencies ( LSF ) carry global structure and shape information in a scene whereas high spatial frequencies ( HSF ) carry local information such as edges and borders of objects in a scene ( Kauffmann et al. , 2014 ) ; in fact , spatial frequencies are differentially processed in distinct channels of the visual cortex in the brain to learn features at different scales ( Appendix B ) . When information is destroyed or corrupted in frequency bands that a model relies on , performance suffers . Hence , understanding the spatial frequency sensitivity of a DNN can help us characterise the features it relies on to make predictions . DNNs have been demonstrated to be sensitive to Fourier-basis directions in the input ( Tsuzuku & Sato , 2019 ; Yin et al. , 2019 ) both empirically and using theoretical analysis of linear convolutional networks ( Tsuzuku & Sato , 2019 ) . In fact , the existence of so-called “ universal adversarial perturbations ” ( Moosavi-Dezfooli et al. , 2017 ) , simple semantics-preserving distortions that can degrade models ’ accuracy across inputs and architectures , is attributed to this structural sensitivity and the use of convolution operations . Yin et al . ( 2019 ) also showed that many natural and digital image corruptions that degrade model performance may also be targeting this vulnerability . The Fouriercharacteristics of adversarial examples are also known to closely match the Fourier-sensitivity of models ( Yin et al. , 2019 ) . Hence , understanding and modifying Fourier-sensitivity can aid efforts to improve model robustness . While this Fourier-sensitivity has been studied empirically , the precise definition and measurement of a computer vision model ’ s spatial frequency sensitivity still lacks a rigorous approach across studies . In addition , there has been no principled method to specifically modify the spatial frequency sensitivity of a model . Existing works have heuristically applied filters on convolution layer parameters ( Wang et al. , 2020 ; Saikia et al. , 2021 ) and data augmentations ( Yin et al. , 2019 ) to modify a model ’ s frequency sensitivity . In this work , we propose a novel and rigorous measure of a deep neural network ’ s spatial frequency sensitivity using the input-Jacobian represented in the Fourier-basis and show that deep neural networks demonstrate consistent spatial frequency sensitivities across samples , an observation that suggests DNNs are more likely to consistently use some frequencies more than others , and has implications for robustness . In addition , using our proposed measure , which is differentiable with respect to model parameters , we propose a novel family of spatial frequency regularizers to directly induce specific frequency sensitivities in a model . We hypothesize and show in empirical evaluations that spatial frequency regularization can modify the frequency sensitivity characteristics of computer vision models and can significantly improve the generalization performance of models on o.o.d . datasets where the Fourier-statistics are unfavorably shifted . In summary , the main contributions of this work are as follows : 1 . We propose a novel and rigorous measure of a model ’ s spatial frequency sensitivity based on its input-Jacobian represented in the Fourier-basis 2 . We propose a novel family of spatial frequency regularizers to directly induce specific spatial frequency sensitivities 3 . We demonstrate that spatial frequency regularization can significantly improve generalization performance on out-of-distribution data where Fourier-statistics are unfavorably shifted 2 RELATED WORK . Characterising frequency sensitivity : Yin et al . ( 2019 ) ; Tsuzuku & Sato ( 2019 ) characterised the Fourier characteristics of trained CNNs using perturbation analysis of their test error under Fourierbasis noise . They showed that a naturally trained model is most sensitive to all but low frequencies whereas models adversarially trained ( Madry et al. , 2018 ) models are sensitive to low-frequency noise . They further showed that these Fourier characteristics relate to model robustness on corruptions and noise , with models biased towards low frequencies performing better under high frequency noise and vice versa . Abello et al . ( 2021 ) took a different approach by measuring the impact on accuracy of removing individual frequency components from the input using filters whereas OrtizJimenez et al . ( 2020 ) computed the margin in input space along basis directions of the discrete cosine transform ( DCT ) . Wang et al . ( 2020 ) made observations about the Fourier characteristics of CNNs in different training regimes including standard and adversarial training by evaluating accuracy on band-pass filtered data . In contrast to these disparate approaches , in this work , we propose a rigorous measure of a model ’ s spatial frequency sensitivity . Regularizing frequency sensitivity : Yin et al . ( 2019 ) proposed adversarial training ( Madry et al. , 2018 ) and gaussian noise augmentations as methods that induce a low-frequency sensitivity . Wang et al . ( 2020 ) proposed smoothing convolution filter parameters to induce a low-frequency sensitivity in models . We note that , in general , techniques that apply filters on convolutional parameters to affect a model ’ s frequency sensitivity do not take into account complex operations such as nonlinearities , pooling and other transformations that often follow convolutional layers that can modify as well as undo the effects of such filters . In addition , data augmentations do not provide precise control over the Fourier-sensitivity of a model . In this work , we propose a family of spatial frequency regularizers that can precisely modify the overall spatial frequency sensitivity of any differentiable model . Jacobian regularization : Methods that regularize the Jacobian of a model ’ s output-logits or loss with respect to its input can broadly be divided into two types ; one regularizes the norm of the input-Jacobian and the other regularizes its direction or directional derivatives . Drucker & Le Cun ( 1991 ) proposed a method that penalized the norm of the input-Jacobian to improve generalization , more recently this has been explored to improve robustness to adversarial perturbations ( Ross & Doshi-Velez , 2018 ; Jakubovitz & Giryes , 2018 ; Hoffman et al. , 2019 ) . Simard et al . ( 1992 ) proposed ” Tangent Prop ” , which minimized directional derivatives of classifiers in the direction of local inputtransformations ( e.g . rotations , translations ; called ” tangent vectors ” ) to reduce model sensitivity to such transformations . Czarnecki et al . ( 2017 ) proposed Sobolev training of neural networks to improve model distillation by matching the input-Jacobian of the original model . Regularizing the direction of the input-Jacobian has also been used to improve adversarial robustness ( Chan et al. , 2020 ) . In the present work , we regularize Fourier-components in the input-Jacobian to modify the spatial frequency sensitivity of models . As such , we are directly modifying the direction of the input-Jacobian instead of its norm . 3 METHODS . Preliminaries : We introduce all relevant definitions and notations before describing the proposed methods . Consider an image classification task with input images x , target labels y , and the standard cross-entropy loss function LCE . Let f denote any differentiable model , F ( · ) the unitary 2D discrete Fourier transform ( DFT ) , F−1 ( · ) its inverse , and F−1∗ ( · ) the adjoint of the inverse-Fourier transform , and let xf denote the Fourier space representation of the input , i.e . xf = F ( x ) . We denote the input-Jacobian in the standard input basis as ∂LCE∂x , and ∂LCE ∂xf as the input-Jacobian in the Fourier-basis . Let N denote the height of the input images x ( although not necessary , all images used in this work are square ) . The zero-shifted 2D-DFT of the input-Jacobian is denoted F = F ( ∂LCE∂x ) . Since the input-Jacobian typically has three color channels , they are averaged before computing the 2D-DFT . Fourier coefficients in F are complex numbers with real and imaginary components ; F ( u , v ) = Real ( u , v ) + Imag ( u , v ) , where ( u , v ) are indices of coefficients . The power in a coefficient is its squared amplitude , P ( u , v ) = |F ( u , v ) |2 = Real ( u , v ) 2 + Imag ( u , v ) 2 and the matrix of powers is denoted P ( power-matrix ) . Each coefficient has a radial distance r ( u , v ) from the centre of the matrix , r ( u , v ) = d ( ( u , v ) , ( cu , cv ) ) , where ( cu , cv ) denotes the centroid of P and d ( · , · ) is Euclidean distance . Distinct radial distances , rounded to the nearest integer , of coefficients in the matrix are the set of integers { 1 , . . . , N/ √ 2 } and correspond to low to high spatial frequency bands , the highest spatial frequency being limited by the Nyquist frequency . We denote PTotal as the total power in P , excluding the zero-frequency coefficient , PTotal = ∑ r ( u , v ) > =1 P ( u , v ) . Similarly , we define P̃Total as the total power in P excluding the zero-frequency coefficient as well as coefficients with radial distance r ( u , v ) > N/2 , i.e . outside the largest circle inscribed in the power-matrix P ; P̃Total = ∑ 1 < =r ( u , v ) < =N/2 P ( u , v ) ( please see Figure 5 in Appendix A.1 for an illustration ) . We denote Pk as the total power at radial distance k normalized by PTotal , Pk = 1 PTotal ∑ r ( u , v ) =k P ( u , v ) and P̃k as total power at radial distance k normalized by P̃Total in- stead , P̃k = 1P̃Total ∑ r ( u , v ) =k P ( u , v ) . 3.1 SPATIAL FREQUENCY SENSITIVITY OF A MODEL . In this section , we define the proposed spatial frequency sensitivity ( SFS ) of any differentiable model using its input-Jacobian represented in the Fourier-basis . As the input-Jacobian provides the direction of highest input sensitivity , we show below that its Fourier transform , which is simply a change of basis , provides the model ’ s sensitivities to spatial frequency components in the input . The input-Jacobian in the Fourier-basis , ∂LCE∂xf , comprises the sensitivities of the model with respect to individual frequencies in the input and we obtain this by simply computing the Fourier transform of the input-Jacobian ∂LCE∂x . In order to justify this , consider the computation graph where the input x is mapped to a scalar loss via a model f and loss function LCE ( Figure 2 ) . We introduce an implicit operation ( shown in red ) that maps the Fourier space representation of the input , xf , to the standard input x through the inverse Fourier-transform ; xf F−1−−−→ x , so in order to compute the input-Jacobian in the Fourier-basis , ∂LCE∂xf , we must differentiate through this implicit operation in the forward graph . Since the inverse-Fourier transform is a unitary operator , its adjoint is also its inverse , i.e . F−1∗ = ( F−1 ) −1 = F . Hence , following the chain rule for complex operators , ∂LCE∂xf is simply the Fourier transform of the input-Jacobian ∂LCE∂x . Input image Input-Jacobian of model 2D DFT input-Jacobian SFS=proportion of power in circular frequency bands ∂LCE ∂xf = F−1∗ ( ∂LCE ∂x ) = F ( ∂LCE ∂x ) Hence , even though we do not explicitly use the Fourier representation of the input , this shows that the Fourier transform of the input-Jacobian provides us the sensitivity of the model with respect to spatial frequencies of the input . In fact , similar results hold for any unitary operation on the inputJacobian , such as the discrete cosine transform ( DCT ) . Interested readers can replace the DFT ( F ) with other such transforms if they would like to understand the sensitivity of a model with respect to other components in the input . We now define the spatial frequency sensitivity fSFS ( x , y ) with respect to an individual input ( x , y ) as , fSFS ( x , y ) = { P1 , . . . , PN/√2 } where Pk is the proportion of total power in Fourier coefficients at radial distance k in the power matrix P of the Fourier-transformed input-Jacobian . The overall spatial frequency sensitivity of a model is defined as the expectation of fSFS ( x , y ) over the data distribution , i.e . fSFS ( · ; θ ) = E ( x , y ) ∼p [ fSFS ( x , y ) ] ( please see Figure 1 for an illustration and Algorithm 1 in Appendix A.1 ) . We note that although we use the Jacobian of the loss function to estimate the SFS of a model , this formulation is valid for other output functions and tasks as well . An alternative approach that does not require image labels is to compute the Jacobian of the model ’ s output logits or softmax probabilities . In addition , while the spatial frequency sensitivity assumes the input is spatial 2D data ( e.g . images ) , we can extend this approach to any n-dimensional data by using the n-dimensional Fourier-transform as well as to other computer vision tasks .

Summary: This paper proposes a measure for a model’s spatial frequency sensitivity (SFS) based on the input-Jacobian in the Fourier basis. With this measure, the authors observe standard CNN training biases towards certain particular spatial frequencies consistently across samples. Based on this measure, the authors propose a family of spatial frequency regularization techniques to suppress the model’s sensitivities to certain spatial frequencies.