Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets

Deep networks have achieved impressive results on a range of well curated benchmark datasets . Surprisingly , their performance remains sensitive to perturbations that have little effect on human performance . In this work , we propose a novel extension of Mixup called Robustmix that regularizes networks to classify based on lower frequency spatial features . We show that this type of regularization improves robustness on a range of benchmarks such as Imagenet-C and Stylized Imagenet . It adds little computational overhead and furthermore does not require a priori knowledge of a large set of image transformations . We find that this approach further complements recent advances in model architecture and data augmentation attaining a state-of-the-art mean corruption error ( mCE ) of 44.8 with an EfficientNet-B8 model and RandAugment , which is a reduction of 16 mCE compared to the baseline . 1 INTRODUCTION . Deep neural networks have achieved state-of-the-art accuracy across a range of benchmark tasks such as image segmentation ( Ren et al. , 2015 ) and speech recognition ( Hannun et al. , 2014 ) . These successes have led to the widespread adoption of neural networks in many real-life applications . However , while these networks perform well on curated benchmark datasets , their performance can suffer greatly in the presence of small data corruptions ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ; Moosavi-Dezfooli et al. , 2017 ; Athalye et al. , 2018 ; Hendrycks & Dietterich , 2018 ) . This poses significant challenges to the application of deep networks . Hendrycks & Dietterich ( 2018 ) show that the accuracy of a standard model on Imagenet can drop from 76 % to 20 % when evaluated on images corrupted with small visual transformations . This shows modern networks are not robust to certain small shifts in the data distribution . That is a concern because such shifts are common in many real-life applications . Secondly , Szegedy et al . ( 2014 ) show the existence of adversarial perturbations which are imperceptible to humans but have a disproportionate effect on the predictions of a network . This raises significant concerns about the safety of using deep networks in critical applications such as self driving cars ( Sitawarin et al. , 2018 ) . These problems have led to numerous proposals to improve the robustness of deep networks . Some of these methods such as those proposed by Hendrycks et al . ( 2019 ) require a priori knowledge of the visual transformations in the test domain . Others , such as Geirhos et al . ( 2018 ) use a deep network to generate transformations which comes with significant computation cost . This paper proposes a new data augmentation technique to improve the robustness of deep networks by regularizing frequency bias . This new regularization technique is based on Mixup and has many advantages compared to related robustness regularizers : ( 1 ) it does not require knowledge of a large set of priori transformations , ( 2 ) it is inexpensive and ( 3 ) it doesn ’ t have many hyper-parameters . The key idea is to bias the network to rely more on lower spatial frequencies to make predictions . We demonstrate on Imagenet-C that this method works well with recent advances and reaches a state-of-the-art mCE of 44.8 with 85.0 clean accuracy with Efficientnet-B8 and RandAugment ( Cubuk et al. , 2019 ) . This is an improvement of 16 mCE compared to the baseline Efficientnet-B8 and matches ViT-L/16 ( Dosovitskiy et al. , 2020 ) , which is trained on 300× more data . We find that our implementation of the method with DCT transform adds negligible overhead in our experiments . We find that Robustmix improves accuracy on Stylized-Imagenet by up to 15 points and we show that it can increase adversarial robustness . 2 RELATED WORK . The proposed approach can be seen as a generalization of Mixup ( Zhang et al. , 2018 ) . Mixup is a data augmentation method that regularizes models to behave more linearly between examples . It does so by training the model on linear interpolations of two input examples and their respective labels . These new examples are generated as follows x̃ = mix ( x1 , x2 , λ ) , where x1 , x2 are input images ỹ = mix ( y1 , y2 , λ ) , where y1 , y2 are labels with mix being the linear interpolation function mix ( x1 , x2 , λ ) = λx1 + ( 1− λ ) x2 ( 1 ) where λ ∼ Beta ( α , α ) , α is the Mixup coefficient hyper-parameter . Zhang et al . ( 2018 ) show that Mixup improves the accuracy of networks and can also improve the robustness of the network . Augmix ( Hendrycks et al. , 2019 ) is a data augmentation technique to improve robustness by training on a mix of known image transformations . This method add little computational overhead , but requires knowledge of a diverse set of domain specific transformations . Hendrycks et al . ( 2019 ) mixes a set of 9 different augmentation to reach 68.4 mCE on Imagenet . In contrast , the proposed method does not rely on specific image augmentations and instead relies on the more general principle that natural images are a kind of signal where most of the energy is concentrated in the lower frequencies . Zhang ( 2019 ) uses low pass filters directly inside the model to improve the frequency response of the network . Our method also makes use of low-pass filtering but does not completely remove high frequency features . Additionally , we only uses frequency filtering during training and therefore no computational overhead is incurred during evaluation . 3 METHOD . In this section , we introduce a novel extension of Mixup called Robustmix that increases robustness by regularizing the network to focus more on the low frequency features in the signal . Motivation Wang et al . ( 2020 ) suggest that convolutional networks trade robustness for accuracy in their use of high frequency image features . Such features can be perturbed in ways that change the prediction of the model even though humans can not perceive the change . This can lead models to make puzzling mistakes such as with adversarial examples . Our aim is to increase robustness while retaining accuracy by regularizing how high frequency information is used by the model . Robustmix We propose to regularize the sensitivity of the model to each frequency band by extending Mixup ’ s linear interpolations with a new type of band interpolation . The key insight is that we can condition the sensitivity to each band using images that mix the frequency bands of two different images . Suppose that we mix the lower frequency band of an image of a boathouse with the high frequency band of an image of a dog . We can encourage sensitivity to the lower band by training the model to predict dog for this mixed image . However , this approach is too simplistic because it completely disregards the impact of the image in the high band . 0.0 0.1 0.2 0.4 0.6 0.8 1.0 Low-pass cut-off 0.0 0.2 0.4 0.6 0.8 1.0 En er gy band . Specifically , the mixing formula for Robustmix is given by x̃ = Low ( mix ( x1 , x2 , λL ) , c ) + High ( mix ( x1 , x2 , λH ) , c ) ỹ = λcmix ( y1 , y2 , λL ) + ( 1− λc ) mix ( y1 , y2 , λH ) where λL , λH ∼ Beta ( α , α ) , α is the Mixup coefficient hyper-parameter , and Low ( · , c ) , High ( · , c ) are a low pass and high pass filter respectively with a uniformly sampled cutoff frequency c ∈ [ 0 , 1 ] . And λc is the coefficient that determines how much weight is given to the lower frequency band . It is given by the relative amount of energy in the lower frequency band for natural images λc = E [ ‖Low ( xi , c ) ‖2 ] E [ ‖xi‖2 ] . ( 2 ) This coefficient can be efficiently computed on a mini-batch of examples . Implementation Computational overhead is an important consideration for data augmentation techniques since training deep networks is computationally intensive and practitioners have limited computational budget . We note that many popular techniques such as Mixup ( Zhang et al. , 2018 ) add little overhead . The frequency separation is implemented using a Discrete Cosine Transform ( DCT ) to avoid the complex multiplication required by an Discrete Fourier Transform . We multiply the images with the 224x224 DCT matrix directly because the spatial dimensions are relatively small and ( non-complex ) matrix multiplication is well-optimized on modern accelerators . A batch of images is transformed into frequency space and the low and high pass filtered images must be transformed back to image space . Additionally , we must apply the DCT transform over the x and y dimension separately . Thus , 6 DCT matrix multiplications are required which results in 0.2 GFLOPs per image . In contrast , just the forward pass of ResNet50 requires 3.87 GFLOPs ( Hasanpour et al. , 2016 ) . In our implementation of Robustmix , we reorder commutative operations ( low pass and mixing ) in order to compute the DCT only a single time per minibatch . The pseudocode is provided in Algorithm 1 , where reverse is a function that reverses the rows of its input matrix . Algorithm 1 Robustmix Input : Minibatch of inputs X ∈ RN×H×W×D and labels Y ∈ RN×C , α ∈ R Output : Augmented minibatch of inputs X̃ ∈ RN×W×H×D and labels Ỹ ∈ RN×C λL , λH ∼ Beta ( α , α ) and c ∼ U ( 0 , 1 ) L← Low ( X , c ) H ← 1− L λc ← ‖L‖ 2 ‖X‖2 X̃ ← mix ( L , reverse ( L ) , λL ) + mix ( H , reverse ( H ) , λH ) Ỹ ← mix ( Y , reverse ( Y ) , λc ∗ λL + ( 1− λc ) ∗ λH ) 4 RESULTS . 4.1 DATASETS AND METRICS . ImageNet . ImageNet ( Deng et al. , 2009 ) is a classification dataset that contains 1.28 million training images and 50000 validation images with 1000 classes . We evaluate the common classification accuracy which will be referred to as clean accuracy . We use the standard Resnet preprocessing resulting in images of size 224x224 ( He et al. , 2015 ) . The standard models , without any additional data augmentation process , will be qualified as the baseline . ImageNet-C . This dataset is made of 15 types of corruption drawn from four main categories : noise , blur , weather and digital ( Hendrycks & Dietterich , 2018 ) . These corruptions are applied to the validation images of ImageNet at 5 different intensities or levels of severity . Following ( Hendrycks & Dietterich , 2018 ) , we evaluate the robustness of our method by reporting its mean corruption error ( mCE ) normalized with respect to AlexNet errors : mCE = ∑ corruption c CEc Total Number of Corruptions , with CEc = ∑ severity s Ec , s∑ sE AlexNet c , s Stylized-ImageNet . Stylized-ImageNet ( SIN ) is constructed from ImageNet by replacing the texture in the original image using style transfer , such that the texture gives a misleading cue about the image label ( Geirhos et al. , 2018 ) . The 1000 classes from ImageNet are reduced to 16 shape categories , for instance all labels for dog species are grouped under one dog label , same for chair , car , etc . There are 1280 generated cue conflict images ( 80 per category ) . With SIN , we evaluate the classification accuracy ( SIN accuracy ) and measure the model ’ s shape bias . Following Geirhos et al . ( 2018 ) , the model ’ s bias towards shape versus texture is measured as shape bias = correct shapes correct shapes + correct textures .

Summary: This paper discusses a new proposal on improving the robustness of image classification/segmentation by a frequency-guided data augmentation approach. The proposed technique is based on a DCT transformation to determine frequency bands with high energy in order with respect to their sensitivity. The data augmentation will be done by a linear combination of images weighted by the energy contribution of the different energy bands.