Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets

Deep networks have achieved impressive results on a range of well curated benchmark datasets . Surprisingly , their performance remains sensitive to perturbations that have little effect on human performance . In this work , we propose a novel extension of Mixup called Robustmix that regularizes networks to classify based on lower frequency spatial features . We show that this type of regularization improves robustness on a range of benchmarks such as Imagenet-C and Stylized Imagenet . It adds little computational overhead and furthermore does not require a priori knowledge of a large set of image transformations . We find that this approach further complements recent advances in model architecture and data augmentation attaining a state-of-the-art mean corruption error ( mCE ) of 44.8 with an EfficientNet-B8 model and RandAugment , which is a reduction of 16 mCE compared to the baseline . 1 INTRODUCTION . Deep neural networks have achieved state-of-the-art accuracy across a range of benchmark tasks such as image segmentation ( Ren et al. , 2015 ) and speech recognition ( Hannun et al. , 2014 ) . These successes have led to the widespread adoption of neural networks in many real-life applications . However , while these networks perform well on curated benchmark datasets , their performance can suffer greatly in the presence of small data corruptions ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ; Moosavi-Dezfooli et al. , 2017 ; Athalye et al. , 2018 ; Hendrycks & Dietterich , 2018 ) . This poses significant challenges to the application of deep networks . Hendrycks & Dietterich ( 2018 ) show that the accuracy of a standard model on Imagenet can drop from 76 % to 20 % when evaluated on images corrupted with small visual transformations . This shows modern networks are not robust to certain small shifts in the data distribution . That is a concern because such shifts are common in many real-life applications . Secondly , Szegedy et al . ( 2014 ) show the existence of adversarial perturbations which are imperceptible to humans but have a disproportionate effect on the predictions of a network . This raises significant concerns about the safety of using deep networks in critical applications such as self driving cars ( Sitawarin et al. , 2018 ) . These problems have led to numerous proposals to improve the robustness of deep networks . Some of these methods such as those proposed by Hendrycks et al . ( 2019 ) require a priori knowledge of the visual transformations in the test domain . Others , such as Geirhos et al . ( 2018 ) use a deep network to generate transformations which comes with significant computation cost . This paper proposes a new data augmentation technique to improve the robustness of deep networks by regularizing frequency bias . This new regularization technique is based on Mixup and has many advantages compared to related robustness regularizers : ( 1 ) it does not require knowledge of a large set of priori transformations , ( 2 ) it is inexpensive and ( 3 ) it doesn ’ t have many hyper-parameters . The key idea is to bias the network to rely more on lower spatial frequencies to make predictions . We demonstrate on Imagenet-C that this method works well with recent advances and reaches a state-of-the-art mCE of 44.8 with 85.0 clean accuracy with Efficientnet-B8 and RandAugment ( Cubuk et al. , 2019 ) . This is an improvement of 16 mCE compared to the baseline Efficientnet-B8 and matches ViT-L/16 ( Dosovitskiy et al. , 2020 ) , which is trained on 300× more data . We find that our implementation of the method with DCT transform adds negligible overhead in our experiments . We find that Robustmix improves accuracy on Stylized-Imagenet by up to 15 points and we show that it can increase adversarial robustness . 2 RELATED WORK . The proposed approach can be seen as a generalization of Mixup ( Zhang et al. , 2018 ) . Mixup is a data augmentation method that regularizes models to behave more linearly between examples . It does so by training the model on linear interpolations of two input examples and their respective labels . These new examples are generated as follows x̃ = mix ( x1 , x2 , λ ) , where x1 , x2 are input images ỹ = mix ( y1 , y2 , λ ) , where y1 , y2 are labels with mix being the linear interpolation function mix ( x1 , x2 , λ ) = λx1 + ( 1− λ ) x2 ( 1 ) where λ ∼ Beta ( α , α ) , α is the Mixup coefficient hyper-parameter . Zhang et al . ( 2018 ) show that Mixup improves the accuracy of networks and can also improve the robustness of the network . Augmix ( Hendrycks et al. , 2019 ) is a data augmentation technique to improve robustness by training on a mix of known image transformations . This method add little computational overhead , but requires knowledge of a diverse set of domain specific transformations . Hendrycks et al . ( 2019 ) mixes a set of 9 different augmentation to reach 68.4 mCE on Imagenet . In contrast , the proposed method does not rely on specific image augmentations and instead relies on the more general principle that natural images are a kind of signal where most of the energy is concentrated in the lower frequencies . Zhang ( 2019 ) uses low pass filters directly inside the model to improve the frequency response of the network . Our method also makes use of low-pass filtering but does not completely remove high frequency features . Additionally , we only uses frequency filtering during training and therefore no computational overhead is incurred during evaluation . 3 METHOD . In this section , we introduce a novel extension of Mixup called Robustmix that increases robustness by regularizing the network to focus more on the low frequency features in the signal . Motivation Wang et al . ( 2020 ) suggest that convolutional networks trade robustness for accuracy in their use of high frequency image features . Such features can be perturbed in ways that change the prediction of the model even though humans can not perceive the change . This can lead models to make puzzling mistakes such as with adversarial examples . Our aim is to increase robustness while retaining accuracy by regularizing how high frequency information is used by the model . Robustmix We propose to regularize the sensitivity of the model to each frequency band by extending Mixup ’ s linear interpolations with a new type of band interpolation . The key insight is that we can condition the sensitivity to each band using images that mix the frequency bands of two different images . Suppose that we mix the lower frequency band of an image of a boathouse with the high frequency band of an image of a dog . We can encourage sensitivity to the lower band by training the model to predict dog for this mixed image . However , this approach is too simplistic because it completely disregards the impact of the image in the high band . 0.0 0.1 0.2 0.4 0.6 0.8 1.0 Low-pass cut-off 0.0 0.2 0.4 0.6 0.8 1.0 En er gy band . Specifically , the mixing formula for Robustmix is given by x̃ = Low ( mix ( x1 , x2 , λL ) , c ) + High ( mix ( x1 , x2 , λH ) , c ) ỹ = λcmix ( y1 , y2 , λL ) + ( 1− λc ) mix ( y1 , y2 , λH ) where λL , λH ∼ Beta ( α , α ) , α is the Mixup coefficient hyper-parameter , and Low ( · , c ) , High ( · , c ) are a low pass and high pass filter respectively with a uniformly sampled cutoff frequency c ∈ [ 0 , 1 ] . And λc is the coefficient that determines how much weight is given to the lower frequency band . It is given by the relative amount of energy in the lower frequency band for natural images λc = E [ ‖Low ( xi , c ) ‖2 ] E [ ‖xi‖2 ] . ( 2 ) This coefficient can be efficiently computed on a mini-batch of examples . Implementation Computational overhead is an important consideration for data augmentation techniques since training deep networks is computationally intensive and practitioners have limited computational budget . We note that many popular techniques such as Mixup ( Zhang et al. , 2018 ) add little overhead . The frequency separation is implemented using a Discrete Cosine Transform ( DCT ) to avoid the complex multiplication required by an Discrete Fourier Transform . We multiply the images with the 224x224 DCT matrix directly because the spatial dimensions are relatively small and ( non-complex ) matrix multiplication is well-optimized on modern accelerators . A batch of images is transformed into frequency space and the low and high pass filtered images must be transformed back to image space . Additionally , we must apply the DCT transform over the x and y dimension separately . Thus , 6 DCT matrix multiplications are required which results in 0.2 GFLOPs per image . In contrast , just the forward pass of ResNet50 requires 3.87 GFLOPs ( Hasanpour et al. , 2016 ) . In our implementation of Robustmix , we reorder commutative operations ( low pass and mixing ) in order to compute the DCT only a single time per minibatch . The pseudocode is provided in Algorithm 1 , where reverse is a function that reverses the rows of its input matrix . Algorithm 1 Robustmix Input : Minibatch of inputs X ∈ RN×H×W×D and labels Y ∈ RN×C , α ∈ R Output : Augmented minibatch of inputs X̃ ∈ RN×W×H×D and labels Ỹ ∈ RN×C λL , λH ∼ Beta ( α , α ) and c ∼ U ( 0 , 1 ) L← Low ( X , c ) H ← 1− L λc ← ‖L‖ 2 ‖X‖2 X̃ ← mix ( L , reverse ( L ) , λL ) + mix ( H , reverse ( H ) , λH ) Ỹ ← mix ( Y , reverse ( Y ) , λc ∗ λL + ( 1− λc ) ∗ λH ) 4 RESULTS . 4.1 DATASETS AND METRICS . ImageNet . ImageNet ( Deng et al. , 2009 ) is a classification dataset that contains 1.28 million training images and 50000 validation images with 1000 classes . We evaluate the common classification accuracy which will be referred to as clean accuracy . We use the standard Resnet preprocessing resulting in images of size 224x224 ( He et al. , 2015 ) . The standard models , without any additional data augmentation process , will be qualified as the baseline . ImageNet-C . This dataset is made of 15 types of corruption drawn from four main categories : noise , blur , weather and digital ( Hendrycks & Dietterich , 2018 ) . These corruptions are applied to the validation images of ImageNet at 5 different intensities or levels of severity . Following ( Hendrycks & Dietterich , 2018 ) , we evaluate the robustness of our method by reporting its mean corruption error ( mCE ) normalized with respect to AlexNet errors : mCE = ∑ corruption c CEc Total Number of Corruptions , with CEc = ∑ severity s Ec , s∑ sE AlexNet c , s Stylized-ImageNet . Stylized-ImageNet ( SIN ) is constructed from ImageNet by replacing the texture in the original image using style transfer , such that the texture gives a misleading cue about the image label ( Geirhos et al. , 2018 ) . The 1000 classes from ImageNet are reduced to 16 shape categories , for instance all labels for dog species are grouped under one dog label , same for chair , car , etc . There are 1280 generated cue conflict images ( 80 per category ) . With SIN , we evaluate the classification accuracy ( SIN accuracy ) and measure the model ’ s shape bias . Following Geirhos et al . ( 2018 ) , the model ’ s bias towards shape versus texture is measured as shape bias = correct shapes correct shapes + correct textures .

Summary: This paper proposes a new spectral augmentation Robustmix, aiming to improve the robustness of image classifiers. In detail, the presented transformation consists of two preliminary Mixup steps and one final stage mixing low and high frequencies from the images obtained during the two initial steps. The pseudo-label is computed by weighing the labels, produced by Mixup steps, according to the relative amount of energy in low and high bands.  As the experiments show, applying Robustmix during training always leads to improved robustness to noise, corruptions, and adversarial perturbations, though in some cases trading off with a performance on vanilla datasets.