Year: 2021

Abstract:
Deep networks have achieved impressive results on a range of well curated benchmark datasets. Surprisingly, their performance remains sensitive to perturbations that have little effect on human performance. In this work, we propose a novel extension of Mixup called Robustmix that regularizes networks to classify based on lower frequency spatial features. We show that this type of regularization improves robustness on a range of benchmarks such as Imagenet-C and Stylized Imagenet. It adds little computational overhead and furthermore does not require a priori knowledge of a large set of image transformations. We find that this approach further complements recent advances in model architecture and data augmentation attaining a state-of-the-art mean corruption error (mCE) of 44.8 with an EfficientNet-B8 model and RandAugment, which is a reduction of 16 mCE compared to the baseline.

text: Deep networks have achieved impressive results on a range of well curated benchmark datasets. Surprisingly, their performance remains sensitive to perturbations that have little effect on human performance. In this work, we propose a novel extension of Mixup called Robustmix that regularizes networks to classify based on lower frequency spatial features. We show that this type of regularization improves robustness on a range of benchmarks such as Imagenet-C and Stylized Imagenet. It adds little computational overhead and furthermore does not require a priori knowledge of a large set of image transformations. We find that this approach further complements recent advances in model architecture and data augmentation attaining a state-of-the-art mean corruption error (mCE) of 44.8 with an EfficientNet-B8 model and RandAugment, which is a reduction of 16 mCE compared to the baseline.

heading: 1 INTRODUCTION
text: Deep neural networks have achieved state-of-the-art accuracy across a range of benchmark tasks such as image segmentation (Ren et al., 2015) and speech recognition (Hannun et al., 2014). These successes have led to the widespread adoption of neural networks in many real-life applications. However, while these networks perform well on curated benchmark datasets, their performance can suffer greatly in the presence of small data corruptions (Szegedy et al., 2014; Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2017; Athalye et al., 2018; Hendrycks & Dietterich, 2018). This poses significant challenges to the application of deep networks.
Hendrycks & Dietterich (2018) show that the accuracy of a standard model on Imagenet can drop from 76% to 20% when evaluated on images corrupted with small visual transformations. This shows modern networks are not robust to certain small shifts in the data distribution. That is a concern because such shifts are common in many real-life applications. Secondly, Szegedy et al. (2014) show the existence of adversarial perturbations which are imperceptible to humans but have a disproportionate effect on the predictions of a network. This raises significant concerns about the safety of using deep networks in critical applications such as self driving cars (Sitawarin et al., 2018).
These problems have led to numerous proposals to improve the robustness of deep networks. Some of these methods such as those proposed by Hendrycks et al. (2019) require a priori knowledge of the visual transformations in the test domain. Others, such as Geirhos et al. (2018) use a deep network to generate transformations which comes with significant computation cost.
This paper proposes a new data augmentation technique to improve the robustness of deep networks by regularizing frequency bias. This new regularization technique is based on Mixup and has many advantages compared to related robustness regularizers: (1) it does not require knowledge of a large set of priori transformations, (2) it is inexpensive and (3) it doesn’t have many hyper-parameters. The key idea is to bias the network to rely more on lower spatial frequencies to make predictions.
We demonstrate on Imagenet-C that this method works well with recent advances and reaches a state-of-the-art mCE of 44.8 with 85.0 clean accuracy with Efficientnet-B8 and RandAugment(Cubuk et al., 2019). This is an improvement of 16 mCE compared to the baseline Efficientnet-B8 and matches ViT-L/16 (Dosovitskiy et al., 2020), which is trained on 300× more data. We find that our implementation of the method with DCT transform adds negligible overhead in our experiments. We
find that Robustmix improves accuracy on Stylized-Imagenet by up to 15 points and we show that it can increase adversarial robustness.

heading: 2 RELATED WORK
text: The proposed approach can be seen as a generalization of Mixup (Zhang et al., 2018). Mixup is a data augmentation method that regularizes models to behave more linearly between examples. It does so by training the model on linear interpolations of two input examples and their respective labels. These new examples are generated as follows
x̃ = mix(x1, x2, λ), where x1, x2 are input images ỹ = mix(y1, y2, λ), where y1, y2 are labels
with mix being the linear interpolation function
mix(x1, x2, λ) = λx1 + (1− λ)x2 (1)
where λ ∼ Beta(α, α), α is the Mixup coefficient hyper-parameter. Zhang et al. (2018) show that Mixup improves the accuracy of networks and can also improve the robustness of the network.
Augmix (Hendrycks et al., 2019) is a data augmentation technique to improve robustness by training on a mix of known image transformations. This method add little computational overhead, but requires knowledge of a diverse set of domain specific transformations. Hendrycks et al. (2019) mixes a set of 9 different augmentation to reach 68.4 mCE on Imagenet. In contrast, the proposed method does not rely on specific image augmentations and instead relies on the more general principle that natural images are a kind of signal where most of the energy is concentrated in the lower frequencies.
Zhang (2019) uses low pass filters directly inside the model to improve the frequency response of the network. Our method also makes use of low-pass filtering but does not completely remove high frequency features. Additionally, we only uses frequency filtering during training and therefore no computational overhead is incurred during evaluation.

heading: 3 METHOD
text: In this section, we introduce a novel extension of Mixup called Robustmix that increases robustness by regularizing the network to focus more on the low frequency features in the signal.
Motivation Wang et al. (2020) suggest that convolutional networks trade robustness for accuracy in their use of high frequency image features. Such features can be perturbed in ways that change the prediction of the model even though humans cannot perceive the change. This can lead models to
make puzzling mistakes such as with adversarial examples. Our aim is to increase robustness while retaining accuracy by regularizing how high frequency information is used by the model.
Robustmix We propose to regularize the sensitivity of the model to each frequency band by extending Mixup’s linear interpolations with a new type of band interpolation. The key insight is that we can condition the sensitivity to each band using images that mix the frequency bands of two different images. Suppose that we mix the lower frequency band of an image of a boathouse with the high frequency band of an image of a dog. We can encourage sensitivity to the lower band by training the model to predict dog for this mixed image. However, this approach is too simplistic because it completely disregards the impact of the image in the high band.
0.0 0.1 0.2 0.4 0.6 0.8 1.0 Low-pass cut-off
0.0
0.2
0.4
0.6
0.8
1.0
En er
gy
band.
Specifically, the mixing formula for Robustmix is given by
x̃ = Low(mix(x1, x2, λL), c) + High(mix(x1, x2, λH), c)
ỹ = λcmix(y1, y2, λL) + (1− λc)mix(y1, y2, λH)
where λL, λH ∼ Beta(α, α), α is the Mixup coefficient hyper-parameter, and Low(·, c),High(·, c) are a low pass and high pass filter respectively with a uniformly sampled cutoff frequency c ∈ [0, 1]. And λc is the coefficient that determines how much weight is given to the lower frequency band. It is given by the relative amount of energy in the lower frequency band for natural images
λc = E[‖Low(xi, c)‖2]
E[‖xi‖2] . (2)
This coefficient can be efficiently computed on a mini-batch of examples.
Implementation Computational overhead is an important consideration for data augmentation techniques since training deep networks is computationally intensive and practitioners have limited computational budget. We note that many popular techniques such as Mixup (Zhang et al., 2018) add little overhead.
The frequency separation is implemented using a Discrete Cosine Transform (DCT) to avoid the complex multiplication required by an Discrete Fourier Transform. We multiply the images with the 224x224 DCT matrix directly because the spatial dimensions are relatively small and (non-complex) matrix multiplication is well-optimized on modern accelerators. A batch of images is transformed into frequency space and the low and high pass filtered images must be transformed back to image space. Additionally, we must apply the DCT transform over the x and y dimension separately. Thus, 6 DCT matrix multiplications are required which results in 0.2 GFLOPs per image. In contrast, just the forward pass of ResNet50 requires 3.87 GFLOPs (Hasanpour et al., 2016).
In our implementation of Robustmix, we reorder commutative operations (low pass and mixing) in order to compute the DCT only a single time per minibatch. The pseudocode is provided in Algorithm 1, where reverse is a function that reverses the rows of its input matrix.
Algorithm 1 Robustmix Input: Minibatch of inputs X ∈ RN×H×W×D and labels Y ∈ RN×C , α ∈ R Output: Augmented minibatch of inputs X̃ ∈ RN×W×H×D and labels Ỹ ∈ RN×C λL, λH ∼ Beta(α, α) and c ∼ U(0, 1) L← Low(X, c) H ← 1− L λc ← ‖L‖ 2
‖X‖2
X̃ ← mix(L, reverse(L), λL) + mix(H, reverse(H), λH) Ỹ ← mix(Y, reverse(Y ), λc ∗ λL + (1− λc) ∗ λH)

heading: 4 RESULTS

heading: 4.1 DATASETS AND METRICS
text: ImageNet. ImageNet (Deng et al., 2009) is a classification dataset that contains 1.28 million training images and 50000 validation images with 1000 classes. We evaluate the common classification accuracy which will be referred to as clean accuracy. We use the standard Resnet preprocessing resulting in images of size 224x224 (He et al., 2015). The standard models, without any additional data augmentation process, will be qualified as the baseline.
ImageNet-C. This dataset is made of 15 types of corruption drawn from four main categories: noise, blur, weather and digital (Hendrycks & Dietterich, 2018). These corruptions are applied to the validation images of ImageNet at 5 different intensities or levels of severity. Following (Hendrycks & Dietterich, 2018), we evaluate the robustness of our method by reporting its mean corruption error (mCE) normalized with respect to AlexNet errors:
mCE =
∑ corruption c CEc
Total Number of Corruptions , with CEc =
∑ severity s
Ec,s∑ sE AlexNet c,s
Stylized-ImageNet. Stylized-ImageNet (SIN) is constructed from ImageNet by replacing the texture in the original image using style transfer, such that the texture gives a misleading cue about the image label (Geirhos et al., 2018). The 1000 classes from ImageNet are reduced to 16 shape categories, for instance all labels for dog species are grouped under one dog label, same for chair, car, etc. There are 1280 generated cue conflict images (80 per category). With SIN, we evaluate the classification accuracy (SIN accuracy) and measure the model’s shape bias. Following Geirhos et al. (2018), the model’s bias towards shape versus texture is measured as
shape bias = correct shapes
correct shapes + correct textures .

heading: 4.2 EXPERIMENTAL SETUP
text: We chose to do evaluations on residual nets (ResNet-50 and ResNet-152) and EfficientNets (EfficientNet-B0, EfficientNet-B1, EfficientNet-B5 and EfficientNet-B8). Experiments were run on 8x8 TPUv3 instances for the the bigger EfficientNets (EfficientNet-B5 and EfficientNet-B8); and the other experiments were run on 4x4 TPUv3 slices. For the Resnet models, we use the same standard training setup outlined in Goyal et al. (2017). However, we use cosine learning rate Loshchilov & Hutter (2016) with a single cycle for Resnets that are trained for 600 epochs.

heading: 4.3 ROBUSTNESS RESULTS
text: Imagenet-C First, we evaluate the effectiveness of the proposed method in improving robustness to the visual corruptions considered in Imagenet-C. In Table 1, we can see that Robustmix consistently improves robustness to the considered transformations, with a 15 point decrease in mCE over the baseline for ResNet-50. Robustmix with ResNet-50 achieves 61.2 mCE without degrading accuracy on the clean dataset compared to the baseline. In fact, we find a small improvement over the baseline
of 0.8% on the clean error. While Mixup yields a larger gain of 1.9% on the clean accuracy, we find that Robustmix improves mCE by up to 6 points more than Mixup. These results also compare favorably to Augmix, which needs to be combined with training on Stylized ImageNet (SIN) to reduce the mCE by 12 points. And this improvement comes at significant cost to the accuracy due to the use of the Stylized Imagenet dataset. We also observe a similar trade-off between accuracy and robustness as we can observe in Figure 3. We observe that Mixup consistently produces lower clean error for smaller models, but the accuracy gap with Robustmix disappears as the model gets bigger.
While it is not directly comparable to ViT-L/16 due to its use of 300× more data, we see that Efficientnet-B8 with Robustmix and RandAugment has better robustness at 44.8 mCE. It is also competitive with DeepAugment (Hendrycks et al., 2020) which requires training additional specialized image-to-image models on tasks such as super-resolution to produce augmented images. By comparison, our approach does not rely on extra data or extra trained models.
In our cross-validation of α, we found small values less than 0.2 perform poorly both on accuracy and mCE. Values of α such that 0.2 ≤ α ≤ 0.5 not only give the best accuracies and mCEs but also the best trade-off of mCE versus accuracy as bigger values of α have shown giving good values for accuracy but do not do as well on mCE. In our experiments, we found that we typically achieve good results with a frequency cutoff c sampled between [0, 1] as described in Algorithm 1. However, for ResNet-50 trained with a training budget that is too limited (200 instead of 600 epochs) and its smaller versions (ResNet-18 and ResNet-34), it can be beneficial to fix a minimum c ≥ τ for the cutoff by sampling in the interval [τ, 1]. The minimum cutoff determines the range at which band mixing will occur. We can remove band interpolation entirely and recover standard Mixup by setting τ = 1. For Resnet-50 with too few training epochs, we found that a good value for the minimum is 0.1, but we found much better results can be achieved with 600 epochs without any modications to Algorithm 1.
Stylized-ImageNet. We confirm that our method indeed increases both accuracy on Stylized ImageNet and the shape bias as shown in table 2. For ResNet-50, Robustmix almost doubles the shape
bias from baseline (from 19 to 37) and improves it by 63% over Mixup; while relative improvements on SIN accuracy are of 72% and 33% respectively over baseline and Mixup. The same observation for EfficientNet-B5 wich improves shape bias by near 50% and SIN accuracy by near 60 % over the baseline.

heading: 4.4 ADVERSARIAL PERTURBATIONS
text: In this section, we consider robustness to adversarial perturbations (Goodfellow et al., 2014). Adversarial perturbations make no visually distinguishable difference to the human eye but make models misclassify examples. Wang et al. (2020) and Yin et al. (2019) have shown that adversarial perturbations of unregularized models disproportionally affect higher frequencies. As we have seen in Figure 5, Robustmix encourages the model to rely more on the lower frequencies to make predictions. Our hypothesis is that this will have a beneficial effect on adversarial robustness. For our experiment we use one of the first methods proposed in the Deep Learning community to construct adversarial examples, the ”Fast Gradient Sign Method”(FGSM) (Goodfellow et al., 2014). The adversarial example is constructed by adding a perturbation proportional to the sign of the gradient of the loss with respect to the input: x = x + sign (∆xJ(θ,x, y)) . As seen in Figure 4, we find Robustmix is more robust than the baselines to this type of adversarial attack.

heading: 4.5 ANALYSIS AND DISCUSSION
text: Low frequency bias In order to quantify the degree to which models rely on lower frequencies, we measure how much accuracy drops as we remove higher frequency information with a low-pass filter. Figure 5 shows that Robustmix is comparatively more robust to the removal of high frequencies. This indicates that models trained with Robustmix rely significantly less on these high-frequency features to make accurate predictions.

heading: 5 CONCLUSION
text: In this paper, we have introduced a new method to improve robustness called Robustmix which regularizes models to focus more on lower spatial frequencies to make predictions. We have shown that this method yields improved robustness on a range of benchmarks including Imagenet-C and Stylized Imagenet. In particular, this approach attains an mCE of 44.8 on Imagenet-C with Efficientnet-B8, which is competitive with models trained on 300× more data. Our method offers a promising new research direction for robustness with a number of open challenges. We have used a standard DCT based low-pass filter on images and L2 energy metric to determine the contribution of each label. This leaves many alternatives to be explored, such as: different data modalities like audio; more advanced frequency separation techniques like Wavelets; and alternative contribution metrics for mixing labels.

Review summaries:
This paper proposes a new spectral augmentation Robustmix, aiming to improve the robustness of image classifiers. In detail, the presented transformation consists of two preliminary Mixup steps and one final stage mixing low and high frequencies from the images obtained during the two initial steps. The pseudo-label is computed by weighing the labels, produced by Mixup steps, according to the relative amount of energy in low and high bands.  As the experiments show, applying Robustmix during training always leads to improved robustness to noise, corruptions, and adversarial perturbations, though in some cases trading off with a performance on vanilla datasets.

This paper discusses a new proposal on improving the robustness of image classification/segmentation by a frequency-guided data augmentation approach. The proposed technique is based on a DCT transformation to determine frequency bands with high energy in order with respect to their sensitivity. The data augmentation will be done by a linear combination of images weighted by the energy contribution of the different energy bands.

The paper proposes a novel data augmentation method for training classifiers. The idea is to mix two images in two different ways using standard Mixup, and then compose a final image by taking different frequency bands from the two mixed images. The training label is based on signal energy in different frequency bands, generally strongly favoring the label of the low-frequency content. Evaluations suggest the new scheme is more robust against image corruptions than previous methods, but not all relevant (combinations of) methods have been evaluated.