Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets
Authors: Jonas Ngnawé
MARIANNE NJIFON
Jonathan Heek
Yann Dauphin
Venue: ICLR 2022 Submitted (2022)
Keywords: robustness
imagenet-c
mixup

Abstract: Deep networks have achieved impressive results on a range of well curated benchmark datasets. Surprisingly, their performance remains sensitive to perturbations that have little effect on human performance. In this work, we propose a novel extension of Mixup called Robustmix that regularizes networks to classify based on lower frequency spatial features. We show that this type of regularization improves robustness on a range of benchmarks such as Imagenet-C and Stylized Imagenet. It adds little computational overhead and furthermore does not require a priori knowledge of a large set of image transformations. We find that this approach further complements recent advances in model architecture and data augmentation attaining a state-of-the-art mCE of 44.8 with an EfficientNet-B8 model and RandAugment, which is a reduction of 16 mCE compared to the baseline.

Under review as a conference paper at ICLR 2022
ROBUSTMIX: IMPROVING ROBUSTNESS BY REGULAR- IZING THE FREQUENCY BIAS OF DEEP NETS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep networks have achieved impressive results on range of well curated bench- mark datasets. Surprisingly, their performance remains sensitive to perturbations that have little effect human performance. In this work, we propose novel extension of Mixup called Robustmix that regularizes networks to classify based lower frequency spatial features. We show that this type of regularization improves robustness on range of benchmarks such as Imagenet-C and Stylized Imagenet. It adds little computational overhead and furthermore does not require priori knowledge of large set of image transformations. We find that this approach further complements recent advances in model architecture and data augmentation attaining state-of-the off-the-art art mean corruption error (mCH) of 44.8 with an EfficientNet-B model and RandAugment, which is a reduction mCE compared to the baseline.
INTRODUCTION
Deep neural networks have achieved state-of-the-art accuracy across range of benchmark tasks such image segmentation (Ren al.) (015)) and speech recognition (Hannun al. (014). These successes have led to the widespread adoption of neural networks in many real-life applications. However, while these networks perform well on curated benchmark datasets, their performance can suffer greatly in the presence of small data corruptions (Szegedy al.
et al. 2014]]
Moosavi- Dezfooli et al., 2017 Athalye et al. 2018; Hendrycks & Dietterich 2018). This poses significant challenges the application of deep networks.
Hendrycks & Dietterich (2018) show that the accuracy a standard model on Imagenet can drop from 76% to 20% when evaluated images corrupted with small visual transformations. This shows modern networks are not robust to certain small shifts in the data distribution. That is a concern because such shifts are common in many real-life applications. Secondly, (Szegeyy al.) (2014) show the existence of adversarial perturbations which are imperceptible to humans but have a disproportionate effect the predictions of network. This raises significant concerns about the safety of using deep networks in critical applications such as self driving cars (Sitawarin et al. 2018).
These problems have led numerous proposals improve the robustness deep networks. Some of these methods such as those proposed by [Hendrycks et al. (2019) require a priori knowledge of the visual transformations the test domain. Others, such as Geirhos et al. (2018) use deep network to generate transformations which comes with significant computation cost.
This paper proposes new data augmentation technique to improve the robustness of deep networks by regularizing frequency bias. This new regularization technique is based on Mixup and has many advantages compared to related robustness regularizers: (1) does not require knowledge of large set of priori transformations, (2) it is inexpensive and (3) it doesn't 't have many syper-parametent The key idea is to bias the network to rely more on lower spatial frequencies to make predictions.
We demonstrate on Imagenet-C that this method works well with recent advances and reaches state-of-tte the-art mCE of 44.8 with 85.0 clean accuracy with Efficientnet-Bi and RandAugmeent(Cubkk let al., 2019). This is an improvement of 16 mCE compared to the baseline Efficientnet-Bi and matches VIT-L/16 (Dasovitskiy et al. 2020), which is trained on 300 more data. We find that our implementation of the method with DCT transform adds negligible overhead in experiments. We

---

Under review as a conference paper at ICLR 2022
find that Robustmix improves accuracy on Stylized- Stylized-Imagenee by up to 15 points and we show that it can increase adversarial robustness.
2 RELATED WORK
The proposed approach can be seen generalization of Mixup (Zhang et al. 2018). Mixup is data augmentation method that regularizes models to behave more linearly between examples. It does so by training the model on linear interpolations of two input examples and their respective labels. These new examples are generated as follows
J = m. x T2, A),
where 3i, D2 are input images
m mix(()1 8/2 A),
where yr, 322 are labels
with mix being the linear interpolation function
mi x A) Ao (1 A)
where A Beta(a, o),, is the Mixup coefficient hyper-parameter.
Zhang et al. (018) show that Mixup improves the accuracy of networks and can also improve the robustness of the network.
Augmix (Hendrycks et al. 2019) is a data augmentation technique to improve robustness by training mix of known image transformations. This method add little computational overhead, but requires knowledge of a diverse set of domain specific transformations. [eeddrycks et al. (2019) mixes of 9 different augmentation reach 68.4 mCE on Imagenet. In contrast, the proposed method does not rely specific image augmentations and instead relies on the more general principle that natural images are kind of signal where most of the energy is concentrated in the lower frequencies.
Zhang 2019) uses low pass filters directly inside the model to improve the frequency response of the network. Our method also makes use of low-pass filtering but does not completely remove high frequency features. Additionally, we only uses frequency filtering during training and therefore no computational overhead is incurred during evaluation.
3
METHOD
Dog
Boathouse (100%)
Law pacs fiteer
Minup
Boafhouse Dog
Weighted Sum [Band Energy)
Dog (100%)
High pasa Shtr
Mixup
Dog
Boathouse
Figure 1: Illustration of the method. In order to better illustrate the method, display the Fourier spectrum of the images next to them. We can see that even though 90% of the higher frequencies belong the image of dog, Robustmix assigns more weight to the boathouse label because assigns more weight to the lower frequencies.
In this section, we introduce novel extension of Mixup called Robustmix that increases robustness by regularizing the network to focus more on the low frequency features the signal.
Motivation Wang et al. (2020) suggest that convolutional networks trade robustness for accuracy in their use of high frequency image features. Such features can be perturbed in ways that change the prediction of the model even though humans cannot perceive the change. This can lead models to
2

---

Under review as a conference paper at ICLR 2022
make puzzling mistakes such with adversarial examples. Our aim is increase robustness while retaining accuracy by regularizing how high frequency information is used by the model.
Robustmix We propose to regularize the sensitivity of the model to each frequency band by extending Mixup's linear interpolations with a new type of band interpolation. The key insight is that we can condition the sensitivity to each band using images that mix the frequency bands of two different images. Suppose that we mix the lower frequency band of an image of a boathouse with the high frequency band of an image of dog. We can encourage sensitivity the lower band by training the model to predict dog for this mixed image. However, this approach is too simplistic because it completely disregards the impact of the image in the high band.
Instead, we interpolate the label of such mixed im- ages according to an estimate of the importance of each frequency band. We propose to use the relative amount of energy in each band as an estimate of the importance. Thus the sensitivity of the model to high frequency features will be proportional to their en- ergy contribution in natural images. And as we can see in Figure 2 most of the spectral energy in natural images is concentrated in the lower end of the spec- trum. This should limit the ability of high frequency perturbations unilaterally change the prediction.
Furthermore, we use linear interpolations of images like in mixup within each band instead of raw im- ages. This closely reflects the more common case where the features in the bands are merely corrupted instead of entirely swapped. It also has the benefit of encouraging linearity inside the same frequency
Figure 2: Plot of the cumulative energy in Im- agenet images as a function of the frequency cutoff.
band.
Specifically, the mixing formula for Robustmix is given by
Low(mix(x1, x2, Az), High(mix((1,, (2, A)),c) c) X.mix(Mi (VV1 V2, ÀL) + (1
W2, Aω)
where ÀL, Au Beta(a, a), is the Mixup coefficient are a low pass and high pass filter respectively with a uniformly sampled cutoff frequency e E [0, 1]. And A is the coefficient that determines how much weight is given the lower frequency band. is given by the relative amount of energy in the lower frequency band for natural images
and Low(, c)), High c)
Low(,,)) ||
This coefficient can be efficiently computed on mini-batch of examples.
Implementation Computational overhead is an important consideration for data augmentation techniques since training deep networks is computationally intensive and practitioners have limited computational budget. We note that many popular techniques such as Mixup (Zhang et al,, (2018) add little overhead.
The frequency separation is implemented using a Discrete Cosine Transform (DCT) to avoid the complex multiplication required by an Discrete Fourier Transform. We multiply the images with the 224x224 DCT matrix directly because the spatial dimensions are relatively small and (non-complex) matrix multiplication is well-optimized on modern accelerators. A batch images transformed into frequency space and the low and high pass filtered images must be transformed back image space. Additionally, we must apply the DCT transform over the and y dimension separately. Thus, 6 DCT matrix multiplications the forward pass of ResNet50 requires 3.87 GFLOPs (Hasanpour et al. 2016).
required which results in 0.2 GFLOPs per image. In contrast, just
In our implementation of Robustmix, reorder commutative operations (low pass and mixing) in order to compute the DCT only a single time per minibatch. The pseudocode is provided in Algorithm 1. where reverse is function that reverses the rows of its input matrix.
3

---

Under review as a conference paper at ICLR 2022
Algorithm Robustmix
Input: Minibatch of inputs X E RNXHHWWD and labels Y E RRXC, a EEE R Output: Augmented minibatch of inputs X E RNNWwHxD and labels E Nx ΛE AØ (eta o:) and e U(0,1) 1)
L + c)
H - 1-1 L
Aa f ||
|
X + mi (Lx reverse( ()), Ar.) + mi +mix(H, reverse(H() AH)
Y < mi reverse (Y), d dL + (1 - Ao) AH)
RESULTS
4.1 DATASETS AND METRICS
ImageNet. ImageNet (Deng et al. 2009) is classification dataset that contains 1.28 million training images and 50000 validation images with 1000 classes. We evaluate the common classification accuracy which will be referred to clean accuracy. We use the standard Resnet preprocessing resulting in images of size 224x224 (He et al. 2015). The standard models, without any additional data augmentation process, will be qualifted as the baseline.
This dataset is made of 15 types of corruption drawn from four main categories: noise, blur, weather and digital (Hendrycks & Dietterich] 2018). These corruptions are applied to the validation images of ImageNet at 5 different intensities or levels severity. Following (Hendrycks & Dietterich 2018), evaluate the robustness of our method by reporting its mean corruption error (mCE) normalized with respect to AlexNet errors:
ImageNet-C.
CE
Σ Ec,s
mCE
comruption
with CE2-
severily &
Total Number of Corruptions'
Σ. FAlexNet G,8
Stylized-ImageNet (SIN) is constructed from ImageNet by replacing the texture in the original image using style transfer, such that the texture gives a misleading cue about the image label (Geirhos et al., 2018). The 1000 classes from ImageNet are reduced to 16 shape categories, for instance all labels for dog species are grouped under one dog label, same for chair, car, etc. There are 1280 generated cue conflict images (80 per category). With SIN, we evaluate the classification accuracy (SIN accuracy) and measure the model's s shape bias. Following Geirhos et (0118 , the model's 's bias towards shape versus texture is measured as
correct shapes
shape bias
correct shapes + correct textures
4.2 EXPERIMENTAL SETUP
We chose to do evaluations on residual nets (ResNet-50 and ResNet-152) and EfficientNets (EffficienNeet-B0, EfficientNet-B1, EfficientNet-B3 and Eff)cientNet-B8). Experiments were run on 8x8 TPUv3 instances for the the bigger EfficientNets (EfficientNet-B5 and EfficientNet-BB) and the other experiments were on 4x4 TPUv3 slices. For the Resnet models, training setup outlined in Goyal et al. (2017). However, we use cosine learning rate Loshchilov & Hutter 2016) with single cycle for Resnets that trained for 600 epochs.
the same standard
4.3 ROBUSTNESS RESULTS
Imagenet-C First, we evaluate the effectiveness of the proposed method in improving robustness to the visual corruptions considered in Imagenet-C. In Table we can see that Robustmix consistently improves robustness to the considered transformations, with 15 point decrease in mCE over the baseline for ResNet-50. Robustmix with ResNet-50 achieves 61.2 mCE without degrading accuracy on the clean dataset compared to the baseline. In fact, find small improvement over the baseline
4

---

Under review as a conference paper at ICLR 2022
of 0.8% the clean emror. While Mixup yields larger gain of 1.9% the clean accuracy, find that Robustmix improves mCE by up to 6 points more than Mixup. These results also compare favorably to Augmix, which needs to be combined with training on Stylized ImageNet (SIN) to reduce the mCE by 12 points. And this improvement comes at significant cost to the accuracy due to the of the Stylized Imagenet dataset. We also observe similar trade-off between accuracy and robustness as we can observe in Figure 3 We observe that Mixup consistently produces lower clean error for smaller models, but the accuracy gap with Robustmix disappears the model gets bigger.
While it is not directly comparable to ViT-L/16 due to its use of 300 more data, we see that Efficientnet-B8 with Robustmix and RandAugment has better robustness 44.8 mCE. It is also competitive with DeepAugment (Hendrycks et al.] (0200) which requires training additional special- ized image-to-image models on tasks such as super-resolution to produce augmented images. By comparison, our approach does not rely on extra data or extra trained models.
Clean
Extra
Method
mCE
Size
Accuracy
Data
ResNet-50 Baseline (200 epochs)
76.3
76.9
26M
0
ResNet-50 Baseline (600 epochs)
76.3
78.1
26M
ResNet-50 BlurPool (Zhang 2019)
77.0
73.4
26M
0
ResNet-50 Mixup (200 epochs)
77.5
68.1
26M
ResNet-50 Mixup (600 epochs)
78.2
67.5
26M
0
ResNet-50 Augmix
77.6
68.4
26M
0
ResNet-50 Augmix SIN
74.8
64.9
26M
0
ResNet-50 Robustmix (600 epochs)
77.1
61.2
26M
0
EfficienttNetBOO Baseline
76.8
72.4
5.3M
0
EfficientNet-B0 Mixup (a = 0.2)
77.1
68.3
5.3M
0
EfficientNer-B30 Robustmix (ax = 0.2)
76.8
61.9
5.3M
EfficientNet-B1 Baseline
78.1
69.4
7.8M
0
EfficientNer-B11 Mixup (a = 01.)
78.9
64.7
7.8M
0
EfficientNNe-BB1 Robustmix (a 0.2)
78.7
57.8
7.8M
0
EfficientNet-B5 Baseline
82.7
65.6
30M
0
EfficientNet-Bs Mixup (a 0.2)
83.3
58.9
30M
0
EfficientNet-B5 Robustmix (a = 0.2)
83.3
51.7
30M
0
EfficientNee-135 RandAug+Robustmixx (a = 0.2)
83.8
48.7
30M
BiT m-r101x3 (Kolesnikov al. (020)
84.7
58.27
387.9M
12.7M
ResNeXt-101 32 544DDeepAuumeettAAugg
Extra
(Hendrycks al. (020))
79.9
44.5
88.8M
models
VIT-L/T6 (Dosovitskiy et al. 2020
85.2
45.5
304.7M
300M
EfficientNer-BS Baseline
83.4
60.8
87.4M
0
EfficientNet-B88 Robustmix (a 0.4)
84.4
49.8
87.4M
0
EfficientNer-B8 RandAug+Robustmix (a = 0.4)
85.0
44.8
87.4M
0
Table 1: Comparison of various models based on Imagenet accuracy and Imagenet- robustness
(mCH). The robustness results for BiT and ViT are as reported by Paul & Chen (2021).
In our voos--validtton of a, we found small values less than 0.2 perform poorly both accuracy and mCE. Values of a such that 0.2 < < 0.5 not only give the best accuracies and mCEs but also accuracy as bigger values
the best trade-off of mCE accuracy but do not do as well on mCE. In our experiments, we found that we typically achieve good results with a frequency cutoff e sampled between [0, 1] as described in Algorithm| I However, for ResNet-50 trained with a training budget that is too limited (200 instead of 600 epochs) and its smaller versions (ResNet-15 18 and ResNet-34), can be beneficial fix minimum 2 r for the cutoff by sampling in the interval [T, 1]. The minimum cutoff determines the range at which band mixing will occur. We can remove band interpolation entirely and recover standard Mixup by setting For Resnet-50 with too few training epochs, found that good value for the minimum is 0.1, but we found much better results can be achieved with 600 epochs without any modications to Algorithm [
have shown giving good values for
Stylized-ImageNeer We confirm that our method indeed increases both accuracy on Stylized Ima- geNet and the shape bias shown in table For ResNet-50, Robustmix almost doubles the shape

---

Under review as a conference paper at ICLR 2022
-a- Reskie-Se 50
Sux
-- Rechee-112
-8 Efrcerthie-12 10
75-
-8 Eftcerthie 21
-8. Effcerthie 20
Baseline
72-
Regaste
Rius
Bassiin
Mbup
Houp
Rebaatmik
Robustn
Clean Emerr
Figure 3: Highlighting the tradeoff between mCE and Clean Error for various models.
bias from baseline (from 19 37) and improves it by 63% over Mixup; while relative improvements on SIN accuracy are of 72% and 33% respectively over baseline and Mixup. The same observation for EfficientNet-B3 wich improves shape bias by near 50% and SIN accuracy by near 60 % over the baseline.
SIN
Shape Bias
Method/Parameters
Accuracy
ResNet-50 Baseline
15.6
19.25
ResNet-50 Mixup
20.1
22.7
ResNet-50 Robustmix
26.8
37.0
EfficientNNe-BB Baseline
25.3
44.4
EfficientNet-BB Mixup
28.75
48.3
ElficientNet-B5 Robustmix
40.3
66.1
Table 2: Accuracy and shape bias computed on Stylized Imagenet.
4.4 ADVERSARIAL PERTURBATIONS
In this section, we consider robustness to adversarial perturbations (Goodfellow et al. 2014). Adver- sarial perturbations make no visually distinguishable difference the human eye make models misclassify examples. Wang et al. (2020) and Yin et al. (2019) have shown that adversarial perturba- tions of unregularized models disproportionally affect higher frequencies. As we have seen in Figure 5] Robustmix encourages the model to rely more the lower frequencies make predictions. Our hypothesis that this will have a beneficial effect on adversarial robustness. For our experiment use of the first methods proposed in the Deep Learning community construct adversarial examples, the "Fast Gradient Sign example is constructed by adding perturbation proportional to the sign of the gradient of the loss with respect to the input: x = x + sign J(, y)) As seen in Figure 4 we find Robustmix is more robust than the baselines this type of adversarial attack.
(Goodfellow et al. (014).. The adversarial
4.5 ANALYSIS AND DISCUSSION
Low frequency bias In order quantify the degree to which models rely lower frequencies, measure how much accuracy drops as we remove higher frequency information with a low-pass filter. F shows that Robustmix is comparatively more robust to the removal of high frequencies. This indicates that models trained with Robustmix rely significantly less on these high- fighffrequency features make accurate predictions.

---

Under review as a conference paper at ICLR 2022
80
Robustmix
Baseline
50
40
10
Strength of perturbation
Figure 4: Test accuracy w.r.t. increasing adversarial perturbation strength. The comparison is done here ResNet-50 models for baseline, Mixup and Robustmix.
0.5
Accuracy
0.4
0.3
0.2
0.1
Mixup
Baseline
0.0
0.2
0.4
0.6
0.8
10
Frequency Cut-effs
Figure 5: Test accuracy Imagenet samples passed through a low-pass filter with increasing cut-off. As expected, we observe that Robustmix is more robust to the removal of high frequencies than Mixup. The comparison is done here ResNet-50 models.
CONCLUSION
In this paper, we have introduced new method to improve robustness called Robustmix which regularizes models to focus more on lower spatial frequencies to make predictions. We have range of benchmarks including Imagenet-C
shown that this method yields improved robustness and Stylized Imagenet. In particular, this approach attains an mCE of 44.8 on Imagenet-C with Efficientnet-B8. which competitive with models trained 300x more data.
Our method offers a promising new research direction for robustness with a number of open challenges. We have used a standard DCT based low-pass filter on images and L2 energy metric to determine the contribution of each label. This leaves many alternatives to be explored, such as: different data modalities like audio; more advanced frequency separation techniques like Wavelets; and alternative contribution metrics for mixing labels.
7

---

Under review as a conference paper at ICLR 2022
REFERENCES
Athalye, A., Engstrom, L., Ilyas, A., and Kwok, K. Synthesizing robust adversarial examples. In International conference on machine learning, pp. 284 284-293. PMLR, 2018.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data aug- mentation with a reduced search space, arxiv e-prints, page. arXiv preprint 2019.
Deng. J., Dong, W., Socher, R., Li, L.., Kai Li, and Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248- 255, 2009. doi: 10. 2000/CCPR.2009.520000
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
Goodfellow, I. J., Shlens, and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:/412.66: 2014.
Goyal, P. Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, I.., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint 17X6v 02677, 2017.
Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5560 2014.
Hasanpour, S. H., Rouhani, M., Fayyaz, M., and Sabokrou, M. Lets keep it simple, using simple archi- tectures to outperform deeper and more complex architectures. arXiv preprint arXiv: 1608.:666000000 06037, 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method improve robustness and uncertainty. In International Conference on Learning Representations, 2019.
Hendrycks, D., Basart, Mu, N.. Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces robustness: A critical analysis of oft-of--istribuue generalization. arXiv preprint arXiv:2006. 16241, 2020.
Kolesnikov, A., Beyer, L.,, Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, Big transfer (bit): General visual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part 16, pp. 491-507. Springer, 2020.
Loshchilov. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint 1608. 2016.
Moosavi-Dezfooli S.-M., Fawzi, A., Fawzi, O., and Frossard, P. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Paul, S. and Chen, P.-Y. Vision transformers are robust learners. arXiv preprint arXiv:2105.07581. 2021.
8

---

Under review as a conference paper at ICLR 2022
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91 18:91-99,00 2015.
Sitawarin, C., Bhagoji, A. N., Mosenia, A., Chiang, M., and Mittal, P. Darts: Deceiving autonomous cars with toxic signs. arXiv preprint arXiv: 06430, 2018.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Wang, H., Wu, X., Huang, Z., and Xing, E. P. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8684-8694, 2020.
Yin, D., Lopes, R. G., Shlens, J., Cubuk, E. D., and Gilmer, J. A fourier perspective model robustness in computer vision. arXiv preprint arXiv:1906. 08988, 2019.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhang, R. Making convolutional networks machine learning, pp. 7324-7334. PMLR, 2019.
again. In International conference


Reviews:
[{"review_id": "koDshO4e1V9", "reviewer": "Reviewer_EYUe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach is interesting but there are still some open questions:\n\n- Generalization of the approach: The generalization of the approach is not thoroughly discussed. The frequency sensitivity is dependent on the image type. The application of your approach e.g. to technical drawings with sharp edges will behave differently than applying it to images with smooth gradients. Furthermore, the discussion of the approach based on Stylized-ImageNet and ImageNet-C is not really convincing to discuss the quality of the approach in general. Stylized-ImageNet aims towards achieving robustness with respect to different (higher-order) textures. Hence, the basic construction metric of both approaches are related, so that similar results can be expected. \n\n- A comparison with images “augmented” by real sensor/environment noise would be interesting for the reader. Likewise, ImageNet-C only applies very artificial noise to the image dataset, which is not really comparable to real noise, especially the very artificial snow, rain and fog effects without consideration of the depth of the image scene.  \n- But of course, this is a serious point of criticism of many robustness augmentation approaches being proposed the last years since most of the proposed approaches are not really free of systematic domain shifts. \n\n- DCT calculation: it becomes not clear why tweo 1-D DCTs are separately calculated instead of a single 2-D DCT. The authors should clarify if the rationale behind this is just complexity and what is the impact w.r.t. accuracy.\n\nMinor comments:\n- Include the term mean corruption error to the paper abstract to explain the acronym mCE.\n\n", "summary_of_the_paper": "This paper discusses a new proposal on improving the robustness of image classification/segmentation by a frequency-guided data augmentation approach. The proposed technique is based on a DCT transformation to determine frequency bands with high energy in order with respect to their sensitivity. The data augmentation will be done by a linear combination of images weighted by the energy contribution of the different energy bands.", "summary_of_the_review": "In summary, I would encourage the authors to discuss the generalizability of the approach,before the paper can be recommended for acceptance at the conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WYckuPmhdMk", "reviewer": "Reviewer_3E2o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths\n\n    To my mind, the presented idea, while being rather simple for implementation, is well-sound with recent advances in the synthesis of signal processing and deep learning. The writing is clear and easy to understand. \n\n\n2. Weaknesses.\n\n    a. First of all, the idea of spectral mixing itself is not particularly novel. E.g., paper [1] that proposed to use it for the domain transfer problem, and [2] presented the *f-mixup* procedure for black-box adversarial attacks.\n\n    b. Although the approach of selecting $\\lambda_c$ according to the relative spectral energy looks inspiring and theoretically motivated, it needs an ablation study. How much does it outperform, e.g. mixing with the weights just proportional to $c$?\n\n    c. The influence of two preliminary Mixup steps also was not assessed. Are they really necessary? Does it not suffice just to mix two images in the spectral domain (i.e., set $\\lambda_L = 0, \\, \\lambda_H = 1$)?\n\n\n3. Questions.\n\n    a. Fig. 4 shows that in some cases stronger perturbation (e.g., the strength of ~5 vs ~12) leads to better test accuracy, especially in Robustmix and Mixup cases. Do the authors have any explanation for such behavior?\n\n4. References\n\n    [1] Yang and Soatto. FDA: Fourier Domain Adaptation for Semantic Segmentation. 2020.\n\n    [2] Li et al. F-mixup: Attack CNNs From Fourier Perspective. 2021.", "summary_of_the_paper": "This paper proposes a new spectral augmentation Robustmix, aiming to improve the robustness of image classifiers. In detail, the presented transformation consists of two preliminary Mixup steps and one final stage mixing low and high frequencies from the images obtained during the two initial steps. The pseudo-label is computed by weighing the labels, produced by Mixup steps, according to the relative amount of energy in low and high bands.\n\nAs the experiments show, applying Robustmix during training always leads to improved robustness to noise, corruptions, and adversarial perturbations, though in some cases trading off with a performance on vanilla datasets.", "summary_of_the_review": "1. Pre-rebuttal score.\n\n    Despite the interesting findings in the paper, currently, I rate the submission below the acceptance threshold. The reasons are twofold: Firstly, the idea is not completely novel. Secondly, the proper ablation study for the proposed augmentation was not conducted. I ask the authors to address this weakness during the rebuttal period.\n\n2. Post-rebuttal update.\n\n    After reading the authors' feedback, I keep the initial assessment. The paper misses a proper ablation study while presenting method is of limited novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1zxsnBbxfon", "reviewer": "Reviewer_5zdd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the method is easy to understand. There are lots of comparison runs, and these illuminate the strengths and weaknesses of the method to some degree. However, I find the evaluation still somewhat lacking.\n\nIn particular, the method appears to not improve over standard Mixup in terms of clean accuracy, and only wins convincingly when measuring the corruption error mCE. However, these corruptions include effects such as blur and weathering that corrupt the high frequencies, so this victory is not very surprising given how Robustmix explicitly creates a low-frequency bias.\n\nThe obvious challenger to Robustmix (and Mixup, for that matter) would be augmenting training data with corruptions that resemble the ones used in the measurements, but this approach has not been measured in isolation, except for AugMix with ResNet-50. But AugMix explicitly removes the sharpness augmentation which can remove high frequencies, so the benefit of RobustMix there (in terms of mCE) could be just because of its ability to corrupt high frequencies in the training data. RandAugment provides plenty of variation to training data, but it is not tested on its own, only when combined with Robustmix. I would have hoped to see tests with RandAugment but without Robustmix to gauge their effectiveness separately. Moreover, the tests with EfficientNet-B8 include neither Mixup or RandAugment alone, or their combination. As such, these measurements are not orthogonal enough to draw firm conclusions about the relative benefits of Robustmix.\n\nThe paper also does not measure a simpler alternative where low and high frequencies are taken from different images (without mixing) and the label is decided based on spectral energy as in Robustmix. Variants like this are briefly discussed on Page 3 but dismissed without measurements. The point about encouraging linearity within a frequency band makes sense to me, but it would be nice to see that backed up with data.\n\nThe extra tests in Sections 4.4 and 4.5, related to adversarial perturbations and low-frequency bias, are very interesting. However, their usefulness is limited because they do not specify the test setup, i.e., which network was used or how it was trained. For completeness, these tests should also discuss how the results compare against state of the art. It is to be expected that methods geared explicitly for these purposes (e.g., adversarial training) are better in that regard, but it would be interesting to know how far the proposed method gets.\n\nThe paper is missing a citation to RandAugment.\n", "summary_of_the_paper": "The paper proposes a novel data augmentation method for training classifiers. The idea is to mix two images in two different ways using standard Mixup, and then compose a final image by taking different frequency bands from the two mixed images. The training label is based on signal energy in different frequency bands, generally strongly favoring the label of the low-frequency content. Evaluations suggest the new scheme is more robust against image corruptions than previous methods, but not all relevant (combinations of) methods have been evaluated.", "summary_of_the_review": "The proposed technique is well described and simple to implement. Its benefits and weaknesses are left unclear by evaluation that includes key challengers such as Mixup, AugMix, and RandAugment only in scattered combinations in different network setups. However, the obtained mCE of 44.8 in Imagenet-C is a very good result, and the paper should emphasize this by showing that no previous methods or their combinations reach that when using the same network architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a data augmentation approach that extends Mixup with high- and low-pass filtering operations, in order to regularize deep networks towards focusing on low frequency components of the input signal.  Reviewers are unconvinced about the significance of the contribution.  Reviewer 5zdd notes that the method does not improve over standard Mixup in the absence of corruption error.  Reviewer 3E2o notes that "the idea of spectral mixing itself is not particularly novel", and also asks for ablation studies concerning the hyperparameters of the method; the author response unfortunately does not provide enough detail on ablation experiments.  The AC agrees with the reviewers and does not believe the author response has addressed weaknesses in a satisfactory manner.
Decision: Reject