Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses a new proposal on improving the robustness of image classification/segmentation by a frequency-guided data augmentation approach. The proposed technique is based on a DCT transformation to determine frequency bands with high energy in order with respect to their sensitivity. The data augmentation will be done by a linear combination of images weighted by the energy contribution of the different energy bands.
main_review: The approach is interesting but there are still some open questions:

- Generalization of the approach: The generalization of the approach is not thoroughly discussed. The frequency sensitivity is dependent on the image type. The application of your approach e.g. to technical drawings with sharp edges will behave differently than applying it to images with smooth gradients. Furthermore, the discussion of the approach based on Stylized-ImageNet and ImageNet-C is not really convincing to discuss the quality of the approach in general. Stylized-ImageNet aims towards achieving robustness with respect to different (higher-order) textures. Hence, the basic construction metric of both approaches are related, so that similar results can be expected. 

- A comparison with images “augmented” by real sensor/environment noise would be interesting for the reader. Likewise, ImageNet-C only applies very artificial noise to the image dataset, which is not really comparable to real noise, especially the very artificial snow, rain and fog effects without consideration of the depth of the image scene.  
- But of course, this is a serious point of criticism of many robustness augmentation approaches being proposed the last years since most of the proposed approaches are not really free of systematic domain shifts. 

- DCT calculation: it becomes not clear why tweo 1-D DCTs are separately calculated instead of a single 2-D DCT. The authors should clarify if the rationale behind this is just complexity and what is the impact w.r.t. accuracy.

Minor comments:
- Include the term mean corruption error to the paper abstract to explain the acronym mCE.


summary_of_the_review: In summary, I would encourage the authors to discuss the generalizability of the approach,before the paper can be recommended for acceptance at the conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel data augmentation method for training classifiers. The idea is to mix two images in two different ways using standard Mixup, and then compose a final image by taking different frequency bands from the two mixed images. The training label is based on signal energy in different frequency bands, generally strongly favoring the label of the low-frequency content. Evaluations suggest the new scheme is more robust against image corruptions than previous methods, but not all relevant (combinations of) methods have been evaluated.
main_review: The paper is well written and the method is easy to understand. There are lots of comparison runs, and these illuminate the strengths and weaknesses of the method to some degree. However, I find the evaluation still somewhat lacking.

In particular, the method appears to not improve over standard Mixup in terms of clean accuracy, and only wins convincingly when measuring the corruption error mCE. However, these corruptions include effects such as blur and weathering that corrupt the high frequencies, so this victory is not very surprising given how Robustmix explicitly creates a low-frequency bias.

The obvious challenger to Robustmix (and Mixup, for that matter) would be augmenting training data with corruptions that resemble the ones used in the measurements, but this approach has not been measured in isolation, except for AugMix with ResNet-50. But AugMix explicitly removes the sharpness augmentation which can remove high frequencies, so the benefit of RobustMix there (in terms of mCE) could be just because of its ability to corrupt high frequencies in the training data. RandAugment provides plenty of variation to training data, but it is not tested on its own, only when combined with Robustmix. I would have hoped to see tests with RandAugment but without Robustmix to gauge their effectiveness separately. Moreover, the tests with EfficientNet-B8 include neither Mixup or RandAugment alone, or their combination. As such, these measurements are not orthogonal enough to draw firm conclusions about the relative benefits of Robustmix.

The paper also does not measure a simpler alternative where low and high frequencies are taken from different images (without mixing) and the label is decided based on spectral energy as in Robustmix. Variants like this are briefly discussed on Page 3 but dismissed without measurements. The point about encouraging linearity within a frequency band makes sense to me, but it would be nice to see that backed up with data.

The extra tests in Sections 4.4 and 4.5, related to adversarial perturbations and low-frequency bias, are very interesting. However, their usefulness is limited because they do not specify the test setup, i.e., which network was used or how it was trained. For completeness, these tests should also discuss how the results compare against state of the art. It is to be expected that methods geared explicitly for these purposes (e.g., adversarial training) are better in that regard, but it would be interesting to know how far the proposed method gets.

The paper is missing a citation to RandAugment.

summary_of_the_review: The proposed technique is well described and simple to implement. Its benefits and weaknesses are left unclear by evaluation that includes key challengers such as Mixup, AugMix, and RandAugment only in scattered combinations in different network setups. However, the obtained mCE of 44.8 in Imagenet-C is a very good result, and the paper should emphasize this by showing that no previous methods or their combinations reach that when using the same network architecture.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new spectral augmentation Robustmix, aiming to improve the robustness of image classifiers. In detail, the presented transformation consists of two preliminary Mixup steps and one final stage mixing low and high frequencies from the images obtained during the two initial steps. The pseudo-label is computed by weighing the labels, produced by Mixup steps, according to the relative amount of energy in low and high bands.

As the experiments show, applying Robustmix during training always leads to improved robustness to noise, corruptions, and adversarial perturbations, though in some cases trading off with a performance on vanilla datasets.
main_review: 1. Strengths

    To my mind, the presented idea, while being rather simple for implementation, is well-sound with recent advances in the synthesis of signal processing and deep learning. The writing is clear and easy to understand. 


2. Weaknesses.

    a. First of all, the idea of spectral mixing itself is not particularly novel. E.g., paper [1] that proposed to use it for the domain transfer problem, and [2] presented the *f-mixup* procedure for black-box adversarial attacks.

    b. Although the approach of selecting $\lambda_c$ according to the relative spectral energy looks inspiring and theoretically motivated, it needs an ablation study. How much does it outperform, e.g. mixing with the weights just proportional to $c$?

    c. The influence of two preliminary Mixup steps also was not assessed. Are they really necessary? Does it not suffice just to mix two images in the spectral domain (i.e., set $\lambda_L = 0, \, \lambda_H = 1$)?


3. Questions.

    a. Fig. 4 shows that in some cases stronger perturbation (e.g., the strength of ~5 vs ~12) leads to better test accuracy, especially in Robustmix and Mixup cases. Do the authors have any explanation for such behavior?

4. References

    [1] Yang and Soatto. FDA: Fourier Domain Adaptation for Semantic Segmentation. 2020.

    [2] Li et al. F-mixup: Attack CNNs From Fourier Perspective. 2021.
summary_of_the_review: 1. Pre-rebuttal score.

    Despite the interesting findings in the paper, currently, I rate the submission below the acceptance threshold. The reasons are twofold: Firstly, the idea is not completely novel. Secondly, the proper ablation study for the proposed augmentation was not conducted. I ask the authors to address this weakness during the rebuttal period.

2. Post-rebuttal update.

    After reading the authors' feedback, I keep the initial assessment. The paper misses a proper ablation study while presenting method is of limited novelty.
review_title: null

Meta-review: The paper proposes a data augmentation approach that extends Mixup with high- and low-pass filtering operations, in order to regularize deep networks towards focusing on low frequency components of the input signal.  Reviewers are unconvinced about the significance of the contribution.  Reviewer 5zdd notes that the method does not improve over standard Mixup in the absence of corruption error.  Reviewer 3E2o notes that "the idea of spectral mixing itself is not particularly novel", and also asks for ablation studies concerning the hyperparameters of the method; the author response unfortunately does not provide enough detail on ablation experiments.  The AC agrees with the reviewers and does not believe the author response has addressed weaknesses in a satisfactory manner.
Decision: Reject