Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a metric to quantify the perceptual similarity between different kind of corruptions and uses it to show that training on a corruptions induces robustness to other corruptions which are perceptually similar. Analyzing the current data augmentation based methods for robustness using this lens, the authors hypothesis that we are currently overfitting to the existing robustness benchmark (ImageNet-C) and proposes a new benchmark with a new set of corruptions. 

I think the author's observations that training on some corruptions helps the network to be robust to similar corruptions in test time is quite intuitive. I appreciate that the paper tries to quantify this and performs an extensive empirical study. The insight from this study and the new proposed benchmark will be useful to further advance the research in this area. 

I have a some questions/clarifications:
1. What are the augmentations that were used while training the network used to extract features for the metric? Does the metric will probably depend on the architecture of the network used as well. Have you verified that the metric is robust to different architectures and the same conclusion holds? I am not sure if this is in the supplementary material somewhere and I missed it. 
2. Do we find any non-intuitive pairs of corruptions which are similar? It occurs to me that most geometric based corruptions are similar and noise based corruptions are similar etc, but is there any pair of corruptions across these groups that the metric identifies as similar?


Update after rebuttal: I appreciate the author response. I will maintain my original score.

review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces the Minimal Sample Distance (MSD): a measure of the minimal distance, in a trained network representation space, between samples modified with an augmentation and the average of all samples modified by a corruption. It uses this metric to claim that there exists a high correlation between the corruption error and the MSD of a given augmentation. This way, it claims that focusing on benchmarks like ImageNet-C may lead to overfitting to the corruptions present in that benchmark. 

One problem is that this correlation isn’t true for all augmentations. Only 4 are highlighted in the main text. As the paper describes, a few corruptions have spearman coefficient of less than 0.5. Particularly notable is brightness which, despite having very low spearman coefficient, is used to show that Patch Gaussian has “overfit” to the noise corruptions in the ImageNet-C benchmark. This is especially worrying since in their original paper, Patch Gaussian shows improvement in non-noise as well, which couldn’t have come from overfitting. Additionally, why was AutoAugment, but not RandAugment tested?

Another concern is that it may not make sense to compare single augmentations (such as Patch Gaussian) with augmentation policies, such as AutoAugment, RandAugment, and AugMix. It’s possible that individual augmentations in these policies “overfit” to the corruptions as well, but that this isn’t shown in MSD due to the use of many corruptions. In which case, using PatchGaussian in the RandAugment search space (for instance) would resolve any issues. 

The paper argues that AutoAugment "overfits" while AugMix doesn't, but that's only because AugMix explicitly removed any augmentations in ImageNet-C from its search space, so it would make sense to repeat this experiment with the augmentations present in AugMix in order to confirm that it doesn't indeed "overfit".

The paper then suggests that one solution would be to use MSD to sample dissimilar corruptions to test on. However, given that it seems like there’s no evidence of overfitting for augmentation policies that encourage a diversity of augmentations, such as AugMix (which is in line with Yin et al 2019). Then I’m not sure what the benefit of expanding the robustness benchmark is. If current methods have indeed overfit, why won’t we also overfit to the new benchmark’s corruptions? 

Overall, the paper presents interesting results and discussion. 

Update after rebuttal: I appreciate the authors' response and clarifications. I maintain my original score.

review_title: Interesting analysis, but some strange comparisons

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary 
The paper studies the importance of similarity between augmentations and corruptions for improving performance on those corruptions. To measure the distance between the augmentation and corruption distributions, the paper proposes a new metric, Minimal Sample Distance (MSD), which is the perceptual similarity between an average corruption and the closest augmentation from a finite set of samples sampled from the augmented data distribution. It is shown that MSD overcomes the drawbacks of distributional distance measures like Maximum Mean Discrepancy (MMD). A new benchmark, called ImageNet-C-bar, made up of corruptions that are perceptually dissimilar to ImageNet-C, is introduced. Using standard evaluation, it is empirically shown that several recent augmentation schemes show degraded performance on the new dataset, suggesting that they generate augmentations only perceptually similar to ImageNet-C and thus are prone to overfitting. 

+ves
+ Although the notion of the relation between augmentations and test-time corruptions has somewhat already been empirically observed and stated in many previous works, the paper tries to correlate this relation statistically. To my knowledge, this is the first such work.
+ Through comprehensive evaluations, the paper shows that the proposition of computing MSD rather than MMD correlates well with the relation between augmentations and corruptions observed in reality.
+ A new benchmark, ImageNet-C-bar is proposed, which shows a useful result to the community that recent SOTA augmentation methods have degraded performance on the new dataset because they generate augmentations close to ImageNet-C corruptions. 

Concerns
- The paper says (pg 2) - “we empirically show an intuitive yet surprisingly overlooked finding: Augmentation-corruption perceptual similarity is a strong predictor of corruption error”. However, this notion of the relation between augmentations and test-time corruptions does not seem very surprising. It’s perhaps well-known that DNNs will generalize well only when test distributions are fairly similar to training distributions. Hence, the importance of this observation does seem to be overemphasized, although this is certainly of use.  

- It's been recently shown that removing texture biases by introducing stylized transformations also improves robustness to common-corruptions (Geirhos et al, ICLR 2019). However, stylized transformations don’t look close to any Imagenet-C corruptions. If MSD of stylized transforms is large (which intuitively seems so), then it will mean that MSD is not a reliable metric in such a case. Was this studied? Would MSD be a reliable metric even in such cases? It would be good to understand this, to get a more well-rounded picture of the proposed metric. 

- Paper introduces MSD as a distance metric. However, distance metrics should be symmetric in nature. It is not quite evident from just Eq-1 if MSD is symmetric. 

- In addition, adding high severity corruption as training augmentation leads to better performance on the same low severity corruption but vice-versa is not true in general. This suggests that measures should perhaps ideally be asymmetric. Clarifying the notion of “distance metric” in this work may be important to make the work mathematically correct.

- Standard choices for measuring perceptual distances are VGG-16 or 19 networks pre-trained on ImageNet. However, the paper chooses to use WRN-40-10 trained on CIFAR-10. This seems to deviate from standard settings. The paper should explain the rationale behind their choice. It will be great to show an ablation on how this choice of feature extractor (VGG-19, Robust VGG etc) affects the MSD. Ideally, the metric should be robust i.e. shouldn’t be sensitive to the choice of feature extractor.

- The practical utility of ImageNet-C-bar seems limited and unclear. The only use that I can think of is using it to identify overfitting on ImageNet-C. I would be happy to understand what I am missing here.

POST-REBUTTAL:

I thank the authors for the response and the revisions to the paper. I appreciate the authors' efforts towards the rebuttal. I am however left with some concerns which did not have a convincing resolution:

* Regarding the comment on how the proposed analysis would look for stylized transforms, the authors say in the response that "...we don’t expect that perceptual similarity is the only cause of improved corruption error, only that perceptual similarity is particularly salient for predicting generalization to dissimilar corruptions...". The work seems to be one-sided in this regard. If stylized transforms don't look perceptually similar to ImageNet-C corruptions but provide robustness, this counters the proposed hypothesis. It then becomes important to say where the proposed analysis is meaningful and where it is not. This seems to be lacking at this time in the work.

* Regarding the robustness of MSD to the choice of feature extractor (as also asked by R1), the revised paper includes results on VGG as feature extractor (thanks to the authors for this), but uses a model that is finetuned for CIFAR-10. In general, perceptual similarity is studied directly taking VGG pre-trained on ImageNet - without finetuning on the target dataset. This leaves this question open, and makes one wonder if the latter features did not support the analysis.

* The utility of Imagenet-C-bar as an additional benchmark to check the goodness of performance on Imagenet-C seems a bit convoluted. Would we need a Imagenet-C-bar-bar to check the goodness for corruptions that may be beyond perceptual similarity (such as stylized transforms)? This is not very convincing.

Overall, I am still on the fence on this work (and retain my original decision at this time). I think the paper does present an interesting insight, but I am not very convinced it has been studied and explored comprehensively enough. I would have ideally preferred to give a borderline (neither positive nor negative) decision, and will not be disappointed if the work is accepted, considering the interesting insights it offers. 
review_title: Review of AnonReviewer2

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes ImageNet-\bar{C} which uses a smaller number of carefully chosen corruptions, compared to ImageNet-C. The authors try to argue that previous work is overfitting to ImageNet-C. They claim "overfitting indeed occurs." Additionally, they propose "Minimum Sample Distance," showing that they can predict generalization performance using feature embedding distances.

I don't think they marshaled substantial, far-reaching evidence that "overfitting indeed occurs." They rendered numerous corruptions from Huxtable, 2006; Gladman, 2016 (which I very much appreciate) and adversarially chose the worst corruptions. As a result, the best models performed worse by a few percent compared to ImageNet-C.
_A drop is inevitable and expected given the adversarial selection. If the drop was consistently more than, say, 20%, then there might be strong evidence of overfitting._ Since the degradation for some techniques is small, this strikes me as evidence that by-and-large the community isn't overfitting. (Note it was evident to all that Patch Gaussian and ANT were specialized to noise corruptions.) "Is Robustness Robust?" It seems like the answer is "yes" but the authors are trying to argue that the answer is "no."

This paper is fairly similar to _Increasing the Coverage and Balance of Robustness Benchmarks by Using Non-Overlapping Corruptions_ (submission #1101). If these papers have very different ratings, then there's a problem with this review process.

"AugMix, which introduces a broad array of both geometric and color augmentations"
No it doesn't. It removes several color augmentations from AutoAugment and doesn't introduce any augmentation primitive beyond elementwise convex combinations.

"On other corruptions, the increase in error is even worse than the mean would suggest, and even broad augmentations like AugMix...."
AutoAugment is more broad; AugMix is narrower.

"Second, AutoAugment is the only tested augmentation scheme that was designed before the release of ImageNet-C"
AugMix uses a proper subset of AutoAugment's augmentations. It's not as though AugMix added in distortions to fit ImageNet-C; it removes augmentations because AA fits some of ImageNet-C's corruptions directly. This observation makes their overfitting case hard to maintain. 

Also, if AutoAugment's full augmentation list is fair game for ImageNet-\bar{C}, then the authors should train AugMix with the full set of augmentations and make that comparison; the generalization discrepancy would likely be even smaller were the augmentation sets made equal.

From the "Corruption robustness as a secondary learning task" section:
"To mitigate overfitting, standard machine learning practice would dictate a training/validation/test set split; it is only the size and breadth of modern vision datasets that has allowed this to be neglected in certain cases recently."
"having a held-out test set that is not used during model development seems necessary."
"ImageNet-C has only 15 corruption types"
"ImageNet-C could serve as a validation set and ImageNet-\bar{C} could serve as a held-out test set"
Essentially, since the community lacks a validation set, ImageNet-C should become the validation set, and ImageNet-\bar{C} should become the test set.
This section might be negligent or worse for not acknowledging the already existent ImageNet-C validation set. ImageNet-C provides a validation set with about half the corruptions of ImageNet-\bar{C}. There are 19 available ImageNet-C corruptions, so the community already has a validation set.

The authors should cite or compare to other works that use feature distances to predict generalization. An example is "The Frechet Distance of training and test distribution predicts the generalization gap."

Update: "Noisy Student and Assemble-ResNet use without removing overlapping augmentations, yet they test on ImageNet-C." This is a bad practice and I should hope the authors of this paper only have experiments where there is a train-test mismatch (otherwise we're not testing robustness to distribution shift).
review_title: "Is Robustness Robust?" It would appear so.

Meta-review: The paper investigates the relationship between data augmentations used during training and their effect on the accuracy when evaluated on unseen corruptions at test-time. The paper proposes a metric called minimal sample distance (MSD) to measure the similarity between augmentations during training time and corruptions at test time.

The reviewers agree that the paper aims to solve an important problem and the paper has some interesting findings. However, the current version has a few shortcomings:
- Some of the claims about “overfitting” are confusing, especially for data augmentations that use ops similar to those in ImageNet-C. This is already known and which is why some papers uses a subset of operations (e.g. AugMix uses a subset of AutoAugment operations).
- The main take-home message and novelty is unclear: The initial version titled (“Is Robustness Robust?“) seemed to argue that we may be overfitting to Imagenet-C, but the rebuttal and the updated version revised some of the claims (see response to R3 and R4).  In light of the revision, I’m not sure how the main take-home messages differ from existing papers such as Yin et al. 2019 or “Many faces of robustness”. 
- One of the main differences is quantification of the distribution similarity, however, as pointed out by R2, this analysis does not explain when stylized corruptions would help, so the current version of the paper feels a bit incomplete to me.

I recommend the authors to revise the draft based on reviewer feedback and resubmit the paper to another venue.

Decision: Reject