On interaction between augmentations and corruptions in natural corruption robustness
Authors: Eric Mintun
Alexander Kirillov
Saining Xie
Venue:  (2021)
Keywords: corruption robustness
data augmentation
perceptual similarity
deep learning

Abstract: Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed that significantly improve performance on ImageNet-C, a benchmark of such corruptions. However, there is still a lack of basic understanding on the relationship between data augmentations and test-time corruptions. To this end, we develop a feature space for image transforms, and then use a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate there is a strong correlation between similarity and performance. We then investigate recent data augmentations and observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C in this feature space. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may risk overfitting to the existing benchmark.  We hope our results and tools will allow for more robust progress towards improving robustness to image corruptions.

Under review as a conference paper at ICLR 2021
ON INTERACTION BETWEEN AUGMENTATIONS AND CORRUPTIONS IN NATURAL CORRUPTION ROBUSTNESS
Anonymous authors Paper under double-bling review
ABSTRACT
Invariance to broad array of image corruptions, such as warping, noise, color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed that significantly improve performance on ImageNet-C, a benchmark of such corruptions. However, there is still a lack of basic understanding on the relationship between data augmen- tations and test-time corruptions. To this end, we develop a feature space for image transforms, and then corruptions called the Minimal Sample Distance to demonstrate there a strong correlation between similarity and performance. We then investigate recent data augmentations and observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet- C in this feature space. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may risk overfitting to the existing benchmark. We hope our results and tools will allow for more robust progress towards improving robustness image corruptions.
new measure in this space between augmentations and
INTRODUCTION
Robustness distribution shift, i.e. when the train and test distributions differ, is an important feature of practical machine learning models. Among many forms of distribution shift, one particularly relevant category for computer vision are image corruptions. For example, test data may come from sources that differ from the training set in terms of lighting, camera quality, other features. Post- processing transforms, such as photo touch-up, image filters, or compression effects are commonplace in real-world data. Models developed using clean, undistorted inputs typically perform dramatically worse when confronted with these sorts of image corruptions (Hendrycks Dictterich) 2018 Geirhos ct al. 2018). The subject of corruption robustness has long history in computer vision (Simard et al.
1998 Bruna & Mallat, 2013 Dodge & Karam 2017) and recently has been studied actively with the release of benchmark datasets such as ImageNNet-C (Hendrycks & Dietterich 2018).
One particular property of image corruptions is that they are low-level distortions in nature. Corrup- tions transformations of an image that affect structural information such as colors, textures, geometry (Ding et al. (2200) and are typically free of high-level semantics. Therefore, is natural expect that data augmentation techniques, which expand the training set with random low-level transformations, can help with learning robust models. Indeed, data augmentation has become
central technique in several recent methods (Hendrycks al., 2019 Lopes et 2019 Rusak et al. 2020) that achieve large improvements on ImageNet-C and related benchmarks.
One caveat for data augmentation based approaches is the test corruptions are expected be unknown at training time. If the corruptions are known, they may simply be applied to the training set as data augmentations to trivially adapt to the test distribution. Instead, an ideal robust model needs to be robust to any valid corruption, including ones unseen in any previous benchmark. Of course, in practice the robustness of a model can only be evaluated approximately by measuring its corruption error representative corruption benchmark. To avoid trivial adaptation to the benchmark, recent works manually exclude test corruptions from the training augmentations. However, with a toy experiment presented in Figure [[ we argue that this strategy alone might not be enough and that visually similar augmentation outputs and test corruptions can lead to sigmificant benchmark improvements even if the exact corruption transformations excluded.

---

Under review as a conference paper at ICLR 2021
30
top-1 error
20
test set corrupted by motion blur
10
0-
train time augmentation:
clean
motion blur
translate
shear
rotate
posterize
equalize
solarize
Figure 1: A toy experiment. We train multiple models on CIFAR-10 Krizhevsky et al. 2009) using different augmentation schemes. Each scheme is based single basic image transformation type and enhanced by overlaying random instantiations of the transformation for each input image following Hendrycks et al. (2019) compare these models on the CIFAR-I0 10 test set corrupted by motion blur, corruption used in the ImageNet-C corruption benchmark Hendrycks & Dietterich (018). None of the augmentation schemes contains motion blur; however, the models trained with geometric- based augmentations significantly outperform the bascline model trained on the clean images while color-basee augmentations show gains. We note the geometric augmentations can produce result visually similar blur by overlaying copics of shifted images
This observation raises two important questions. One, how exactly does the similarity between train time augmentations and corruptions of the lest affect the error? And two, if the gains due to the similarity, the improvements may not translate into better robustness to other possible corruptions, so do we ever risk overfitting existing corruption benchmarks using a new augmentation scheme? In this work, we take a step towards answering these questions, with the goal of better understanding the relationship between data augmentation and test-time corruptions. Using a feature space on image transforms and a new measure called Minimal Sample Distance (MSD) on this space, we are able to quantify the distance between augmentation schemes and classes corruption transformation. With our approach, we empirically show an intuitive yet surprisingly overlooked finding:
Augmenttation
perceptual similarity is strong predictor of corruption error.
Based on this finding, we perform additional experiments to show that data augmentation aids corruption robustness by increasing perceptual similarity between (possibly small) fraction of the training data and the test set. To further support our claims, we introduce a set of new corrup- tion, called CIFAR/ImageNettCC overfit original the CIFAR/ImageNet-CC To choose these corruptions. we expand the set of natural corruptions and sample new corruptions that far away from CIFAR/ImageNet-C in our feature space for measuring perceptual similarity. We then demonstrate that augmentation schemes designed specifically to improve robustness show significantly degraded performance on CIFAR/ImageNett Some augmentation schemes still show some improvement over baseline, which suggests meaningful progress towards general corruption robustness being made, but different augmentation schemes exhibit different degrees of generalization capability. As an implication, caution is needed for fair robustness evaluations when additional data augmentation is introduced.
test the degree to which common data augmentation methods
These results suggest major challenge that is often overlooked in the study of corruption robustness: overfitting indeed occurs. Since perceptual similarity can predict performance, for any fixed finite set of test corruptions, improvements that set may generalize poorly to dissimilar corruptions. However, perceptual similarity is not expected to be the only interaction between augmentations and corruptions, so proposed augmentation scheme's degree of generalization capability may not be immediately clear. We hope that our results, together with new tools and benchmarks, will help researchers better understand why given augmentation scheme has good corruption error and whether it should be expected to generalize to dissimilar corruptions. On the positive side, our experiments show that generalization does emerge within perceptually similar classes of transform, and that only small fraction of sampled augmentations need to be similar to a given corruption. Section discusses these points in more depth.
Example transforms for illustrative purpose only and exaggerated. Base image c Schee Park.
2

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
Corruption robustness benchmarks and analysis. ImageNet-C (Hendrycks & Dietterich] 2018 is corruption dataset often used as a benchmark in robustness studies. Other corruption datasets (Hendrycks et al. 2020, Shankar et al. 2019) collect corrupted images from real world sources and thus have a mixture of semantic distribution shifts and perceptual transforms. Corruption robustness differs from adversarial robustness (Szegedy et al., 2014), which seeks invariance to small, worst case distortions. One notable difference that improving corruption robustness often slightly improves regular test error, instead of harming it. Yin et al. (2019) analyzes corrpption robustness in the context of transforms' frequency spectra; this can also influence corruption error independently from perceptual similarity. Dao et al. (2019); Wu et al. (2020) study the theory of data augmentation for regular test error. Hendrycks et al. (2020) : Taori et al.] (2020) study how the performance on synthetic comuptiin transforms generalizes to performance on corruption datasets collected from the real world. Here we do not address this issue directly but touch upon it in the discussion.
Improving corruption robustness. Data augmentations designed to improve robustness include AugMix (Hendrycks et al., 2019), which composites common image transforms, Patch Gaussian (Lopes al.] 2019), which applies Gaussian noise in square patches, and ANT (Rusak et al. 2020).
which augments with an adversarially learned noise distribution. AutoAuggentt (Cubuk et al. 2019)
learns augmentation policies that optimize clean error but has since been shown to improve corruption
error (Yin et al.) 2019). Mixup (Zhang et al. (a) can improve robustness (Lee et al. 2020),
but its label augmentation complicates the dependence on image augmentation. Stylized- -mageeeet (Geirhos et al.) (219)) can improve robustness, but it is at the cost of clean test error without additional fine-tuning, Noisy Student (Xie al. (220)) and Assemble-ResNet (Lee et al.) 2020) combine data
augmentation with new models and training procedures and greatly enhance corruption robustness.
PERCEPTUAL SIMILARITY FOR AUGMENTATIONS AND CORRUPTIONS
First, we study the importance of similarity between augmentations and corruptions for improving performance on those corruptions. To do so, we need a means to compare augmentations and corruptions. Both types of transforms are perceptual in nature, meaning they affect low-level image structure while leaving high-level semantic information intact, so we expect good distance to be measure of perceptual similarity. Then, we need to find the appropriate measure of distance between the augmentation and corruption distributions. We will argue below that distributional equivalence is not appropriate the context corruption robustness, and instead introduce the minimal sample distance, simple measure that does capture relevant sense of distribution distance.
Measuring similarity between perceptual transforms. We define a perceptual transform as a transform that acts on low-level image structure but not high-level semantic information. As such, we expect two transforms should be similar if their actions on this low-level structure are similar, independent of algorithmic or per-pixel differences between them. A closely related, well-studied problem is the perceptual similarity between images. A common approach is to train a neural network on a classification task and use intermediate layers feature space for measuring distances (Zhang eet al. 2018b). Here we adapt this idea to instead obtain a feature space for measuring distances between perceptual transforms.
We start with a feature extractor for images, which we call f(x).. To train the model from which we will extract features, we assume access to dataset ID of image label pairs (I, y) associated with classification task. The model should be trained using only default data augmentation for the task in question so that the feature extractor is independent of the transforms we will use it to study. In order to obtain a very simple measure, we will use just the last hidden layer of the network as a feature space.
A perceptual transform t(x) may be encoded by applying to all images in D, encoding the transformed images, and averaging the features over these images. For efficiency, we find it sufficient to average over only a randomly sampled subset of images Ds in D. We show in Appendix C produces stable estimates for reasonable numbers of images. The random choice of images is property of the feature extractor, and so remains fixed when encoding multiple transforms. This reduces variance when computing distances between transforms. The transform feature extractor
3

---

Under review as a conference paper at ICLR 2021
MMD
Distannce to impulee Naise
2.0
60
MSD
Impulse Noise Error
A
daNDD
MSD
0.6
AM1D
Corrupton
Corruption
Augmentation
0,6
0.0
1.0
04 Impulse
Augreniaation sample
Closest sample
Impulse Noise
(a) Schematic comparison of MMD to MSD. MMD measure between distribution centers and is only small if the augmentation overlaps with corruption. MSD measures to the nearest sampled point in set of samples (marked by star), and small even for broad distributions that overlap with multiple corruptions. (b) We test on images corruption with impulse noise, and train images augmented with mixture of impulse noise and motion blur. As the mixing fraction of impulse noise decreases, MMD between the augmentation and corruption grows linearly while MSD and error stay low until nearly 0% mixing fraction.
Figure 2:
is given by S(() Enen, ( ([([))))) The perceptual similarity between augmentation and corruption can taken as the Lg distance on this feature space f.
Minimal sample distance. We now seek to compare the distribution of augmentation scheme Pa. to distribution of a corruption benchmark Pc. A simple first guess would be to measure how close to equivalent the distributions are. Indeed, if the goal was optimize error known corruption distribution, exact equivalence of distributions is the correct measure to minimize. But since the goal is robustness to general, unknown corruption distributions, good augmentation scheme should be equivalent to no single corruption distribution.
To illustrate this behavior, consider toy problem where have access to the corruption transforms at training time. A very rough, necessary but insufficient measure of distributional similarity is dMMDD (Pa, Pe) = [|EEa-p [f()] Ee- [c)]ll This is the maximal mean discrepancy on a fixed, finite feature space, so for brevity we will refer to it as MMD. We still employ the fcaturization ft), since we are comparing transforms and not images, unlike in typical domain adaptation. Consider two corruption distributions, here impulse noise and motion blur, and an augmentation scheme that is mixture of the two corruption distributions. Figure shows that MMD between the augmentation augmentti and impulse noise corruption scales linearly with mixing fraction, but enror on impulse noise remains low until the mixing fraction almost 0% impulse noise. This implies distributional similarity is a poor predictor corruption error. Indeed, in the context of corruption robustness, low dMMD with any one corruption distribution likely bad thing: it suggests the augmentation scheme overfits that one other, dissimilar corruption distributions.
corruption distribution expense of performance
Our expectation for the behavior of the error in Figure 2b that networks can often successfully memorize examples seen during training, so that only very small fraction of sampled images need impulse noise augmentations to perform well on impulse noise corruptions. An appropriate distance should then measure how close augmentation samples can come the corruption distribution, even if the density of those samples is low. We thus propose a very simple measure called minimal sample distance (MSD), which is just the perceptual similarity between average corruption and the closest augmentation from a finite set of samples A Pa:
dMsD(pas (c))
min (1[(a) (e-pe(((c)))
aEk<pa
schematic comparison of MMD and MSD shown in Figure 2h. While both MMD and MSD are small for an augmentation scheme that is distributionally similar to corruption distribution, only MSD remains small for a broad distribution that occasionally produces samples near multiple corruption distributions. Figure 2b shows MSD, like test error, remains small for most mixing fractions in the toy problem described above. Note that the need for our measure to accommodate robustness to general, unknown corruption distributions has led it to be asymmetric, it differs from more formal distance metrics that may be used to predict generalization error, such as the Wasserstein distance (Zilly et al. 2019).
4

---

Under review as a conference paper at ICLR 2021
4 PERCEPTUAL SIMILARITY IS PREDICTIVE OF CORRUPTION ERROR
We are now equipped to measure how important this augmentation-coruptoon similarity is for corruption error. For a large number of augmentation schemes, we will measure both the MSD to corruption distribution and the corruption error of model trained with that scheme. We will find a correlation between MSD and corruption error, which provides evidence that networks can successfully generalize across perceptually similar transforms. Then, we will calculate the MSD for augmentation schemes the literature that have been shown to improve error on corruption benchmarks. We will find correlation between MSD and error here well, which suggests the success of these models is in part explained by their perceptual similarity to the benchmark. This implies there may be risk that the augmentation schemes overfit the benchmark, since we would not expect this improvement to transfer to a dissimilar corruption.
4.1 EXPERIMENTAL SETUP
Corruptions. We use CIFAR-IO-C (Hendrycks & Dictterich, 2018), which a common benchmark used for studying corruption robustness. It consists of 15 corruptions, each further split into five different severities of transformation, applied to the CIFAR- 10 test set. The 15 corruptions fall into four categories: per-pixel noise, blurring, synthetic weather effects, and digital transforms. We treat each corruption at each severity as a separate distribution for the sake of calculating MSD and error; however, for simplicity average errors and distances over severity to present a single result per corruption. Examples of each corruption are shown in Figure 13 in Appendix
Space of augmentation schemes. To build each sampled augmentation transform, we will composite a set of base augmentations. For base augmentations, we consider the nine common image transforms used in Hendrycks et al. (2019), shown Figure T2 of Appendix E There are five geometric transforms and four color transforms. By taking all subsets of these base augmentations, we obtain 2 512 unique augmentation schemes, collectively called the augmentation powerset. Also following Hendrycks et al. (2019), we composite transforms in two ways: by applying one after another, or by applying them to copies of the image and then linearly superimposing the results.
Computing similarity and corruption error. A WideResNer-40-2 ZZgoruykoo & Komodakis 2016) model is pre-trained on CIFAR- 10 using default augmentation and training parameters from Hendrycks et al. (2019). WideResNet is a common baseline model used when studying data aug- mentation on CIFAR- 10 (Hendrycks et al. 2019 Cubuk et al., 2019 Zhang et al., 2018a). Its last hidden layer is used as the feature space. For MSD, average over 100 images. 100 corruptions, and minimize over 100k augmentations. We argue in Appendix C that these are reasonable choices. Images from the training and do not have default training augmentation. For corruption error evaluation, we also use a WideResNer-40022 and the same training parameters.
4.2 ANALYSIS
MSD correlates with corruption error. First, we establish the correlation between MSD and corruption error on the augmentation powerset. Figure 3 shows the relationship between distance and corruption error for both MSD on four example corruption distributions. MSD shows strong correlation with corruption error across corruptions types all four categories of CIFAR-IO-C. In Figure 14 in Appendix we compare to MMD and confirm that MMD correlates poorly with corruption error, as expected. In particular, our expectation that broad augmentation schemes with many base transforms produce samples similar to larger set of corruptions. even if those samples occur less frequently. This leads to both lower MSD and lower corruption error but higher MMD. Additionally, the correlation between MSD and corruption error suggests that perceptual similarity a predictor of corruption error. However, as shown Table most but not all corruptions show strong correlation between MSD and error: 12 of 15 have Spearman rank correlation greater than 0.6. complete set of correlation plots is shown in Figure 15 in Appendix F
An example of perceptual similarity. Here we briefly illustrate the perceptual nature of the similar- ity measure, using an example with two base augmentations and two corruptions. The augmentation solarize and the corruption impulse noise both insert brightly colored pixels into the image, though in different ways. The augmentation x translation and the corruption motion blur are both geometric transforms, and linear superpositions of translation visually similar blurring. Examples of
5

---

Under review as a conference paper at ICLR 2021
Gaussian Noise, p /= 0.76
Zoom Blur, p= 0.77
Elastic Transform, p 0.77
Frost, / = 0.66
24
Corruption Error
50
25
18
20-
40-
14-
30-1
20
12
1.0
1.5
2.0
2.5
3.0
0.4
0.8
0.2
0.4
0.6
0.8
0.4
0.8
1.2
1.6
Minimal Sample Distance
Figure 3: Example relationships between MSD and corruption error for different corruptions, p is the Spearman rank correlation. MSD correlates well with error across all four categories of corruption in CIFAR-10-C.
Table I:
Spearman's 's rank coefficient for the correlation between MSD and coruptton eroor Correlations high for most corruptions, including those in all four categories in ImageNet-C. However, brightness, biaggttess.piilattee. and JPEG compression show poor correlation.
Corruption
Corruption
p
Corruption
p
Gaussian Noise
0.76
Zoom Blur
0.77
Snow
0.65
Shot Noise
0.83
Glass Blur
0.69
Contrast
0.66
Impulse Noise
0.90
Brightness
0.27
Pixelate
0.35
Motion Blur
0.86
Fog
0.68
JPEG Compression
0.33
Defocus Blur
0.83
Frost
0.66
Elastic Transform
0.77
Figure shows MSD vs error where augmentation schemes that include solarize and x translation are colored. It clear that including augmentation greatly decreases MSD to its perceptually similar corruption, while having little effect on MSD to its perceptually dissimilar corruption.
these transforms are shown in Figure 12 and Figure 13 in Appendix E
MSD and corruption error in real augmentation methods. The augmentation powerset may be used as a baseline for comparing real data augmentation schemes. Figure3]showe example MSD-error correlations for Patch Gaussian, AutoAugment, and Augmix, along with the cloud of augmentation powerset points. The real augmentation schemes follow the same general trend that lower error predicts lower MSD. A few intuitive correlations are also captured in Figure Patch Gaussian has low MSD to corruptions with noise corruptions and to glass blur which introduces random pexel-level permutations as noise, and may be distributionally similar the noise corruptions, as we argue in Appendix A AutoAugment, which contains contrast and Gaussian blurring augmentations in its sub-policies, has low MSD with contrast and defocus blur. AugMix, which introduces fewer base augmentations that Auto Augment but composites them in ways that may produce perceptually new effects (such as blurring from superimposing translations), has low MSD with all corruptions.
The fact that improved corruption error typically implies greater similarity suggests overfitting may be an issue the study of corruption robustness. For two augmentation schemes compared on
Impulse Noise
Motion Blur
wio solarize wi solerize
wio transiation w/ x-transiation
wio solarize solarize
velo transiationn w/ x-transiation
Corruption Error
Corruption Error
25
20
15-
10
1.0
1.5
1.5
2.0
0.2
1.0
0.8 1.4
Minimum Sample Distance
Minimum Sample Distance
Figure 4: Example relationships between base augmentations and corruptions. Including 'solarize" reduces MSD on the perceptually similar impulse noise corruption, while including translation reduces MSD the perceptually similar motion blur corruption. MSD is not decreased for dissimilar petpeenttationccorpptti pairs.

---

Under review as a conference paper at ICLR 2021
Shot p 0.83
Defocus Blur, o 0.83
Contrast, = 0.66
Glass Blur, = 0.69
Corruption Test Error
25
40
20
30
45-
16
15
20
35-
10
10
25
0.5
1.0
1.5
2.0
2.5
0.2
0.4
0.8
0.8
1.0
0.4
0.8
0.8
1.0
12
1.0
1.5
2.0
Minimum Sample Distance
Baseline
# AutoAugment
# Patch Gaussian
Augmix"
Augmentation powerset
Figure 5: Example correlations for augmentation schemes from the literature. Patch Gaussian is similar to noise, while AutoAugment is similar to contrast and blur, as expected from their formulation. Glass blur acts like noise comuption than a blur for these augmentation schemes. likely because it randomly permuting pixels. AugMix here refers to just augmentation distribution in Hendrycks al. (2019), not the proposed Jensen-Shane ShasnnShannon divergence loss, which gives additional improvements in corruption error.
fixed corruption benchmark, the one that is more perceptually similar to the corruptions may perform better, but also generalize to dissimilar corruptions worse. This perceptual similarity may be exact and explicit, such as between Augment, which contains contrast augmentations, and the contrast corruption. But it may also be naively unexpected, such as the similarity between Patch Gaussian and glass blur (since glass blur shares similarity with the noise corruptions) or between AugMix blur-like). Note, however, that perceptual similarity
and blurs (since superimposed translations is not expected to be the only interaction between augmentations and corruptions (for instance, the frequency dependence of et al. (2019) is another), and other interactions may or may not lead to good generalization. Thus, tools such as MSD help us determine why an augmentation scheme improves corruption error, we can better analyze and understand if newly proposed methods will generalize beyond their tested benchmarks. In the next section, we test this generalization directly by finding corruptions that are dissimilar to ImageNet-C.
5 IMAGENET-C: BENCHMARKING WITH DISSIMILAR CORRUPTIONS
We now introduce set of corruptions, called ImageNet-C, that are perceptually dissimilar to ImageNet-C: our transform feature space and will show that several augmentation schemes have degraded performance on the new dataset, suggesting that they have indeed overfit ImageNet-C: to at least some degree. We emphasize that the dataset selection method does not involve any augmentation scheme beyond the default one used to train the feature extractor and was fixed before we looked at the results for different augmentations, so there should be no inadvertent adversarial selection against the augmentation schemes.
Dataset construction. An overview of the dataset construction is presented here, with specific details described in Appendix D.I. We construct set of 30 corruptions types in 10 severities, from which the 10 most dissimilar corruptions types will be chosen. These corruptions are adapted from common filters and noise distributions available online (Huxtable) 2006 Gladman 2016) and are designed to produce human interpretable images. The transforms include warps. blurs, color distortions, noise additions, and obscuring effects. Examples shown in Appendix
To assure that the new dataset is no harder than ImageNet-C, we restrict the average corruption error of the new dataset be similar that of ImageNet-C for default augmentation. We then generate many potential datasets and measure the average shift in distance to ImageNet-C that each corruption contributes, shown in Figure 16 Appendix Note that while MSD was used establish correlation between perceptual similarity and error for augmentations and corruptions, here we are comparing corruptions to other corruptions and thus use MMD as the measure of distance in our transform feature space. Some generally intuitive results are recovered. First, scatter and blue noise, which are conceptually similar to ImageNet-C", glass blur and gaussian noise corruptions, consistently lead to more similarity with ImageNet-C. Second, lower- frer-freyynncy noise distortions tend to be included in more dissimilar datasets, possibly in contrast to ImageNet-C's high frequency pixel noise. However, blue noise sample, which shares some visual similarities with impulse noise,
7

---

Under review as a conference paper at ICLR 2021
GIARSOO- Comuptoon
Imaggshart Compttani
Soortio
Brown Nobe
Checkrteend
Cocentlc Morzz
Nertn
Precuenc Neve
Rui Nike Sonelh
Hwata Spordd
Prod and Tuet
Piyce
Sie Towe Canpo
Cana Fatedan
Enstaa Scats
Nore
Cartley
Figure Example CIFAR-IO-C ImageNet-C corruptions. While human interpretable, new corruptions are sampled to dissimilar from CIFAR-10/lmagNee.CC Base images ( Sehee Park and Chenxu Han.
also leads to larger distances. ImageNet-C then consists of the 10 corruptions types with the largest average shift distance. Like ImageNet-C, each included in five different severities, with severities chosen so that the average error matches ImageNet-C for default augmentation. Example transforms from ImageNer-C and CIFAR-1O-C are shown in Figure
Results. We test Augment, Patch Gaussian, AugMix, and ANT on our new datasets and show results in Table 2 CIFAR-10 models are W0deResNet-40-2 with training parameters from Hendrycks al. (2019), ImageNet (Deng et al. 2009) models are ResNet-50 (He al. (016) with training parameters from Goyal et al. (0117) Models use default data augmentation as well as the augmentation being tested, except ImageNet color jittering is not used. All corruptions are applied in-memory, instead of loaded from a compressed file; this can affect results especially on high frequency corruptions. Given the evidence of overfitting from Section 4 expect these methods to have worse error on the new corruptions. Indeed, every augmentation scheme performs worse, even when baseline improves slightly.
Additionally, there are few other suggestive patterns. First, intuitively broader augmentation schemes perform better: AugMix and AutoAugment degrade less than Patch Gaussian or ANT, Patch Gaussian in particular, which has been identified as perceptually similar to pixel-noise type corruptions alone, sees particularly large drop in performance. Second, Augment is the only tested augmentation scheme that was designed before the release of ImageNet-C, and it has the smallest performance drop, despite known overlaps with the brightness, contrast, and defocus blur corruptions in ImageNet-C. This suggests that having ImageNet-C on hand may make it difficult to avoid overfitting to. Finally, it is possible that blue noise sample a failure mode of our distance: both shares visual features with impulse noise and most data augmentations have low error it. On other corruptions, the increase in error is even worse than the mean would suggest, and even broad augmentations like AugMix
better than baseline several individual corruptions.
Despite this degradation, note that the data augmentation schemes, especially broad ones such AugMix, still show meaningful improvement on ImageNet-C over baseline. What we want to emphasize is that care must be taken when comparing methods, since the degradation differs dramatically for different augmentation schemes. Moreover, it does appear exact augmentation- corruption overlap is especially predictive of poor generalization: AutoAugment includes exact overlap with some ImageNet-C corruptions but loses the least eгтог compared to ImageNet-C. This suggests it may not be sufficient to simply remove such augmentations from an augmentation scheme when testing on a corruption benchmark. Instead, interaction between an augmentation scheme and the tested corruptions. We hope that MSD and ImageNet-C may be an additional tools to help researchers understand why a robust augmentation scheme works and whether it may be expected to generalize to unknown corruptions.
important understand more in-depth the
DISCUSSION
Corruption robustness secondary learning task. We have provided evidence that data augmen- tation can overfit a corruption benchmark. To explore this further, consider an analogy to a regular learning problem. We may think of corruption robustness in the presence of data augmentation sort secondary task layered on the primary classification task: the set of data augmentations the training set, the set of corruptions is the test set, and the goal is to achieve invariance of the underlying
8

---

Under review as a conference paper at ICLR 2021
Table Test error for several data augmentation methods on CIFAR-IO-C ImageNet-10-C. Every method performs worse on these new datasets than on ImageNet-C.oo or CIFAR- Example corruptions and descriptions of the abbreviations are given in Appendix [E] and standard deviations for individual corruption are given in Table Appendix D.2 'Baseline' refers default augmentation only. ANT uses the single pre-trained model provided with the paper and has slightly different training parameters.
IN-C Err
IN-C
ImageNet-C Corruptions
Aug
Err
AIN-C
BSmpl
Pism
Ckbd
CSin
SFrq
Brwn
Prin
1Sprk
Sprk
Rfrac
Baseline
58.2+05 57.7
-0.5
68.6
71.7
49.4
84.7
79.0
37,5
34.3
32.4
76.7
42.8
AA
54.8
55.7
+0.9
54.8
68.3
43.8
86.5
78.8
34.5
33.8
36.1
77.1
43.8
48.1 102 56.6204
+8.5
60.3
74.1
48.5
82.1
76.7
38.9
34.6
32.1
76.5
42.1
A
48.8
53.9
+5.1
35.8
75.5
56.9
76.4
63.7
41.0
35.2
35.0
76.1
43.3
AugMix
49.1 +0. 52.4
+3.3
43.2
72.2
46.1
76.3
67.4
38.8
32.4
32.3
76.4
39.2
C10-C
C10-C
CIFAR-IO-C Corruptions
Aug
Er
Err
AC10-C
BSmpl Brwn
Ckhd
CBlur 1Sprk
Line
P&T
Rppl
Sprk
TCA
Baseline
27.0+06 27.1 +0.5
+0.1
42.9
27.2
23.3
11.8
43.3
26.2
11.3
21.6
21.0
42,9
AA
19,4
21.0=04
+1.6
17.7
17.5
17.6
9.5
40.4
23.6
10.7
23.5
17.5
31.8
PG
17.0403 23.8
+6.8
9.0
30.1
21.6
12.8
35.4
20.6
8.8
21.5
19.3
59.5
AugMix
11.1 402 =6.0=03
+5.9
9.8
27.8
13.4
5.9
30.3
18.0
8.3
12.1
15.5
19.2
primary task. In this language, the 'datasets' involved are quite small: ImageNet-C has only 15 comuptiio types, and several augmentation schemes composite only around 10 basic transforms. To mitigate overfitting, standard machine learning practice would dictate a training/validationnn validation/test set split; it is only the size and breadth of modern vision datasets that has allowed this to be neglected in certain cases recently. But the effective dataset size of a corruption robustness problem is tiny, so having held-out test that not used during model development seems necessary. To emphasize, this is not a test set of the underlying classification task, for which generalization has been studied in Recht et al. 2018 2019). Instead, it is test set of corruption transforms themselves. This means there would be two sets of dissimilar transformations, both applied to the ImageNet validation set, that would act as validation/test split on transform
Real-world corruption robustness. Recently, Hendrycks et al. (2020) and Taori et al. (2220) study how performance on corruption transforms generalizes to real-world corruptions and come to con- flicting conclusions. Though we do not study real-world corruptions directly, we have proposed a mechanism that may explain the conflict: performance will generalize between transforms and real-world corruptions if they are perceptually similar, but will likely not if they are dissimilar. Since Hendrycks et al. (200) and Taori et al. (2020) draw on different real-world and synthetic corruptions, may simply be that the perceptual similarity between datasets differs in the two analyses. This also suggests a way to find additional corruption transforms that correlate with real-world corruptions: transforms should be sought that have maximal perceptual similarity with real-world corruptions.
Generalization does occur. Finally, let us end on a positive note. Through our study of overfitting, have encountered two features of data augmentation that may explain why it can be such powerful tool for corruption robustness. First, within class of perceptually similar transforms, generalization does occur. This means single, simple data augmentation may confer robustness to many, much more complicated corruptions, as long as they share perceptual similarity. Second, the presence of dissimilar augmentations an augmenation scheme often causes little to no loss in performance, as long Appendix by demonstrating that adding many dissimilar augmentations increases error much less than adding few similar augmentations decreases it. Together, these features suggest broad augmentation schemes with many dissimilar augmentations may be capable of conferring robustness to large class of unknown corruptions. More generally, we think data augmentation is promising direction of study for corruption robustness, as long as significant care is taken to avoid overfitting.
similar augmentation is also present. We study this in bit more detail
Thhe validation set provided in Hendryeks & Dietterich 2018) consists of perceptually similar transforms to ImageNet-C and would not he expected to work well for the validation discussed here.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-1888 1886, 2013.
Ekin D Cubuk, Barret Zoph. Dandelion Mané, Vijay Vasudevan, and Quoc V Le. AutoAugment: Learning augmentation strategies from data. In CVPR, 2019.
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Christopher De Sa, and Christopher Ré. A kernel theory of modern data augmentation. Proceedings of machine learning research, 97:1528, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. arXiv preprint arXiv:2004.07728 2020.
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recognition performance under visual distortions. In ICCCN, 2017.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In NeurIPS, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Simon Gladman. F
Filterpedia, 2016.
URL
Filterpedia
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet hour. arXiv preprint arXiv:1706.0267. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICIRR, 2018.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: simple data processing method to improve robustness and uncertainty. In ICL.R 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of duttof-distribution generalization. arXiv preprint arXiv:2006. / 2020.
Jerry Huxtable. JH Labs Java Image Processing, 2006. URL BoLhttp:/// http:://www... Filters/
com. /pp
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Jungkyu Lee, Taeryun Won, and Kiho Hong. Compounding the performance improvements of assembled techniques convolutional neural network. arXiv preprint arXiv:2001.0623 006668, 2020.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with Patch Gaussian augmentation. arXiv preprint /oXiv:/996.02//1 2019.
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár, On network design spaces for visual recognition. In ICCV, 2019.
10

---

Under review as a conference paper at ICLR 2021
Iijaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprint arXiv:/806.004: 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In ICML, 2019.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. arXiv preprint arXiv:2001.00057 2020.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? arXiv preprint arXiv:1906.02168, 2019.
Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition- tangent distance and tangent propagation. In Neurul networks: tricks of the Irade, pp. 239-2744 Springer, 1998.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. arXiv preprint 1rXiv:2007.00644 2020.
Sen Wu, Hongyang R Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. In ICML, 2020.
Qizhe Xic. Minh-Thang Luong, Eduard Hovy, and Quoc Le. Self-training with Noisy Student improves imagenet classification. In CVPR, 2020.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A Fourier perspective on model robustness in computer vision. In NeurIPS, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018b.
Julian Zilly, Hannes Zilly, Oliver Richter, Roger Wattenhofer, Andrea Censi, and Emilio Frazzoli. The Frechet Distance of training and test distribution predicts the generalization gap. OpenReview preprint, 2019. URL htt.tttttt//pneeeeeetss ew. net forum? f
d=SJgSf1HKD

---

Under review as a conference paper at ICLR 2021
Impulse Noise
Defocus
Snow
Elastic Transform
Corruption Test Error
20
35
16
18
25
15-
16
14
15
10-
14-
12
2.0
2,2
2.4
2.6
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
MMD
(a)
Contrast
Brightness
Corruption Test Erro
25
7.0
6.5
15
6.0
10
5.5
5.0
1.0
2.0
0.5
1.0
15
2.
MMD
(b)
Baseline AutoAugment
Patch Gaussian
Augmix
Transform Subsets
Figure 7: (a) Patch Gaussian shows low MMD distance on the noise corruptions and high MMD distance every other corruption, suggesting that overfits the noise comuptions. (b) While AutoAugment contains contrast brightness augmentations. broad enough that doesn't have MMD these corruptions. Note that since brighiness shows poor correlation for MSD, is possible that in this case the MMD does not change for other reasons.
A ANALYZING OVERFITTING WITH MMD
Overfitting be analyzed in a bit more detail by using the MMD distance from Section 3 Recall that low MMD may indicative of overfitting a single type of corruption, since it suggests possible equivalence between the augmentation and corruption distributions. Broad augmentation schemes will likely have low MMD with no single corruption distribution, but narrow ones that overfit to a single corruption will have low MMD with it and high MMD for others. Figure 7 shows example MMD-error correlations. For Patch Guassian, MMD is low for the noise corruptions and high for everything else, while Autmmntment and AugMix, which are constructed out of many visually distinct transforms, show no strong correlation. We might then expect greater overfitting from Patch Gaussian.
B SAMPLING SIMILAR AUGMENTATIONS MORE FREQUENTLY GIVES MINOR PERFORMANCE IMPROVEMENTS
Here we describe an alternative experiment that shows how the introduction of dissimilar augmenta- tions affects corruption error. For broad data augmentation scheme that provides robustness many dissimilar corruptions, each corruption may only have similar augmentation sampled some small fraction of the time. This small fraction of samples must be sufficient to yield good performance on each corruption to obtain robustness overall. We expect that this should be the case, since neural networks are often good at memorizing rare examples. Additionally, the toy problem in Figure 2 suggests that large fraction of sampled augmentations may be dissimilar without significant loss corruption error. Here we show the effect using a real augmentation scheme.
We consider performance their Jensen- Shannnnn divergence loss, which gives additional improvements). However, instead of sampling directly from the AugMix distribution during training, first sample 100k transforms and
CIFAR-10-C when training with AugMix augmentations (we do not
12

---

Under review as a conference paper at ICLR 2021
Random
Closest to Corruptions
Farthest from Corruptions
24
Corruption Enor
22
20
18
18
14
12
10
10'
10
10
10"
10 10
10'
10'
Number of Transforms
Figure 8: Average corruption error on ImageNet-C as a function the size of a fixed subset of AugMix augmenta- tions. During training. augmentations only sampled from the subset. The subset is chosen one of three ways: randomly, the similar augmentations to ImageNet-C, the least similar augmentations ImageNet-C. Choosing similar corruptions improves error beyond AugMix, but not by as much that choosing dissimilar augmentations harms it.
sort these transforms by their distance the CIFAR-IO-C corruptions. This sorting is done to evenly distribute the augmentations among the 75 (15 corruptions in 5 severities) individual corruptions; e.g. the first augmentations in the list are the closest augmentation to each corruption. Then we take fixed-size subset A of these transforms and train augmentations sampled only from this subset using the training parameters from Hendrycks et al. (009). We select three different ways: randomly, taking the |A| closest augmentations, and taking the A farthest augmentations. We then measure the average corruption error on CIFAR-1O-C and plot this error against |A| in Figure 8
First, note that for randomly sampled augmentations, A does not need to be very large to match AugMix in performance. Even though training on AugMix with our training parameters would normally would produce 5 million uniquely sampled augmentations, only around 1000 needed to achieve equivalent performance. Training on the closest augmentations exceeds regular AugMix performance with only around 100 unique transforms, which acts as additional evidence that cianeentation-oomppiiin similarity correlates with corruption error. This gain in accuracy comes not from having access better transformations, but from having more frequent access them at training time. However, the gain is fairly mild at only around 1%, even though the best transformations are sampled all of the time instead of rarely. The gain from frequency much less than the gain from having more similar augmentations, since choosing the most dissimilar augmentations gives around 5% drop in accuracy. This suggests that it is net positive decrease the frequency of sampling similar augmentations in order to include augmentations similar another set of corruptions: the gain in accuracy on the new corruption set will likely out weight the small loss in accuracy on the original set.
C MSD ABLATION
C. ARCHITECTURE CHOICE
Here we provide evidence that changing the architecture of the feature extractor used in the definition of MSD does not have any qualitative effect on the correlation with corruption error. We use version of VGG-19 with batch normalization that has been modified for CIFAR- 10. Otherwise, all other parameters are chosen the same. We then repeat the experiment of Section 4 In Table and Figure g we show that the qualitative results of this experiment are unchanged when using VGG- VGG-19-BN as the feature extractor.
C.2 PARAMETER DEPENDENCIES
In calculating the feature space for transforms and MSD, it is necessary to both pick a number of images average over and number of corruptions average In our experiments, we use 100 images and 100 corruptions. Here we provide evidence that these are reasonable choices for these parameters.
13

---

Under review as a conference paper at ICLR 2021
Gaussian Noise
Shot Noise
Impulse Noise
Motion Blur
Defocus Blur
12
15
10
12
1.0
25
365
40
100
200 Gloss Blur
05
15
125
022
01 Fog
12
106 Froet
15
Corruption Test Error
Zoom Blur
Brightnees
24
25
70
GO
1.0 Contrast
G.0
0.25
011 0.5
220
D1
05 JPEG Compression
Snow
Pixelate
Elastic Transtorm
02
04
05
25
10
02
0.4
05
MSD
Baseline
AutoAugment
Patch Gaussian
Augmix
Transform Subsets
Figure 9: MSD vs corruption test error for which MSD is calculated using VGG-19-BN as the architecture for feature extraction. The corruption error is still calculated using WedeResseet40-2. Compare to Figure 16 to that the qualitative structure of the correlation the regardless of which architecture is used for the feature extractor.
Table 3: Spearman's 's rank coefficient for the correlation between MSD and corruption error for two architectures in the feature extractor: WideResNeet-0-22 and VGG-19-BN. While WideResNet has slightly better correlations overall, the relative behavior across corruptions remains the same for the two architectures.
Corruption
WRS
VGG
Corruption
WRS
VGG
Gaussian Noise
0.76
0.70
Fog
0.68
0.60
Shot Noise
0.83
0.78
Frost
0.66
0.66
Impulse Noise
0.90
0.92
Snow
0.65
Motion Blur
0.86
0.81
Contrast
0.66
0.65
Defocus Blur
0.83
0.78
Pixelate
0.35
0.29
Zoom Noise
0.77
0.68
JPEG Compression
0.33
0.26
Glass Blur
0.69
0.66
Elastic Transform
0.77
0.74
Brightness
0.27
0.08
14

---

Under review as a conference paper at ICLR 2021
Distance variation at 100 corruptions
Distance variation
at 100 images
(std. dist.) (mean dist.)
0.4
0.3
0.2
0.1
0.07
0.06
10
10
10P
10
10P
10
10P
Number of images
Number of corruptions
Figure 10: The standard deviation of the distance between augmentation and corruption center, taken over 100 resamplings of images and corruptions. The standard deviation calculated as percentage of the mean distance, then averaged over augmentttiion-corrpiion pairs. At our choice of parameters, images and 100 corruptions, the standard deviation is only around 5% of the distance. This is smaller than the feature size in the scatter plots of Figure 14
To do so, we use the augmentation scheme from AugMix and corruptions distributions from CIFAR- 10-C to randomly sample 100 augmentation-coomppiiie pairs. Then, for different samplings of fixed number of images and sampled corruptions, we measure the cetpenttttionccorpptii distance the transform feature space 100 times for each inneentttionccorppttii deviation of the distance as percentage of the mean distance for each pair, and average this over pairs. The results shown in Figure 10] For our choice of image and corruption number, the standard deviation in distance is only around 5% of the mean distance, which smaller than the size of the features in the scatter plots in Figure
pair. We calculate the standard
pegnentationncorrpppin
D IMAGENET-C DETAILS
D.1 DATASET CONSTRUCTION DETAILS
First, 30 new corruptions, examples of which are shown in Figure 11 are adapted from common image filters and noise distributions available online (Huxtable 2006; Gladman 2016). These corruptions are generated in 10 severities such that the image remains human interpretable at all severities and the distribution errors on baseline model roughly matches that of ImageNet-C.
For each corruption, groups of 5 severities are generated that roughly match the average spread in eтror across severities in ImageNet-C on a baseline model. Seven of these groups are formed for each corruption, each with one of severity 3 through
the center severity of the group of 5.
candidate dataset is set 10 groups of severities, each from a different corruption whose average coruptiion error on baseline model is within 1 1% of ImageNet-C. This is necessary so that relative decrease in error of data augmented models normalized against a fixed baseline. Also, more distorted, harder transforms are likely farther away, this wasn't 't fixed maximizing distance would likely just pick the hardest transforms in the highest severities. It was computationally infeasible to enumerate all candidate datasets, they one choice of severities was selected at random so that the average corruption error was within %% of ImageNet-C, if existed. Then random disjoint pairs of two sets of were sampled generate candidate datasets. 100k candidate datasets are sampled.
sampled as follows. For each choice of 5 corruptions,
Call the set of all corruption ceuuppiioo-svveittyy pairs in a dataset C. The distance of candidate dataset to ImageNet-C is defined as
(neww CIN-C) BewCuew
min CINCING
(anD(e,cc))
(2)
where dMMD is defined in Section 3 The minimum helps assure that new corruptions are far from all ImageNet-C corruptions.
This distance calculated for all 100k sampled candidate datasets. For CIFAR-10, the same parameters described in Section 4.1 used calculate the distance. For ImageNet, the feature
15

---

Under review as a conference paper at ICLR 2021
Table 4: Comparison between performance on ImageNes/CIFARIOOC and ImageNe/CCPPAROOCC Standard deviations are over 10 runs for CIFAR-10 10 and runs for ImageNet. *ANT results use the pre-trained model provided with the paper and thus has slightly different training parameters and only one run.
IN-C Err
IN-C
C10-C Err
C10-C
Aug
Err
AIN-C
Aug
Err
ACIO-C
Baseline
58.2 40s
57.7 +02
-0.5
Bascline
27.0+06
27.1
+0.1
AA
34.82nt
55.7
+0.9
AA
19.4
21.0204
+1.6
PG
48.1 +02
56.6+0.4
+8.5
PG
17.0+04
23.8+05
+6.8
ANT*
48.8
53.9
+5.1
AugMix
11.1
16,0
+4.9
AugMix
49.1 +0.
52.4+00
+3.4
Table 5: Breakdown of performance individual corruptions in ImageNet/CIFAROOG Standard deviations over 10 rums for CIFAR-10 and runs for ImageNet. Examples and full names of each corruption are given in Appendix L *ANT results use pre-trained model provided with the paper and thus has slightly different training parameters and only one run.
ImageNet-C Corruptions
Aug Basehie
BSmpl
Plsm
Ckbd
CSin
SFrq
Brwn
Prin
1Sprk
Sprk
Rfrac
68.6
71.7 20.7 49.4 84.7
79.0 37.5 34.3 32.4 76.7 402 42.8
AA
24.
207 68.3
20.7
43.8
86.5
78.8
34.5= =08 33.8 202 36.1 =1.0
77.1 21.2 43.8 202
PG
60.3
74.1 10.
4..
82,
=04 76.7 108
38.9
34.6
32.1
76.5
106
42.1 204
ANT*
35.8
75.5
56.9
76.4
63.7
41.0
35.2
35.0
76.1
43.3
AugMix
43.2 208
72.2 46.1 102 76.3 67.4 38.8 _05 32.4
32.3 202 76.4 39.2
CIFAR-10-4 10-C Corruptions
Aug
BSmpl
Brwn
Ckbd
CBlur
1Sprk
Line
P&T
Rppl
Sprk
TCA
Baseline
42.9 27.2 406 11.8 43.3 26.2 11.3 403 21.6 21.0 42.9
AA
17.7 17.5
17.6 9.5
40.4
23.6
10.7 403 23.5
17.5 107
31.8218
9.0
30.1
20.6 12.8=as
35.4+116
20.6
055
8.8
21.5
19.3 405
59.5 =35
AugMix
9.8 27.8 13.4 5,9 30.3 207 18.0=as
8.3 e02
12.1 =04
15.5 40s 19.2=10
extractor is a ResNet-50 trained according to Goyal et al. (2017), except color jittering is not used as a data augmentation. Since there is much greater image diversity in ImageNet, we jointly sample 10k images and corruptions instead of independently sampling 100 images and 100 corruptions. Code for measuring distances and training models is based on pyCls (Radosavovic et al. 2019 2020).
The corruptions then ranked according the their average contribution to the dataset distance. This entire procedure is repeated 10 times for CIFAR and 5 times for ImageNet, and corruption contributions averaged. The top 10 chosen form the new dataset. Of candidate datasets made up of these 10 corruptions, the one with baseline error closest to ImageNet-C is chosen, though run-to-run fluctuation still causes some variation.
D.2 COMPLETE RESULTS
Here we show average results comparing ImageNet/CIFAR 10-C to ImageNe:/CIFAR-106 -O-C in Table and a breakdown of ImageNe:/CIFAR-IO results by corruption Table
E GLOSSARY OF TRANSFORMS
This appendix contains examples of the augmentations and corruptions discussed in the text. Figure II shows the 30 new corruptions introduced in Section These transforms are adapted from common online filters and noise sources (Huxtable 2006) Gladman 2016). They are designed be human interpretable and cover wide range transforms, including noise additions, obscuring, warping, and color shifts. The 10 transforms chosen for ImageNet-C are blue noise sample (BSmpl), plasma noise (Plsm), checkerboard (Ckbd), cocentric sine waves (CSin), single frequency (SFrq), brown noise (Brwn). perlin noise (Prin). inverse sparkle (ISprk), sparkles (Sprk), and caustic refraction (Rfrac). For CIFAR-10-C. 10-C, there is blue noise sample (BSmpl), brown noise (Brwn), checkerboard (Ckbd), circular motion blur (CBlur), inverse sparkle (ISprk). lines (Line), pinch and twirl (P&T), ripple (Rppl), sparkles (Sprk), and transverse chromatic abberation (TCA).
16

---

Under review as a conference paper at ICLR 2021
Figure 12 shows the base transforms used build augmentation schemes in the analysis. These transforms from the Pillow Image Library that are often used as data augmentation. They have no exact overlap with either the corrptiions of ImageNet-C or the There are five geometric transforms (shear x/y, translate x/y, and rotate) and four color transforms (solarize, equalize, autocontrast, and posterize). We choose this particular set of augmentations following Hendrycks et al. (2019).
corruptions
introduce here.
Figure 13 shows example corruptions from the ImageNet-C benchmark (Hendrycks & Dietterich 2018). They a grouped into four categories: noise (gaussian noise, shot noise, and impulse noise), blurs (motion blur, defocus blur, zoom blur, and glass blur), synthetic weather effects (brightness, fog. frost, and snow), and digital transforms (contrast, pixelate. JPEG compression, and elastic transform).
F SUPPLEMENTARY PLOTS
This appendix contains additional plots for the analysis in the main text.
Fig [4shows a comparison of how MMD and MSD correlate with corruption error. MMD typically shows poor correlation, while MSD has strong correlation in all four categories of corruption.
Figure [5 shows the correlation between MSD and corruption error for all 15 ImageNet-C corruptions, where P the Spearman rank correlation. Here, 'AugMix' refers to just their augmentation scheme, and not their gensen-Shannon divergence loss, which gives additional improvements corruption error. 12 of 15 corruptions have Spearman rank correlation greater than 0.6. The remaining three that show poor correlations are 'brightness', 'JPEG compression' and 'pixelate'.
Figure 16 shows the average contribution of a new corruption to the dataset's 's distance from ImageNet- C. The top large average contributions, colored in blue, are chosen as the corruptions to make up the dataset ImageNet-C.
17

---

Under review as a conference paper at ICLR 2021
Cocentric Sine Waves
Single Frequency
Blue Noise
Blue Noise Sample
Voronoi Noise
Perlin Noise
Plasma Noise
Brown Noise
Caustic Noise
Caustic Refraction
Sparkles
Inverse Sparkle
Checkerboard
Lines
Scatter
Fish Eye Lenses
Circular Motion
Ripple
Pinch and Twirl
Water Drop
Quadrilateral Transform
Perspeclive Transform
Chromatc Aberration
Transverse Chromatic Aberration
Color Dither
Pseudocolor
Technicolor
Hue Shift
Color Balance
Bleach Bypass
Figure 11: Examples of each corruption considered when building the dataset dissimilar to ImageNet-C., Basc image C Sehee Park.
18

---

Under review as a conference paper at ICLR 2021
Shear X
Shear Y
Translate X
Translate Y
Rotate
Solarize
Equalize
AutoContrast
Posterize
Figure 12: The base transforms used as augmentations analysis. Base image C Schee Park.
Gaussian Noise
Shot Noise
Impulse Noise
Motion Blur
Defocus Blur
Zoom Blur
Glass Blur
Brightness
Fog
Frost
Snow
Contrast
Pixelate
JPEG Compression Elastic Transform
Figure 13: Examples of the 15 corruptions the ImageNet-C corruption benchmark (Hendrycks & Dietterich
2018). Base image C Sehee Park.
19

---

Under review as a conference paper at ICLR 2021
Gaussian Noise P = 0.20
Zoom Blur
Elastic Transform
Frost
p 0.38
p
p = 0.25
50
25
18-
MMD
40
15-
14-
16
Corruption Error
12
12
2.6
2.8
30
3.2
3.4
0.9
1.1
1.3
1.5
0.5
0.7
0.9
1.0
1.2
1.4
1.6
= 0.76
0 = 0.77
p = 0.77
p= 0.66
24
50
25-
18-
MSD
14-
30
20
12
1.0
1.5
2.0
0.8
1.2
1.6
0.2
0.4
0.6
0.8
0.4
0.8
1.2
1.6
Distance
Figure 14: Example relationships between pnnnenttationcooruptto distance and corruption error for two distance scores, MMD and MSD. p is the Spearman rank correlation. MMD between an augmentation and corruption distribution is not typically predictive of corruption error. MSD correlates well across all four categories of corruption in CIFAR-10-C.
20

---

Under review as a conference paper at ICLR 2021
Gaussian noise, 0.76
Shot noise, p 0.83
Impulse noise, 0.90
80
50
40
40
30-
30
20-
20
10-
10
1.0
2.0
3.0
0.5
1.0
1.5
2.0
2.5
0.5
1.0
1.5
2.0
Motion blur, /
Defocus blur, p
Zoom blur, /
20-
30
20
20
15
10
0.2
0.6
1.0
0.4
0.6
1.0
0.4
0.8
1.2
Glass blur,
0.69
Brightness, P 0.27
Fog, 0.68
60
7.5
12-
Corruption Error
6.5
40
10-
6.0-
5.5-
1.0
1.5
2.0
0.10
0.14
0.18
0.22
0.2
0.4
0.6
Frost, 0
0.66
Snow, P 0.65
Contrast, P = 0.66
20
24
18-
16
14-
0.4
1.2 0.77
Pixelate,
0.35
JPEG compression, 0.33
Elastic transform,
30
18
16-
22
16
12
14
1.2
1.8
0.6
0.8
0.2
0.4
0.6
0.8
Minimal Sample Distance
Baseline
AutoAugment
Patch Gaussian Augmix
Augmentation powerset
Figure 15: See text for details.
21

---

Under review as a conference paper at ICLR 2021
Cifar10
ImageNet
Chsdaa Mater Bu
Down Nata
Thrmwers Divoral Aberalon
Orga Precuuny
Lee3
Scalkr
Quaale Ho tedion
Bowr Now
Ocoatec din Wanee
Warna hoos
Cucale ho
Pla na hoas
Cindaa Vellin Dur
Has D1
Varor Nolss
diglt Precansy
Techeacior
Wsar Dvo
Color Balana
Gaatt hoos
Thundra Chhomali: Fosmmtoo
Caud:: Ratadonn
Quanh Iskerl Tranalom
Oiena Aberatin
Pirch and Tatt
Oocertic Sin eves
Sciter
tka hoor
Diu boles
0.6 -0.4
-22
0.0
02 0.4
-0.8
-44
0.2
0.0
0.2
0.4
Distance shift per standard deviation
Figure 16: A corruption's 's average contribution to the distance to ImageNet-C, as fraction of the population's 's standard deviation. The blue corruptions are those used to construct ImageNet-C.
22


Reviews:
[{"review_id": "u926VQgaoci", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes ImageNet-\\bar{C} which uses a smaller number of carefully chosen corruptions, compared to ImageNet-C. The authors try to argue that previous work is overfitting to ImageNet-C. They claim \"overfitting indeed occurs.\" Additionally, they propose \"Minimum Sample Distance,\" showing that they can predict generalization performance using feature embedding distances.\n\nI don't think they marshaled substantial, far-reaching evidence that \"overfitting indeed occurs.\" They rendered numerous corruptions from Huxtable, 2006; Gladman, 2016 (which I very much appreciate) and adversarially chose the worst corruptions. As a result, the best models performed worse by a few percent compared to ImageNet-C.\n_A drop is inevitable and expected given the adversarial selection. If the drop was consistently more than, say, 20%, then there might be strong evidence of overfitting._ Since the degradation for some techniques is small, this strikes me as evidence that by-and-large the community isn't overfitting. (Note it was evident to all that Patch Gaussian and ANT were specialized to noise corruptions.) \"Is Robustness Robust?\" It seems like the answer is \"yes\" but the authors are trying to argue that the answer is \"no.\"\n\nThis paper is fairly similar to _Increasing the Coverage and Balance of Robustness Benchmarks by Using Non-Overlapping Corruptions_ (submission #1101). If these papers have very different ratings, then there's a problem with this review process.\n\n\"AugMix, which introduces a broad array of both geometric and color augmentations\"\nNo it doesn't. It removes several color augmentations from AutoAugment and doesn't introduce any augmentation primitive beyond elementwise convex combinations.\n\n\"On other corruptions, the increase in error is even worse than the mean would suggest, and even broad augmentations like AugMix....\"\nAutoAugment is more broad; AugMix is narrower.\n\n\"Second, AutoAugment is the only tested augmentation scheme that was designed before the release of ImageNet-C\"\nAugMix uses a proper subset of AutoAugment's augmentations. It's not as though AugMix added in distortions to fit ImageNet-C; it removes augmentations because AA fits some of ImageNet-C's corruptions directly. This observation makes their overfitting case hard to maintain. \n\nAlso, if AutoAugment's full augmentation list is fair game for ImageNet-\\bar{C}, then the authors should train AugMix with the full set of augmentations and make that comparison; the generalization discrepancy would likely be even smaller were the augmentation sets made equal.\n\nFrom the \"Corruption robustness as a secondary learning task\" section:\n\"To mitigate overfitting, standard machine learning practice would dictate a training/validation/test set split; it is only the size and breadth of modern vision datasets that has allowed this to be neglected in certain cases recently.\"\n\"having a held-out test set that is not used during model development seems necessary.\"\n\"ImageNet-C has only 15 corruption types\"\n\"ImageNet-C could serve as a validation set and ImageNet-\\bar{C} could serve as a held-out test set\"\nEssentially, since the community lacks a validation set, ImageNet-C should become the validation set, and ImageNet-\\bar{C} should become the test set.\nThis section might be negligent or worse for not acknowledging the already existent ImageNet-C validation set. ImageNet-C provides a validation set with about half the corruptions of ImageNet-\\bar{C}. There are 19 available ImageNet-C corruptions, so the community already has a validation set.\n\nThe authors should cite or compare to other works that use feature distances to predict generalization. An example is \"The Frechet Distance of training and test distribution predicts the generalization gap.\"\n\nUpdate: \"Noisy Student and Assemble-ResNet use without removing overlapping augmentations, yet they test on ImageNet-C.\" This is a bad practice and I should hope the authors of this paper only have experiments where there is a train-test mismatch (otherwise we're not testing robustness to distribution shift).", "extra_scores": {}, "extra_text": {}}, {"review_id": "qIGKxk2Q14W", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a metric to quantify the perceptual similarity between different kind of corruptions and uses it to show that training on a corruptions induces robustness to other corruptions which are perceptually similar. Analyzing the current data augmentation based methods for robustness using this lens, the authors hypothesis that we are currently overfitting to the existing robustness benchmark (ImageNet-C) and proposes a new benchmark with a new set of corruptions. \n\nI think the author's observations that training on some corruptions helps the network to be robust to similar corruptions in test time is quite intuitive. I appreciate that the paper tries to quantify this and performs an extensive empirical study. The insight from this study and the new proposed benchmark will be useful to further advance the research in this area. \n\nI have a some questions/clarifications:\n1. What are the augmentations that were used while training the network used to extract features for the metric? Does the metric will probably depend on the architecture of the network used as well. Have you verified that the metric is robust to different architectures and the same conclusion holds? I am not sure if this is in the supplementary material somewhere and I missed it. \n2. Do we find any non-intuitive pairs of corruptions which are similar? It occurs to me that most geometric based corruptions are similar and noise based corruptions are similar etc, but is there any pair of corruptions across these groups that the metric identifies as similar?\n\n\nUpdate after rebuttal: I appreciate the author response. I will maintain my original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GECfZ-aPmhQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces the Minimal Sample Distance (MSD): a measure of the minimal distance, in a trained network representation space, between samples modified with an augmentation and the average of all samples modified by a corruption. It uses this metric to claim that there exists a high correlation between the corruption error and the MSD of a given augmentation. This way, it claims that focusing on benchmarks like ImageNet-C may lead to overfitting to the corruptions present in that benchmark. \n\nOne problem is that this correlation isn’t true for all augmentations. Only 4 are highlighted in the main text. As the paper describes, a few corruptions have spearman coefficient of less than 0.5. Particularly notable is brightness which, despite having very low spearman coefficient, is used to show that Patch Gaussian has “overfit” to the noise corruptions in the ImageNet-C benchmark. This is especially worrying since in their original paper, Patch Gaussian shows improvement in non-noise as well, which couldn’t have come from overfitting. Additionally, why was AutoAugment, but not RandAugment tested?\n\nAnother concern is that it may not make sense to compare single augmentations (such as Patch Gaussian) with augmentation policies, such as AutoAugment, RandAugment, and AugMix. It’s possible that individual augmentations in these policies “overfit” to the corruptions as well, but that this isn’t shown in MSD due to the use of many corruptions. In which case, using PatchGaussian in the RandAugment search space (for instance) would resolve any issues. \n\nThe paper argues that AutoAugment \"overfits\" while AugMix doesn't, but that's only because AugMix explicitly removed any augmentations in ImageNet-C from its search space, so it would make sense to repeat this experiment with the augmentations present in AugMix in order to confirm that it doesn't indeed \"overfit\".\n\nThe paper then suggests that one solution would be to use MSD to sample dissimilar corruptions to test on. However, given that it seems like there’s no evidence of overfitting for augmentation policies that encourage a diversity of augmentations, such as AugMix (which is in line with Yin et al 2019). Then I’m not sure what the benefit of expanding the robustness benchmark is. If current methods have indeed overfit, why won’t we also overfit to the new benchmark’s corruptions? \n\nOverall, the paper presents interesting results and discussion. \n\nUpdate after rebuttal: I appreciate the authors' response and clarifications. I maintain my original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2OMZK3oJOhh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary \nThe paper studies the importance of similarity between augmentations and corruptions for improving performance on those corruptions. To measure the distance between the augmentation and corruption distributions, the paper proposes a new metric, Minimal Sample Distance (MSD), which is the perceptual similarity between an average corruption and the closest augmentation from a finite set of samples sampled from the augmented data distribution. It is shown that MSD overcomes the drawbacks of distributional distance measures like Maximum Mean Discrepancy (MMD). A new benchmark, called ImageNet-C-bar, made up of corruptions that are perceptually dissimilar to ImageNet-C, is introduced. Using standard evaluation, it is empirically shown that several recent augmentation schemes show degraded performance on the new dataset, suggesting that they generate augmentations only perceptually similar to ImageNet-C and thus are prone to overfitting. \n\n+ves\n+ Although the notion of the relation between augmentations and test-time corruptions has somewhat already been empirically observed and stated in many previous works, the paper tries to correlate this relation statistically. To my knowledge, this is the first such work.\n+ Through comprehensive evaluations, the paper shows that the proposition of computing MSD rather than MMD correlates well with the relation between augmentations and corruptions observed in reality.\n+ A new benchmark, ImageNet-C-bar is proposed, which shows a useful result to the community that recent SOTA augmentation methods have degraded performance on the new dataset because they generate augmentations close to ImageNet-C corruptions. \n\nConcerns\n- The paper says (pg 2) - “we empirically show an intuitive yet surprisingly overlooked finding: Augmentation-corruption perceptual similarity is a strong predictor of corruption error”. However, this notion of the relation between augmentations and test-time corruptions does not seem very surprising. It’s perhaps well-known that DNNs will generalize well only when test distributions are fairly similar to training distributions. Hence, the importance of this observation does seem to be overemphasized, although this is certainly of use.  \n\n- It's been recently shown that removing texture biases by introducing stylized transformations also improves robustness to common-corruptions (Geirhos et al, ICLR 2019). However, stylized transformations don’t look close to any Imagenet-C corruptions. If MSD of stylized transforms is large (which intuitively seems so), then it will mean that MSD is not a reliable metric in such a case. Was this studied? Would MSD be a reliable metric even in such cases? It would be good to understand this, to get a more well-rounded picture of the proposed metric. \n\n- Paper introduces MSD as a distance metric. However, distance metrics should be symmetric in nature. It is not quite evident from just Eq-1 if MSD is symmetric. \n\n- In addition, adding high severity corruption as training augmentation leads to better performance on the same low severity corruption but vice-versa is not true in general. This suggests that measures should perhaps ideally be asymmetric. Clarifying the notion of “distance metric” in this work may be important to make the work mathematically correct.\n\n- Standard choices for measuring perceptual distances are VGG-16 or 19 networks pre-trained on ImageNet. However, the paper chooses to use WRN-40-10 trained on CIFAR-10. This seems to deviate from standard settings. The paper should explain the rationale behind their choice. It will be great to show an ablation on how this choice of feature extractor (VGG-19, Robust VGG etc) affects the MSD. Ideally, the metric should be robust i.e. shouldn’t be sensitive to the choice of feature extractor.\n\n- The practical utility of ImageNet-C-bar seems limited and unclear. The only use that I can think of is using it to identify overfitting on ImageNet-C. I would be happy to understand what I am missing here.\n\nPOST-REBUTTAL:\n\nI thank the authors for the response and the revisions to the paper. I appreciate the authors' efforts towards the rebuttal. I am however left with some concerns which did not have a convincing resolution:\n\n* Regarding the comment on how the proposed analysis would look for stylized transforms, the authors say in the response that \"...we don’t expect that perceptual similarity is the only cause of improved corruption error, only that perceptual similarity is particularly salient for predicting generalization to dissimilar corruptions...\". The work seems to be one-sided in this regard. If stylized transforms don't look perceptually similar to ImageNet-C corruptions but provide robustness, this counters the proposed hypothesis. It then becomes important to say where the proposed analysis is meaningful and where it is not. This seems to be lacking at this time in the work.\n\n* Regarding the robustness of MSD to the choice of feature extractor (as also asked by R1), the revised paper includes results on VGG as feature extractor (thanks to the authors for this), but uses a model that is finetuned for CIFAR-10. In general, perceptual similarity is studied directly taking VGG pre-trained on ImageNet - without finetuning on the target dataset. This leaves this question open, and makes one wonder if the latter features did not support the analysis.\n\n* The utility of Imagenet-C-bar as an additional benchmark to check the goodness of performance on Imagenet-C seems a bit convoluted. Would we need a Imagenet-C-bar-bar to check the goodness for corruptions that may be beyond perceptual similarity (such as stylized transforms)? This is not very convincing.\n\nOverall, I am still on the fence on this work (and retain my original decision at this time). I think the paper does present an interesting insight, but I am not very convinced it has been studied and explored comprehensively enough. I would have ideally preferred to give a borderline (neither positive nor negative) decision, and will not be disappointed if the work is accepted, considering the interesting insights it offers. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the relationship between data augmentations used during training and their effect on the accuracy when evaluated on unseen corruptions at test-time. The paper proposes a metric called minimal sample distance (MSD) to measure the similarity between augmentations during training time and corruptions at test time.

The reviewers agree that the paper aims to solve an important problem and the paper has some interesting findings. However, the current version has a few shortcomings:
- Some of the claims about “overfitting” are confusing, especially for data augmentations that use ops similar to those in ImageNet-C. This is already known and which is why some papers uses a subset of operations (e.g. AugMix uses a subset of AutoAugment operations).
- The main take-home message and novelty is unclear: The initial version titled (“Is Robustness Robust?“) seemed to argue that we may be overfitting to Imagenet-C, but the rebuttal and the updated version revised some of the claims (see response to R3 and R4).  In light of the revision, I’m not sure how the main take-home messages differ from existing papers such as Yin et al. 2019 or “Many faces of robustness”. 
- One of the main differences is quantification of the distribution similarity, however, as pointed out by R2, this analysis does not explain when stylized corruptions would help, so the current version of the paper feels a bit incomplete to me.

I recommend the authors to revise the draft based on reviewer feedback and resubmit the paper to another venue.

Decision: Reject