A Critical Analysis of Distribution Shift
Authors: Dan Hendrycks
Steven Basart
Norman Mu
Saurav Kadavath
Frank Wang
Evan Dorundo
Rahul Desai
Tyler Zhu
Samyak Parajuli
Mike Guo
Dawn Song
Jacob Steinhardt
Justin Gilmer
Venue:  (2021)
Keywords: distribution shift
ood

Abstract: We introduce three new robustness benchmarks consisting of naturally occurring distribution changes in image style, geographic location, camera operation, and more. Using our benchmarks, we take stock of previously proposed hypotheses for out-of-distribution robustness and put them to the test. We find that using larger models and synthetic data augmentation can improve robustness on real-world distribution shifts, contrary to claims in prior work. Motivated by this, we introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000x more labeled data. We find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Hence no evaluated method consistently improves robustness. We conclude that future research should study multiple distribution shifts simultaneously.

Under review as a conference paper at ICLR 2021
A CRITICAL ANALYSIS OF DISTRIBUTION SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce three new robustness benchmarks consisting of naturally occurring distribution changes in image style, geographic location, camera operation, and more. Using our benchmarks, we take stock of previously proposed hypotheses for out-of-distribution robustness and put them to the test. We find that using larger models and synthetic data augmentation can improve robustness on real-world distribution shifts, contrary to claims in prior work. Motivated by this, we introduce a new data augmentation method which advances the state-of-the-an and outper- forms models pretrained with 1000 more labeled data. We find that synthetic augmentations can sometimes improve real-world robustness. We also find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geo- graphic changes. Hence evaluated method consistently improves robustness. We conclude that future research must study multiple distribution shifts simultaneously.
INTRODUCTION
While the research community must create robust models that generalize to new scenarios, the robustness literature (Dodge and Karam 2017 Geirhos et al. 2020) lacks consensus on evaluation (200aa find that many recent language models are already robust to many forms of distribution shift, while Yin et al. (2019) and Geirhos et al. (2019) find that vision models are largely fragile and argue that data augmentation offers one solution. In contrast, Taori et al. (2020) provide results suggesting that using pretraining and improving in-distribution test set accuracy improve natural robustness, whereas other methods do not.
benchmarks and contains many dissonant hypotheses, Henddryeks
In this paper we articulate and systematically study seven robustness hypotheses. The first four hypotheses concern methods for improving robustness, while the last three hypotheses concern abstract properties about robustness. These hypotheses are as follows.
Larger Models: increasing model size improves robustness (Hendrycks and Dietterich, 2019 Xie
and Yuille 2020).
Self-Attentio:: adding self-attention layers models improves robustness (Hendryeks al. 2019b).
Diverse Data Augmentation: robustness can increase through data augmentation (Yin et Pretraining: pretraining on larger and more diverse datasets improves robustness (Orhan 2019 Hendrycks al. 2019a).
2019
Texture Bias: convolutional networks are biased towards texture, which harms robustness Geirhos ct al., 2019).
Only IID Accuracy Matters: accuracy on independent and identically distributed test data entirely determines natural robustness.
Synthetic +/ Real: synthetic robustness interventions including diverse data augmentations do not help with robustness on real-world distribution shifts (Taori et al.) (020).
It has been difficult to arbitrate these hypotheses because existing robustness datasets preclude the possibility of controlled experiments by varying multiple aspects simultaneously. For instance, Texture Bias was initially investigated with synthetic distortions (Geirhos et al.) (2018). which conflicts with the Synthetic ++ Real hypothesis. On the other hand, natural distribution shifts often affect many factors (e.g., time, camera, location, etc.) simultancously in unknown ways (Recht et al. 2019

---

Under review as a conference paper at ICLR 2021
ImageNet-Renditions
DeepFashion Remixed
StreetView StoreFronts
Paintings
Occlusion Viewponn
Year
Sculpture
Country
Waternelm
Figure 1: Images from our three new datasets ImageNet-Rendition (ImageNet-R), DeepFashion Remixed (DFR), and View StoreFronts (SVSF). The SVSF images are recreated from the public Google StreetView, copyright Google 2020. Our datasets test robustness to various naturally occurring distribution shifts including rendition style, camera viewpoint, and geography.
Hendrycks et al. 2019b). Existing datasets also lack diversity such that it hard to extrapolate which methods will improve robustness more broadly. To address these issues and test the seven hypotheses outlined above, we introduce three new robustness benchmarks and a new data augmentation method.
First we introduce ImageNet-Renditionn (ImageNet-R), 30,000 image test containing various renditions (e.g., paintings, embroidery, etc.) of ImageNet object classes. These renditions are naturally occurring, with textures and local image statisties unlike those ImageNet images, allowing to Real hypotheses.
more cleanly separate the Texture Bias and Synthetic
+
Next, we investigate natural shifts in the image capture process with StreetView StoreFronts (SVSF) and DeepFashion Remixed (DFR). SVSF contains business storefront images taken from Google Streetview, along with metadata allowing us to vary location, year, and even the camera type. DFR leverages the metadata from DeepFashion2 (Ge et al.) 2019) to systematically shift object occlusion, orientation, zoom, and scale at test time. Both SVSF and DFR provide distribution shift controls and do not alter texture, which remove possible confounding variables affecting prior benchmarks.
Finally, we contribute DeepAugment to increase robustness to some new types of distribution shift. This augmentation technique uses image-to-image neural networks for data augmentation, not inta-inndeendeet Euclidean augmentations like image shearing or rotating as in previous work. DeepAugment achieves ttate-of-the-at robustness and a corruption robustness henchmark. DeepAugment can also be combined with other augmentation methods outperform model pretrained on 1000 more labeled data.
newly introduced ImageNet-R benchmark
After examining our results on these three datasets and others, we can rule out several of the above hypotheses while strengthening support for others. As example, we find that synthetic data augmentation robustness interventions improve accuracy on ImageNet-R and real-world image blur distribution shifts, providing clear counterexamples to Synthetic + Real while lending support to the Diverse Data Augmentation and Texture Bias hypotheses. In the conclusion, we summarize the various strands of evidence for and against each hypothesis. Across our many experiments, we do not find general method that consistently improves robustness, and some hypotheses require additional qualifications. While robustness is often spoken of and measured as a single scalar property like accuracy, our investigations suggest that robustness is not so simple. In light of our results, we hypothesize in the conclusion that robustness multivariate.
RELATED WORK
Robustness Benchmarks. Recent works (Hendrycks and Dietterich, 2019 Recht et al.
2019
Hendrycks et al. (020a) have begun to characterize model performance on out out- distrribtutnnnno (OOD)
data with various test sets, with dissonant findings. For instance, [Hendrycks et al. (2020a)
demonstrate that modern language processing models are moderately robust to numerous naturally
occurring distribution shifts, and that Only IID Accuracy Matters is inaccurate for natural language
2

---

Under review as a conference paper at ICLR 2021
Painting
Sculpture
Embroidery
Origami
Cartoon
Toy
Figure ImageNet-RRnditione (ImageNet-R) contains 30,000 images of ImageNet objects with different textures and styles. This figure shows only a portion of ImageNet-R" numerous rendition styles. The rendition styles (e.g.. "Toy") are for clarity and are nos ImageNet-R's classes; ImageNet- R's classes subset of 200 ImageNet classes. ImageNet-R emphasizes shape over texture.
tasks. For image recognition, Hendrycks and Dietterich 2019) analyze image models and show that they are sensitive to various simulated image corruptions (e.g., noise, blur, weather, JPEG compression, etc.) from their "ImageNet-C" benchmark.
Recht et al. (2019) reproduce the ImageNet (Russakovsky et al. 2015 validation set for use benchmark of naturally occurring distribution shift in computer vision. Their evaluations show 11-14% drop in accuracy from ImageNet to the validation wide range of architectures. TToor et al. (2020) use ImageNetV2 to measure natural robustness and dismiss Diverse Data Augmentation. Recently, |Engstrom et al. (2020) identify statistical biases in ImageNetV2's construction, and they estimate that reweighting ImageNetV2 to correct for these biases results in less substantial 3.6% drop.
named ImageNetV2, across
Data Augmentation. Geirhos et al. (2019): Yin et al. (0019: :: Hendrycks et al. (202bb)
demonstrate
that data augmentation can improve robustness on ImageNet-C. The space of augmentations that
help robustness includes various types of noise (Madry et al., 2017 Rusak et al.
2020
Lopes et al.
2019). highly unnatural image transformations (Geirhos al. 2019 Yun al. 2019
Zhang et al.
2017), or compositions of simple image transformations such as Python Imaging Library operations
(Cubuk al. 22188) Hendrycks et al. 2020b). Some these augmentations can improve accuracy on in-distribution examples as well as out-t--distriuutio (OOD) examples.
3
NEW BENCHMARKS
In order evaluate the seven robustness hypotheses, we introduce three new benchmarks that capture new types of naturally occurring distribution shifts. ImageeNet-Renditon collected test set intended for ImageNet classifiers, whereas StreetView StoreFronts (SVSF) and DeepFashion Remixed (DFR) each contain their own training sets and multiple test sets. SVSF and DFR split data into a training and test sets based on various image attributes stored the metadata. For example, we can select a test set with images produced by camera different from the training set camera. We now describe the structure and collection of each dataset.
(ImageNet-R) is a newly
3.1 IMAGENET-RENDITINN (IMAGENET-R)
While current classifiers can learn some aspects of an object's shape Moordvintsev et al. 2015). they nonetheless rely heavily natural textural cues (Geirhos et al. 2019). In contrast, human vision can process abstract visual renditions. For example, humans can recognize visual scenes from line drawings quickly and accurately as they from photographs (Biederman and Jul (988). Even some primates species have demonstrated the ability to recognize shape through line drawings Tanaka 2006).
(Itakura 1994

---

Under review as a conference paper at ICLR 2021
To measure generalization to various abstract visual renditions, create the ImageNet-Renditionn (ImageNet-R) dataset. ImageNet-R contains various artistic renditions of object classes from the original ImageNet dataset. Note the original ImageNet dataset discouraged such images since annotators were instructed to collect "photos only, no painting, no drawings, etc." (Deng 2012). We do the opposite.
Data Collection. ImageNet-R contains 30,000 image renditions for 200 ImageNet classes. We choose a subset of the ImageNet-IK classes, following Heendrycks et al. (2019b), for several reasons. A handful ImageNet classes already have many renditions, such as "triceratops." We also choose a subset so that model misclassifications are egregious and to reduce label noise. The 200 class subset was also chosen based on rendition prevalence, as "strawberry" renditions were easier to obtain than "radiator" renditions. Were we to use all 1,000 ImageNet classes, annotators would be pressed to distinguish between Norwich terrier renditions as Norfolk terrier renditions, which is difficult. We collect images primarily from Flickr and use queries such as "art," "cartoon," "grafto,,,"rraffit"""""" "embroidery," "graahhcss"" "origami," "painting," " "pattem," "plastic object," "plush object," "sculpture." "line drawing," " "tattoo," "toy." " "video game," and so on. Images are filtered by Amazon MTurk annotators using a modified collection interface from ImageNetV2 (Recht al., 2019). For instance, after scraping Flickr images with the query "lighthouse cartoon," we have MTTurk annotators select true positive lighthouse renditions. Finally, as second round of quality control, graduate students manually filter the resulting images and ensure that individual images have correct labels and do not contain multiple labels. Examples are depicted in Figure 2 ImageNet-R also includes the line drawings from Wang et al.] (2019). excluding horizontally mirrored duplicate images, pitch black images, and images from the incorrectly collected "pirate ship" class.
3.2 STREETVIEW STOREFRONTS (SVSF)
Computer vision applications often rely on data from complex pipelines that span different hardware, times, and geographies. Ambient variations in this pipeline may result in unexpected performance degradation, such as degradations experienced by health care providers in Thailand deploying
laboratory-tuned diabetic retinopathy classifiers in the field (Beede et al. 2020). In order to study the effects of shifts in the image capture process collect the StreetView StoreFronts (SVSF) dataset, new image classification dataset sampled from Google Street imagery (Anguelov et al. 2010) focusing on three distribution shift sources: country, year, and camera.
Data Collection. SVSF consists of cropped images of business store fronts extracted from StreetView images by an object detection model. Each store front image is assigned the class label of the associated Google Maps business listing through a combination of machine learning models and human annotators. We combine several visually similar business types (e.g. drugstores and pharmacies) for total of 20 classes, listed Appendix B
Splitting the data along the three metadata attributes of country, year, and camera, we create one training set and five test sets. We sample a training set and in-distribution test set (200K and 10K images, respectively) from images taken in US/Mexico/Canada during 2019 using a "new" camera system. We then sample four OOD test sets (10K images each) which alter one attribute at a time while keeping the other two attributes consistent with the training distribution. Our test sets are year: 2017, 2018; country: France; and camera: "old."
3.3 DEEPFASHION REMIXED
Changes in day-to-day camera operation can cause shifts in attributes such as object size, object occlusion, camera viewpoint, and camera zoom. To measure this, we repurpose DeepFashion2 (Ge) et al. 2019) to create the DeepFashion Remixed (DFR) dataset. We designate training set with 48K images and create eight out-of-distribution test sets to measure performance under shifts in object size, object occlusion, camera viewpoint, and camera zoom-in. DeepFashion Remixed is a multi-label classification task since images may contain more than one clothing item per image.
Data Collection. Similar SVSF, fix value for each of the four metadata attributes in the training distribution. Specifically, the DFR training set contains images with medium scale, medium occlusion, side/back viewpoint, and zoom-in. After sampling IID test set, construct eight
4

---

Under review as a conference paper at ICLR 2021
Original
DeepAugment
Figure 3: DeepAugment examples preserve semantics, are data-dependent, and are far more visually diverse than augmentations such as rotations.
OOD test distributions by altering one attribute time, obtaining test sets with minimal and heavy occlusion; small and large scale; frontal and not-worn viewpoints; and medium and large zoom-in. See Appendix B for details test set sizes.
DEEPAUGMENT
In order to further explore the Diverse Data Augmentation hypothesis, we introduce a new data augmentation technique. Whereas most previous data augmentations techniques simple augmen- tation primitives applied to the raw image itself, we introduce DeepAugment, which distorts images by perturbing internal representations of deep networks.
Duepengment works by passing a clean image through an image-to-image network and introducing several perturbations during the forward pass. These perturbations are randomly sampled from of manually designed functions and applied to the network weights and to the feed-forward signal random layers. For example, our of perturbations includes zeroing, negating, convolving, transposing, applying activation functions, and more. This setup generates semantically consistent images with unique and diverse distortions Fi Although our set of perturbations is designed with random operations, we show that DeepAugment still outperforms other methods on benchmarks such ImageNet-C and ImageNet-R.
provide the pseudocode in Appendix C
For our experiments, we specifically use the CAE (Theis et al.) 2017) and EDSR (Lim et al. 2017) architectures as the basis for DeepAugment. CAE is an autoencoder architecture, and EDSR is superresolution architecture. These two architectures show the DeepAugment approach works with different architectures. Each clean image in the original dataset and passed through the network and is thereby stochastically distored, resulting in two distorted versions of the clean dataset (one for CAE and one for EDSR). We then train on the augmented and clean data simultancously and call this approach DeepAugment.
5

---

Under review as a conference paper at ICLR 2021
ImageNet-200) (%)
ImageNet-R (%)
Gap
ResNet-50
7.9
63.9
56.0
+ ImageNet-21K Pretraining (10x labeled data)
7.0
62.8
55.8
CBAM (Self-Attention)
7.0
63.2
56.2
+ Lo Adversarial Training
25.1
68.6
43.5
Speckle Noise
8.1
62.1
54.0
+ Style Transfer Augmentation
8.9
58.5
49.6
AugMix
7.1
58.9
51.8
DeepAugment
7.5
57.8
50.3
DeepAugment AugMix
8.0
53.2
45.2
ResNet- 152 (Larger Models)
6.8
58.7
51.9
Table ImageNet-200 and ImageNet-R top-1 error rates. ImageNet-200 uses the same 200 classes as ImageNet-R. DeepAuugment improves the baseline by over 10 percentage points. ImageNet-21 Pretraining tests Pretraining and CBAM tests Self-Attention. Style Transfer, AugMix, and DeepAugment test Diverse Data Augmentation in contrast to simpler noise augmentations such as oo Adversarial Noise and Speckle Noise. While there remains much room for improvement, results indicate that progress ImageNet-R is tractable.
5 EXPERIMENTS
5.1 SETUP
In this section we briefly describe the evaluated models, pretraining techniques, self-attention mecha- nisms, data augmentation methods, and note various implementation details.
Model Architectures and Sizes. Most experiments are evaluated on a standard ResNet-50 model (He et al. 2013). Model size evaluations ResNets ResNeXts (Xie al. 2016) of varying sizes.
Pretraining. For pretraining we use ImageNet- -1 which contains approximately 21,000 classes and approximately 14 million labeled training images, around 10x more labeled training data than ImageNet-IK: We tune Kolesnikov et al. (2019)'s ImageNet-21K model. We also use large pre- trained ResNeXt-101 model from Mahajan et al. (2018). This was pre-trained billion Instagram images with hashtag labels and fine-tuned on ImageNet- IK. This Weakly Supervised Learning (WSL) pretraining strategy uses approximately x more labeled data.
approximately 1
Self- Cell-Attention.
When studying self-attention, we employ CBAM (Woo et al. 2018) and SE (Hu al. 2018) modules, two forms of self-attention that help models learn spatially distant dependencies.
Data Augmentation. We use Style Transfer, AugMix, and DeepAugment to analyze the Diverse Data Augmentation hypothesis, and contrast their performance with simpler noise augmentations such as Speckle Noise and adversarial noise. Style transfer (Geirhos et al. 2019) uses style transfer network apply artwork styles training images. We use AugMix (Hendryeks et al. 2020b) which randomly composes simple augmentation operations (e.g., translate, posterize, solarize). DeepAugment, introduced above, distorts the weights and feedforward passes of image-to-image models to generate image augmentations. Speckle Noise data augmentation muliplies each pixel by (1 x)) with æ sampled from a normal distribution (Rusak et al. 2020 Hendrycks and Dietterich 2019). We also consider adversarial training as form of adaptive data augmentation and use the model from Wong et al. (2020) trained against loo perturbations of size 4/255.
5.2
RESULTS
We now perform experiments on ImageNet-R, StreetView StoreFronts, DeepPashion Remixed. We also evaluate ImageNet-C and compare and contrast it with real distribution shifts.
ImageNet-R. Table shows performance on ImageNet-R as well as ImageNet-200 (the original ImageNet data restricted to ImageNet-R's 200 classes). This has several implications regarding the four method-specifie hypotheses. Pretraining with ImageNet-21K (approximately 10x labeled data) hardly helps. Appendix A shows WSL pretraining can help, but Instagram has renditions, while ImageNet excludes them; hence conclude comparable pretraining was ineffective. Notice

---

Under review as a conference paper at ICLR 2021
Self-Artention increases the IID/OOD gap. Compared to simpler data augmentation techniques such as Speckle Noise, the Diverse Data Augmentation techniques of Style Transfer, AugMix, and DeepAugment improve generalization. Note AugMix and DeepAugment improve in-distribution performance whereas Style transfer hurts it. Also, our new DeepAugment technique is the best standalone method with error rate of 57.8%. Last, Larger Models reduce the IID/OOD gap.
Regarding the three more abstract hypotheses, biasing networks away from natural textures through diverse data augmentation improved performance, so find support for the Texture Bias hypothesis. The IID/OOD generalization gap varies greatly which condtradicts Only IID Accuracy Matters. Finally, since ImageNet-R contains on ImageNet-R, we now have clear evidence against the Synthetic 7 Real hypothesis.
examples, and since synthetic data augmentation helps
Vire StoreFronts. In Table 2 we evaluate data augmentation methods on SVSF and find that all of the tested methods have mostly similar performance and that no method helps much on country shift, where error rates roughly double across the board. Here evaluation limited to augmentations due to a 30 day retention window for each instantiation of the dataset. Images captured in France contain noticeably different architectural styles and storefront designs than those captured in US/Mexico/Canada, meanwhile, we are unable to find conspicuous and consistent indicators of the camera and year. This may explain the relative insensitivity of evaluated methods to the camera and year shifts. Overall Diverse Data Augmentation shows limited benefit, suggesting either that data augmentation primarily helps combat texture bias as with ImageNet-R, or that existing augmentations not diverse enough capture high-level semantic shifts such building architecture.
Hardware
Year
Location
Network
IID
Old
2017
2018
France
ResNet-50
27.2
28.6
27.7
28.3
56.7
+ Speckle Noise
28.5
29.5
29.2
29.5
57.4
Style Transfer
29.9
31.3
30.2
31.2
59.3
DeepAugment
30.5
31.2
30.2
31.3
59.1
AugMix
26.6
28.0
26.5
27.7
55.4
Table 2: SVSF classification error rates. Networks are robust to some natural distribution shifts but are substantially more sensitive the geographic shift. Here Diverse Data Augmentation hardly helps.
Size
Occlusion
Viewpoint
Zoom
Network
IID
OOD
Small
Large
Slight/None
Heavy
No Wear
Side/Back,
Medium
Large
ResNet-50
77.6
55.1
39.4
73.0
51.5
41.2
50.5
63.2
48.7
73.3
+ ImageNet-21K. Pretraining
80.8
58.3
40.0
73.6
55.2
43.0
63.0
67.3
50.5
73.9
SE Seelf-Ariention)
77.4
55.3
38.9
72.7
52.1
40.9
52.9
64.2
47.8
72.8
Random Erasure
78.9
56.4
39,9
75.0
52.5
42.6
53.4
66.0
48.8
73.4
Speckle Noise
78.9
55.8
38.4
74.0
52.6
40.8
55.7
63.8
47.8
73.6
Style Transfer
80.2
57.1
37.6
76.5
54.6
43.2
58.4
65.1
49.2
72.5
+ DecpAugment
79.7
56.3
38.3
74.5
52.6
42.8
54.6
65.5
49.5
72.7
AugMix
80.4
57.3
39.4
74.8
55.3
42.8
57.3
66.6
49.0
73.1
152 (Larger Models)
80.0
57.1
40.0
75.6
52.3
42.0
57.7
65.6
48.9
74.4
Table 3: DeepFashion Remixed results. Unlike the previous tables, higher is better since all values mAP scores for this multi-label classification benchmark. The "OOD" column is the average of the row's rightmost eight OOD values. All techniques do little to close the IID/OOD) generalization gap.
DeepFashion Remixed. Table 3 shows our experimental findings on DFR, in which all evaluated methods have an average OOD mAP that is close the baseline. In fact, most OOD mAP increases track IID mAp increases. In general, DFR's size and occlusion shifts hurt performance the most. We also evaluate with Random Erasure augmentation, which deletes rectangles within the image, to simulate occlusion (Zhong et al. 2017). Random Erasure improved occlusion performance, but Style Transfer helped even more. Nothing substantially improved OOD performance beyond what is explained by IID performance, here it would appear that Only IID Accuracy Matters. Our results do not provide clear evidence for the Larger Models, Self-Attentonn, Diverse Data Augmentation, and Pretraining hypotheses.
ImageNet-C. We now consider a previous robustness benchmark to reassess all seven hypotheses. We use the ImageNet-C dataset (Hendrycks and Dietterich] 2019) which applies 15 common image
7

---

Under review as a conference paper at ICLR 2021
ImageNet- C Error Across Severities
IID Vs. OOD Generalization
80
70
1000x Labeled Data (WSL)
ImageNet-CC Accuracy (%)
70
ImageNet-C Accuracy (%)
OpeepAugment+Auggae
60
60
50
50
ResNeXt
40
ResNet
ResNet-50
Style Transfer
30
20
DeepAugment+AugMe
1
3 Corruption Severity
2
4
5 68
70
72
74
76
78
80
82
ImageNet Accuracy (%)
Figure 4: The left figure shows accuracy as function of corruption severity. the right figure, we show ImageNet accuracy and ImageNet-C accuracy. Previous architectural advances slowly translate to ImageNet-C performance improvements, but the DeepAugment-Augee robustness intervention on a ResNet-50 approximately yields 19% accuracy improvement. Bee
Polar Bear
Birdhouse
Basketball
Figure 5: Examples of real-world blurry images from our collected dataset for ImageNet-C analysis.
corruptions (e.g., Gaussian noise, defocus blur, simulated fog, JPEG compression, etc.) across 5 severities ImageNet-IK validation images. We find that DeepAugment improves robustness on ImageNet-C. Figure 4 shows that when models are trained with AugMix and DeepAugment, they attain the state-of-the-arr break the trendline, and exceed the corruption robustness provided by training on x000 more labeled training data. Note the augmentations from AugMix and DeepAugment are disjoint from ImageNet-C's corruptions. Full results are shown in Appendix [A]s Table 8 This is evidence against the Only IID Accuracy Matters hypothesis and is evidence for the Larger Models, Self-Attention, Diverse Data Augmentation, Pretraining, and Texture Bias hypotheses.
Taori et al. (2200) remind us that ImageNet-C uses various synthetic corruptions and suggest that they divoreed from real-world robustness. Real-world robustness requires generalizing naturally occurring corruptions such as snow, fog, blur, low-lighting noise, and so on, but is an open question whether ImageNet-CC simulated corruptions meaningfully approximate real-worke corruptions.
For our results analysis, we collect small dataset of 1,000 real-world blurry images and find that ImageNet-C can track robustness real-world corruptions. We collect the "Real Blurry Images" dataset with Flickr and query ImageNet object class names concatenated with the word "blurry." Examples are in Figure 5 We then evaluate various models on real-world blurry images and find that all the robustness interventions that help with ImageNet-C also help with real-world blurry images. Hence ImageNet-C can track performance on real-world corruptions. Morcover, DeepAugment+AugMix has the lowest error rate Real Blurry Images, which again contradicts the Synthetic + Real hypothesis. The upshot is that ImageNet-C is a controlled and systematic proxy for real-world robustness.
We collect 1,000 blurry images to see whether improvements on ImageNet-C's simulated blurs correspond improvements real-world blurry images. Each image belongs to ImageNet.
8

---

Under review as a conference paper at ICLR 2021
Hypothesis
ImageNet-C
Real Blurry Images
ImageNet-R
DFR
SVSF
Larger Models
Self-Altention
+
+
Diverse Data Augmentation
Pretraining
Table 4: A highly simplified account of each hypothesis when tested against different datasets. Evidence for is denoted "+", and "_" denntes absence of evidence or evidence against.
class. Results from Table 5 show that Larger Models, Self-Attention, Diverse Data Augmentation, Pretraining all help, just like ImageNet-C. Here DeepAugmenttAugiix attains state-of-the-ar These results suggest ImageNet-C's simulated corruptions track real-world corruptions. In hindsight, this is expected since various computer vision problems have used synthetic corruptions as proxies for real-world corruptions, for decades. In short, ImageNet-C is diverse and systematic benchmark that is correlated with improvements on real-world corruptions.
6 CONCLUSION
In this paper we introduced three new benchmarks, ImageNet-Rendition DeepFashion Remixed, and StreetView StoreFronts. With these benchmarks, thoroughly tested seven robustness hypotheses- four about methods for robustness, and three about the nature of robustness.
Let us consider the first four hypotheses, using the new information from ImageNet-C and our three new benchmarks. The Larger Models hypothesis was supported with ImageNet-C and ImageNet-R, but not with DFR. While Self-Attention noticeably helped ImageNet-C, did not help with ImageNet- R and DFR. Diverse Data Augmentation was ineffective for SVSF and DFR, but it greatly improved ImageNet-C and ImageNet-R accuracy. Pretraining greatly helped with ImageNet-C C but hardly helped with DFR and ImageNet-R. This is summarized in Table E It was not obvious priori that synthetic Diverse Data Augmentation could improve ImageNet-R accuracy, nor did previous research suggest that Pretraining would sometimes be ineffective. While no single method consistently helped across all distribution shifts, some helped more than others.
Our analysis of these four hypotheses have implications for the remaining three hypotheses. Re- garding Texture Bias, ImageNet-R shows that networks do not generalize well to renditions (which have different textures), but that diverse data augmentation (which often distorts textures) can re- cover accuracy. More generally, larger models and diverse data augmentation consistently helped on Net-e, Tetge and Blurry Images, suggesting that these two interventions reduce texture bias. However, these methods helped little for geographic shifts, showing that there is more robustness than texture bias alone. Regarding Only IID Accuracy Matters, while IID accuracy is a strong predictor of OOD accuracy, it is not Pociiiive-Tablee A shows that many methods improve robustness across multiple distribution shifts, and recent experiments in NLP provide further coun- terexamples (Hendrycks et al., 2020a). Finally, Synthetic + Real has clear counterexamples given that Augment greatly increases accuracy on ImageNet-R and Real Blurry Images. In summary, some previous hypotheses are implausible, and the Texture Bias hypothesis has the most support.
Our seven hypotheses presented several conflicting accounts of robustness. What led to this conflict? We suspect it is because robustness is not one scalar like accuracy. The research community is reasonable in judging IID accuracy with a univariate metric like ImageNet classification accuracy, as models with higher ImageNet accuracy reliably have better fine-tuned classification accuracy other tasks (Kornblith et al. 2018). In contrast, we argue it is too simplistic to judge OOD accuracy with univariate metric like, say, ImageNetV2 or ImageNet-C accuracy. Instead we hypothesize that robustness is multivariate. This Multivariate hypothesis means that there is not a single scalar model property that wholly governs natural model robustness.
If robustness has many faces, future work should evaluate robustness using many distribution shifts; for example, ImageNet models should at least be tested against ImageNet-C and ImageNet-R. Future work could further characterize the space of distribution shifts. However, due to this paper, there are now more out-o--distribution robustness datasets than there are published robustness methods. Hence the research community should prioritize creating new robustness methods. If our Multivariate hypothesis is true, research should shift toward using multiple tests to develop models that are both robust and safe.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Fruch, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. Google street view: Capturing the world at street level. Computer, 43(6))32-38, 2010.
Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboon- suk, and Laura M Vardoulakis. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 2020.
Irving Biederman and Ginny Ju. Surface versus edge-based determinants of visual recognition. Cognitive psychology, 20(1):38-64 1988.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. AutoAugment: Learning augmentation policies from data. CVPR, 2018.
Jia Deng. Large scale visual recognition. Technical report, PRINCETON UNIV NJ DEPT OF
COMPUTER SCIENCE, 2012.
Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recogni- tion performance under visual distortions. In 2017 26th international conference computer communication and networks (ICCCN), pages 1-7. IEFE, 2017.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras. Jacob Steinhardt, and Alek- sander Madry. Identifying statistical bias dataset replication. JCML, 2020.
Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin- Yu Zhang, Ming-Hsuan Yang, and Philip H.S. Torr. Res2net: new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019a. ISSN 2019 2938758
doi: 10. 1w1.://aa..201..2938888
URL
Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xinyu Zhang, Ming Hsuan Yang, and Philip H. S. Torr. Res2net: new multi-scas scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 2019b.
Yuying Ge, Ruimao Zhang, Xiaogang Wang. Xiaoou Tang, and Ping Luo. Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3337-5345, 2019.
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. NewrIPS, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge. Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR, 2019.
Robert Geirhos, Hom--enriik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. corr abs/1511.00338. (2015), 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR. 2019.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019a. 9a.
10

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. ArXiv, abs/1907.07174, 2019b.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. ACL, 2020a.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR, 2020b.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, 2018.
Shoji Itakura. Recognition of line-drawing representations by chimpanzee (pan troglodytes). The Journal of General Psychology, 12113)189-197, July 1994, doi: 10. 10.0000000221 3099, 1994. 9921195.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXiv:1912.11370, 2019.
Simon Komblith, Jonathon Shlens, and Quoc V. Le. Do better ImageNet models transfer better? CoRR, abs/1805.08974 2018.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In ICLR, 2020.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Dogus Cubuk. Improv- ing robustness without sacrificing accuracy with patch Gaussian augmentation. arXiv preprint arXiv:1906.02611, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1rxivv/776.0060 070600603, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri abd Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. ECCV, 2018.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. arXivv, 2015.
A. Emin Orhan. Robustness properties of facebook's ResNeXt WSL models. ArXiv, Abs/1997.07644 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? ArXiv, 1081, 2019,
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image corruptions by playing the game of noise. arXiv preprint arXiv:2001.06057 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV). 115 13)221 2015. doi: 10.1007/s| 1233-015-8811
Masayuki Tanaka. Recognition of pictorial representations by chimpanzees (pan troglodytes). Animal Cognition, 169-99169-179, December 2006. doi: 10.1077/1100711-00-00666
11

---

Under review as a conference paper at ICLR 2021
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. When robustmess doesn't t promote robustness: Symthetic vs. natural distribution shifts on imagenet, 2020.
IyrFvH
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXiv preprini arXiv: 170::003330 2017.
Haohan Wang. Songwei Ge, Eric P. Xing, and Zachary C. Lipton. Learning robust global representa- tions by penalizing local predictive power, 2019.
Haotao Wang, Tianlong Chen, Zhangyang Wang. and Kede Ma. am going mad: Maximum discrep- ancy competition for comparing classifiers adaptively. In International Conference on Learning Representations, 2020. URL
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.0300 2020.
Sanghyun Woo, Jongchan Park, Youn- Lee, and So Kweon, Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19, 2018.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International Conference Learning Representations, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. 2016. arXiv preprint arXiv: 1611005311 2016.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D Cubuk, and Justin Gilmer. A Fourier perspective on model robustness computer vision. arXiv preprint arXiv: 1eee:0906000998 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. /CCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X1v:1770.09412, 2017.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. arXiv preprint 1XXx:1770.004000 2017.
12

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL RESULTS
ImageNet-R. Expanded ImageNet-R results are in Table
WSL pretraining on Instagram images appears to yield dramatic improvements on ImageNet-R, but the authors note the prevalence of artistic renditions of object classes on the Instagram platform. While ImageNet's data collec- tion process actively excluded renditions, we do not have reason believe the Instagram dataset excluded rendi-
The Effect of Model Size on ImageNet-R R Error Rates
Baseline
Larger Model
ImageNet- Error (%)
tions. On ResNeXt-101 32 x model, WSL pretraining improves ImageNet-R performance by a massive 37.5% from 57.5% top-1 error to 24.2%. Ultimately, without examining the training images we are unable to determine whether ImageNet-R represents actual distribution shift to the Instagram WSL models. However, we also ob- serve that with greater controls, that is with ImageNet- 21K pre-training, pretraining hardly helped ImageNet-R performance, so it is not clear that more pretraining data improves ImageNet-R performance.
60
55
ResNet
DPN
ResNeXt
Figure 6: Larger models improve ro- bustness on ImageNet-R. The baseline models are ResNet-50, DPN-68, and ResNeXt-50 (32 4d). The larger models are ResNet-152, DPN-98, and ResNeXt-101 (32 8d). The baseline
Increasing model size appears automatically improve ImageNet-R performance, as shown in Figure A ResNet- 50 (25.5M parameters) has 63.9% error, while ResNet- 152 (60M) has 58.7% error. ResNeXt-50 4d (25.0M) attains 62.3% error and ResNeXt-110 32x8d (88M) at- tains 57.5% error.
ResNeXt has a 7.1% ImageNet error rate, while the large has 6.2% error rate.
We also tested whether model size improves performance on ImageNet-C for even larger models. With a different codebase, we trained ResNet-50, ResNet-152, and ResNet-500 models which achieved 80.6, 74.0, and 68.5 mCE respectively.
ImageNet-C. Expanded ImageNet-C results are Table g
Expanded comparisons between ImageNet-C and Real Blurry Images is in Table
Network
Defocus Glass
Motion
Zoom
ImageNet-C Blur Mean
Real Blurry
Blur
Blur
Blur
Blur
Images
ResNet-50
61
73
61
64
65
58.7
+ ImageNet-21K Pretraining
56
69
53
59
59
54.8
+ CBAM (Self-Attentoo))
69
56
62
56.5
tx Adversarial Training
80
71
72
71
74
71.6
+ Speckle Noise
68
60
64
62
56.9
Style Transfer
57
68
55
64
61
56.7
+ AugMix
65
46
54
54.4
DeepAugment
48
60
51
61
55
54.2
DeepAugmeentAuugMis
41
53
39
48
45
51.7
ResNet- 152 (Larger Models)
67
81
66
74
58
54.3
Table 5: ImageNet-CC Blurs (Defocus, Glass, Motion, Zoom) vs Real Blurry Images. All values are emro rates and percentages. The rank orderings of the models on Real Blurry Images are similar to the rank orderings for "ImageNet-C Blur Mean," so ImageNet-C's simulated blurs track real-world blur performance.
ImageNet-A. ImageNet-A (Hendrycks et al. 2019b) is an adversarially filtered test set and is constructed based on existing model weaknesses (see (Wang et al., 2020) for another robustness dataset algorithmically determined by model weaknesses). This dataset contains examples that are difficult for ResNet-50 to classify, so examples solvable by simple spurious cues are are especially infrequent in this dataset. Results are in Table Notice Res2Net architectures (Gao al.) 2019b) can greatly improve accuracy. Results also show that Larger Models, Self-Artention, and Pretraining help, while Diverse Data Augmentation usually does not help substantially.
13

---

Under review as a conference paper at ICLR 2021
Implications for the Four Method Hypotheses.
The Larger Models hypothesis has support with ImageNet-C (+), ImageNet-A (+), ImageNet-R (+), yet does not markedly improve DFR (-) performance.
The Self-Artention hypothesis has support with ImageNet-C (+), ImageNer-A (+), yet does not help
ImageNet-R (-) and DFR (-) performance.
The Diverse Data Augmentation hypothesis has support with ImageNett-C (+), ImageNet-R (+), yet does not markedly improve ImageNet-A (-), DFR(-), nor SVSF (-) performance.
The Pretraining hypothesis has support with ImageNet-C (+), ImageNet-A (+), yet does not markedly
improve DFR (-) nor ImageNet-R (-) performance.
Hypothesis
ImageNet-C
ImageNer-A
ImageNet-R
DFR
SVSF
Larger Models
Self-Attention
Diverse Data Augmentation Pretraining
Table A highly simplified account of each hypothesis when tested against different datasets. This table includes ImageNet-A results.
14

---

Under review as a conference paper at ICLR 2021
ImageNet-200) (%)
ImageNet-R (%)
Gap
ResNet-50 (He et al. 2015)
7.9
63.9
56.0
ImageNet-2TK Pretraiming (10x data)
7.0
62.8
55.8
CBAM (Self-Artention)
7.0
63.2
56.2
fo Adversarial Training
25.1
68.6
43.5
Speckle Noise
8.1
62.1
54.0
Style Transfer
8.9
58.5
49.6
AugMix
7.1
58,9
51,8
Degpentment
7.5
57.8
50.3
DeepAugment + AugMix
8.0
53.2
45.2
ResNet-101 (Larger Models)
7.1
60.7
53.6
SE (Self-Attention)
6.7
61.0
54.3
ResNet- 152 (Larger Models)
6.8
58.7
51.9
SE (Self-Attention)
6.6
60.0
53.4
ResNeXt-101 32x4d (Larger Models)
6.8
58.0
51.2
SE (Self-Attention)
5.9
59.6
53.7
ResNeXt-101 8d (Larger Models)
6.2
57.5
51.3
WSL Pretraining (1000x data)
4.1
24.2
20.1
DeepAugment + AugMixx
6.1
47.9
41.8
Table ImageNet-200 and ImageNet- Rendiition error rates. ImageNet-21K and WSL Pretraining test the Pretraining hypothesis, and here pretraining gives mixed benefits. CBAM and SE test the Self-Attention hy hypothesis, and these hurt robustness. ResNet-152 and ResNeXt-101 32 test the Larger Models hypothesis, and these help. Other methods augment data, and Style Transfer, AugMix, and DeepAugment provide support for the Diverse Data Augmentation hypothesis.
Noise Geuss, Shot Impulse Defocus Glass Motion Zoom Snow
Weather Frost Fog Bright Contrast Elastic Pixel JPBG
Digital
Clean
mCE
ResNer-50
23.9
76.7
80
83
75
89
78
80
78
75
66
57
71
85
77
77
ImageNet Pretraining
53
SE (Self-Antention)
22.4
68.2
63
66
66
71
82
67
74
74
72
64
55
71
73
60
67
CBAM (Self-Atention)
22.4
67
74
71
72
54
79
+ Eo Adversarial Training
46.2
94.0
91
95
97
86
92
88
93
99
118
104
111
90
72
81
+ Speckle Noise
24.2
68.3
51
55
70
77
80
76
71
66
57
82
72
69
Style Transfer
25.4
66
68
70
82
69
71
58
78
70
+ AugMix
22.5
65,3
67
66
68
64
79
59
64
69
68
65
54
57
74
60
66
DeepAugment
DeepAugment AugMix
24.2
53,6
46
45
44
50
64
50
61
58
57
54
52
48
71
43
61
ResNet-152 (Larger Models)
21.7
69.3
73
76
67
66
74
68
62
51
67
76
69
65
ResNeXi-101 (Larger Models)
20.7
50
+ WSL Pretraining (1000x data)
17.8
51.7
49
50
51
53
72
55
63
53
51
42
37
41
67
40
51
DeepAngment AugMix
20.1
36
Table 8: Clean Error, Corruption Error (CE), and mean CE (mCE) values for various models and training methods on ImageNet-C. The mCE value is computed by averaging across all 15 CE values. CE value greater than 100 (e.g. adversarial training on contrast) denotes worse performance than AlexNet. DeepAugment+AugMixx improves robustness by over 23 mCE.
15

---

Under review as a conference paper at ICLR 2021
ImageNet-A (%)
ResNet-50
2.2
ImageNet-21K Pretraining (10x data)
11.4
+ Squeere-and-Fxcitaaiin (Self-Artention)
6.2
CBAM (Self-Attention)
6.9
+ foo Adversarial Training
1.7
Style Transfer
2.0
+ AugMix
3,8
+ Dugpentment
3,5
DeepAugment AugMix
3.9
ResNet-152 (Larger Models)
6.1
http:://22-suueee----dde-eeeeaee (Self-Artention)
9.4
Res2Net-50 vlb
14.6
Res2Net-152 vlb (Larger Models)
22.4
ResNeXt-101 (32 x Sd) (Larger Models)
10.2
+ WSL Pretraining (1000x data)
45.4
DeepAugment AugMix
11.5
Table ImageNet-A top-1 accuracy.
B FURTHER DATASET DESCRIPTIONS
ImageNet-R Classes. follows.
The 200 ImageNet classes and their WordNet IDs in ImageNet-R are as
Goldfish, great white shark, hammerhead, stingray, hen, ostrich, goldfinch, junco, bald eagle, vulture, newt, axolotl, tree frog, iguana, African chameleon, cobra, scorpion, tarantula, centipede, peacock, goose, black swan, koala, jellyfish, snail, egret, pelican, king penguin, grey whale, tzu, afghan hound, basset hound, beagle, weimaraner, yorkshire terrier, boston terrier, golden retriever, labrador retriever, cocker spaniels, collie, german shepherd dog, boxer, french buildog, pomeranian, chow chow, pembroke welsh corgi. toy poodle, standard poodle, timber wolf, hyena, red fox, tabby cat, leopard, snow leopard, lion, tiger, cheetah, polar bear, meerkat, ladybug, fly, bee, ant, grasshopper, cockroach, mantis, dragon- fly. monarch butterfly, starfish, wood rabbit, porcupine, fox squirrel, beaver, guinea pig. zebra, pig. hippopotamus, bison, gazelle, llama, skunk, badger, orangutan, gorilla, chimpanzee, gibbon. baboon, panda, cel, clown fish, dion, ambulance, assault rifle, backpack, barn, wheelbarrow, basketball, bathtub, lighthouse, beer glass, binoculars, birdhouse, bow tie, broom, bucket, cauldron, candle. cannon, canoe, carousel, castle, mobile phone, cowboy hat, electric gui- tar. fire engine, flute, gasmask, grand piano, guillotine, hammer, harmonica, harp, hatchet, jeep, joystick, lab coat, parachute, pickup truck, pirate ship, bus, schooner, shield, soccer ball, submarine, tank, tennis ball, tractor, trombone, vase, violin, military aircraft, bottle, ice cream, hagel, pretzel, cheeseburger, hotdog, cabbage, broccoli, cucum- ber, bell pepper, mushroom, Granny Smith, strawberry, lemon, pineapple, banana, pomegranate, pizza, burrito, espresso, volcano, baseball player, scuba diver, acorn.
lorikeet, hummingbird, toucan, duck, lobster, hermit crab, flamingo, american killer whale, bloodhound, italian greyhound, whippet, west highland white terrier, border collic, rottweiler, husky, dalmatian, pug,
lion, chihuahua, shih
scottish terrier,
saint bernard,
puffer fish, accor-
lawn mower, lipstick, mailbox, missile, mitten, revolver, rugby ball, sandal, saxophone, school space shuttle. spider web, steam locomotive, scarf,
wine
n01443537,
n01484850,
n01494475,
n0149804,,
n01514859,
n01518878,
n01531178,
n01534433,
n01614925,
n01616318,
n01630670,
n01632777.
n01644373,
n01677366.
n01694178,
n01748264,
n01770393.
n01774750,
n01784675.
n01806143,
n01820546.
n01833805,
n01843383,
n01847000,
n01855672,
n01860187.
n01882714,
n01910747.
n019 001944900,
n01983481,
n01986214,
n02007558,
n02009912,
n02051845
n02056570.
16

---

Under review as a conference paper at ICLR 2021
n02066245,
n02071294,
n02077923,
n02085620,
n02086240,
n02088094,
n02088238,
n02088364,
n02088466,
n02091032,
n02091134,
n02092339.
n02094433,
n02096585,
n02097298,
n02098286,
n02099601,
n02099712,
n02102318,
n02106030,
n02106166,
n02106550,
n02106662,
n02108089,
n02108915,
n02109525,
n02110185,
n021 10341,
n021 10958,
n02112018. 88,
n02112137,
n02113023,
n02113624,
n02113799,
n021 14367,
n02117135.
102119022,
n02123045,
n02128385,
n02128757,
n02129165,
n02129604,
n02130308,
n02134084,
n02138441,
n02165456,
n02190166.
n02206856,
n02219486,
n02226429,
n02233338,
n02236044,
n02268443,
n02279972.
n02317335,
n02325366,
n02346627,
n02356798,
n02363005,
n02364673,
n02391049,
n02395406,
n02398521,
102410509,
n02423022,
n02437616,
n02445715,
n02447366,
n02480495,
n02480855,
n0248 112481822 1823,
n02483362,
n02486410,
n02510455,
n02526121,
n02607072.
n02655020,
n02672831, I,
002701002,
n02749479,
n02769748,
n02793495,
m02797295,
n02802426.
n02808440,
n02823750,
n02841315,
n02843684,
n02883205,
n02906734,
n02909870,
n02939185,
n02948072.
n02950826,
n02951358,
n02966193,
n02980441,
n02992529,
703124170,
n03272010,
n03345487,
n03372029,
n03424325,
n03452741,
n03467068,
n03481172.
n03494278.
n03495258,
n03498962,
n03594945,
n03602883,
n03630383,
n03649909,
n03676483,
n03710193,
n03773504,
n03775071,
n03888257,
n03930630,
n03947888,
n04086273,
104118538,
n04133789,
n04141076.
n04146614,
n04147183,
n04192698.
n04254680,
n04266014,
n04275548,
n04310018,
n04325704,
n04347754,
n04389033,
n04409515,
n04465501,
n04487394,
n04522168,
n04536866,
n04552348,
n04591713,
n07614500,
n07693725,
n07695742,
n07697313,
n07697537.
n07714571,
n07714990,
n07718472,
n07720875,
n07734744,
n07742313,
n07745940,
n07749582,
n07753275,
n07753592,
n07768694,
n07873807,
n07880968,
n07920052,
n09472597,
n09835506,
n10565667,
n12267677.
SVSF The classes
auto shop
dentist
hotel
bakery
discount store
liquor store
bank
dry cleaner
pharmacy
beauty salon
furniture store
religious institution
car dealer
gas station
car wash
gym
storage facility
cell phone store
hardware store
veterinary care.
DeepFashion Remixed, The classes
short sleeve top
sling
long sleep dress
long sleeve top
shorts
short sleeve outerwear
trousers
vest dress
long sleeve outerwear
skirt
vest
short sleeve dress
sling dress.
Size (small, moderate, or large) defines how much of the image the article of clothing takes up.
Occlusion (slight, medium, heavy) defines the degree which the object occluded from the camera. Viewpoint (front, sidc/back, not worn) defines the camera position relative to the article of clothing. Zoom (no zoom, medium, or large) defines how much camera zoom was used to take the picture.
17

---

Under review as a conference paper at ICLR 2021
Represented Distribution Shifts
ImageNet-Rendee Renditions
artistic renditions (cartoons, graffiti, embroidery, graphics, origami,
paintings, sculptures, sketches, tattoos, loys....)
DeepFashion Remixed
occlusion, size, viewpoint, zoom camera, capture year, country
StreetView StoreFronts
Table 10: Various distribution shifts represented our three new benchmarks. ImageNet-Rendition is new test set for ImageNet trained models measuring robustness to various object renditions. DeepFashion Remixed and StreetView StoreFronts each contain a training set and multiple test sets capturing variety of distribution shifts.
Training set
Testing images
ImageNet-R 1281167
30000
DFR
48000
42640, 7440, 28160, 10360, 480, 11040, 10520, 10640 10000, 10000, 10000, 8 8195, 9788
SVSF
200000
Table 11: Number of images in each training and test set. ImageNet-R training set refers to the ILSVRC 2012 training set (Deng et al. 2009). DeepFashion Remixed test sets are: in-distribution, occlusion none/slight, occlusion heavy, size small, size large, viewpoint- frontal, viewpoint- not-worn, zoom-in medium, zoom-in large. StreetView StoreFronts test sets are: in-distribution, capture year 2018, capture year 2017, camera system new, country France.
C DEEPAUGMENT DETAILS
Pseudocode. Below is Pythonic pseudocode for DeepAugment. The basic structure of DeepAug- ment is agnostic to the backbone network used, but specifies such which layers chosen for various transforms may vary as the backbone architecture varies. We do not need to train many different mage-to-image models get diverse distortions (Zhang al. 2018 Lee al. 2020). We only use two existing models, the EDSR super-resolution model (Lim et al. 2017) and the CAE image compression model (Theis al. 2017). See full code for such details.
At a high level, DuepAngment processes each image with an image-to-image network. The image-to- image network's weights and feedforward activations distorted with each pass. The distortion is made possible by, for example, negating the network's weights and applying dropout to the feedforward activations. These modifications were not carefully chosen and demonstrate the utility of mixing together diverse operations without tuning. The resulting image is distorted and saved. This process generates an augmented dataset.
18

---

Under review as a conference paper at ICLR 2021
def main()
net aet ghtssssseesssssssseossssssssss (eee (deeppuireen__eeeeeeetotot
EDSR, CAE, for image in dataset: May be the ImageNet training set
if np.random mormor () 0.05: I Arbitrary refresh prob
net apply. eet.epply_weightt weights httpr/uuummettgggettttrrttt
new new_image net net.
(image)
def dccpaugment_geetNeteee
weights [oae__clca__eeghtt 10 weight_distortionn
sample__wcighh__deetrrtttt.. ()
11
for d in weights apply_de stttrticnnd, (d, weights)
weights
14
15
def sample_weight distortiosst ()
distertiens =
negate_weights, zero_weights,
18
19
flip_tranngos__eiihhtt
20
23
return randen_subsed (distorticns)
24
25
def dlup
distortions :):
distortions
gelu,
28
negate_si prrate_siggal_randoonnnags
29
flip_signas
32
33
return random_subsed (distorticns)
34
class Network ()
37
def apply_wer wei ghts whithss (weights):
38
# Apply given weight tensors to network
Clean forward pass. Compare to deepAugment_torwareee forwardPass ()
41
def clean foreardassssssss (X):
42
X blockl (X)
43
X network.block) (X)
network.blockN (X)
46
return
47
48
Our forward pass. Compare to coraa_sorsa ()
def cecpaugment_forwardeee (X):
50
Returns list distertions, each of which
51
will be applied at different layer.
52
signal_distoon dignal_distortions
network. (X)
55
apply_layee listottto (,, signal
56
X = network.block2 (X)
57
apply_layee_22dsssorrtioona/
signal_distortione)
apply_llyyr_---dditttrrreees signal_distortionss))
60
network.bbookk///
61
agply_layer__.dssttttiiooom (8, signal_distortions)
return
19

---

Under review as a conference paper at ICLR 2021
Ablations. We rum ablations DeepAugment to understand the contributions from the EDSR and CAE models independently. Table contains results of these experiments on ImageNet-R and Table ( contaiis results of these experiments ImageNet-C. In both tables, "DeepAugment (EDSR)" and "Deeptugment (CAE)" refer to experiments where we only use single extra augmented training set (+ the standard training set), and train those images.
Noise
Blur
Weather
Digital
Clean
mCCE
Giuss, Shot Impulsc] Detocus Glass Motion Zoom Snow Frost
Fog Bright Contrast Elastic Pixel JPBG
ResNet-50
23.9
78
75
77
+ DeepAugment (EDSR)
23.5
64.0
56
57
54
64
77
71
78
68
64
64
55
64
78
46
67
DeepAugment (CAE)
23.2
67.0
73
66
52
78
DeepAngment (Roth)
23.3
60.4
65
60
61
76
67
Table 12: Clean Error, Corruption Error (CE), and mean CE (mCE) values for DeepAugment ablations on ImageNet-C. The mCE value is computed by averaging across all 15 CE values.
Noise2Net. We show that untrained, randomly sampled neural networks can provide useful deep augmentations, highlighting the efficacy of the DeepAugment approach. While in the main paper we use EDSR and CAE to create DeepAugment augmentations, in this section we explore the use of randomly initialized image-to-image networks to generate diverse image augmentations. We propose a DeepAugment method, Noise2Net.
In Noise2Net, the architecture and weights are randomly sampled. Noise2Net is the composition of several residual blocks: Bzock(z) = x + E fe(z). where θ is randomly initialized and æ is a parameter that controls the strength of the augmentation. For all our experiments, blocks (Gao et al. (2019)) and é ~ (0.375, 0.75). The weights of Noise2Net are resampled at every minibatch, and the dilation and kernel sizes of all the convolutions used in Noise2Net are randomly sampled every epoch. Hence Noise2Net augments an image to an augmented image by processing the image through randomly sampled network with random weights.
use 4 Res2Net
Recall that in the case of EDSR and CAE, we used networks to generate a static dataset, and then we trained normally on that static dataset. This setup could be done on- on-the-fly. That because we fed in one example at a time with EDSR and CAE. If we pass the entire minibatch through EDSR or CAE, we will end up applying the same augmentation all images in the minibatch, reducing stochasticity and augmentation diversity. In contrast, Noise2Net enables us to process batches of images on-the-fly fly and obviates the need for creating a static augmented dataset.
In Noise2Net, each example is processed differently in parallel, tations in real-time. To make this possible, we use grouped convolutions. A grouped convolution with number of groups N will take set of kN channels input, and apply N independent convolutions on channels (1...., k3 ......,,.. {k +1,..... 1,..., 2k)..... {(N - 1)k + 1...... Nk). Given a minibatch of size B. we apply randomly initialized grouped convolution with N B groups in order to apply a different random convolutional filter to each element in the batch in a single forward pass. By replacing all the convolutions in each Res2Net block with a grouped convolution and randomly initializing network weights, we arrive at Noise2Net, a variant of DeepAugment. See Figure for high-level overview of Noise2Net and Figure for sample outputs.
generate more diverse augmen-
We evaluate the Noise2Net variant of DeepAugment on ImageNet-R. Table 1 shows that it out- performs the EDSR and CAE variants of DeepAugment, even though the network architecture is randomly sampled, its weights are random, and the network is not trained. This demonstrates the flexibility of the DeepAugment approach. Below Pythonic pseudocode for training a classifier using the Noise2Net variant of DeepAngment.
20

---

Under review as a conference paper at ICLR 2021
fg(x)
(1,3xB,8,N)
(1,3xB,8,0)
Repeated Times
(,,,,
(B,3,
Figure 7: Parallel augmentation with Noise2Ner. We collapse batches to the channel dimension to ensure that different transformations applied to each image in the batch. Feeding images into the network in the standard way would result in the same augmentation being applied to each image, which is undesirable. The function (e) is a Res2Net block with all convolutions replaced with grouped convolutions.
ImageNet-200) (%)
ImageNet-R (%
Gap
ResNet-50
7.9
63.9
56.0
DeeeAAument (EDSR)
7.9
60.3
55.1
Duepentment (CAF)
7.6
58.5
50.9
DeepAugment (EDSR + CAB)
7.5
57.8
50.3
DeeepAgmeent (Noise2Net)
7.2
57.6
50.4
DeepAugment (All 3)
7.4
56.0
48.6
Table 13: DeepAugment ablations on ImageNet-200 and ImageNet- mmaditet-Rennitions.
def train_cne_epoch (classifier, batch_size, dataloader)
noisc2net Noise2Net baatch_sizo-batch__see. for batch, target datalcader:
noise2net. reloots_etddddddeeihtt.t. neise2net.co lereeettsst__esslooooeoooooo (randem, uni form 30,375, 0.75)) logits model (noi forwar (batch)) ... Calculate loss and backrop
def train():
for epoch in range (epochs) train_one_epoch (classifier, batch_size, datalcader)
11
12
13
class Noise2Net:
def
_ (self, batch_size=5):
15
gelf felfkblock1 Res2NetBlock (ttreicccccttttchsseeeeeeeee size=batch__sizee
16
sclf.block2 Res2NetBlock battch_size-baatee.....
17
self.block3 Res2NetBlock (batchh_sseoo-batccc.....
18
self.block4 = Res2NetBlock htt.eealccckaaccc..sseeeaattta..
def relcad @elcad_weights (s)lf): # Reload Network parameters
21
22
def set apsilon(self, new new_eps):
self epsilon new_eps
25
26
def forward x):
27
x = x self.blocki (x) self.epsilo
self.block2 (x) self. epsiien
self.block3 (x) self.epsilon
30
x x self.block4 (x) self.epsilon
31
return x
21

---

Under review as a conference paper at ICLR 2021
0.00
==
0.75
Figure 8: Example outputs of Noise2Net for different values of Note 0 is the original image.
22


Reviews:
[{"review_id": "ua_AWmH0gFe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper contributes three new datasets to evaluating seven robustness hypotheses. \n\nStrengths:\n\n1.The introduced three databases would be valuable to probe the generalization of classifiers in the real world.\n\n2. The deep augmentation method seems neat. It is a unified method to produce a variety of perturbations (despite in a less controllable way). And the performance gap on the proposed databases is noticeably reduced by DeepAugment. \n\nWeaknesses:\n\n1. The authors may clearly state how they collect human labels for these datasets. In practice, collecting 1 out of 200 possible labels in ImageNet-R is not trivial.\n\nOther comments:\n1. What is the intended solution for solving the StreetView StoreFronts dataset? Is it the structure of the store or just text in the image?\n2. As shown in Fig. 3, DeepAugment seems to distort the input images. Is there a way to systematically control the distortion levels? Does the architecture of the autoencoder matter?\n3. There is another line of research to construct small but adaptive datasets to probe the generalizability of classifiers [C1], published in ICLR2020. The authors may want to be aware of it.\n\n[R1] I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively, https://openreview.net/forum?id=rJehNT4YPr", "extra_scores": {}, "extra_text": {}}, {"review_id": "cF9zkQx9lWI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper investigates the robustness problem of computer vision model. To study the model robustness in a controlled setting, the author introduces three new robustness benchmarks: ImageNet-R, StreetView StoreFronts and DeepFashion Remixed. Each of them address different aspects of distribution drift in the real world. The author evaluates seven popular hypotheses on model robustness in the community on the three new datasets and has found counter-example for most of them. Based on those new results, the author concluded that model robustness problem is multi-variate in nature: no single solution could handle all aspects yet. And future work should be tested on multiple datasets to prove robustness. Moreover, the author also proposes a new data augmentation method using perturbed image-to-image deep learning model to generate visually diverse augmentations.\n\n\nSignificance: This paper is a solid work on the robustness problem. It systematically evaluated common hypotheses and successfully found counter-example on all of them except Texture Bias. The analysis is insightful and supported by the experimental results. The authors also provides three new carefully designed datasets for future work evaluation. While the study of using deep neural network to generate training image is not new, DeepAugmentation is still an innovative and practical way for data augmentation purpose.\n\nQuestion: On DeepFashion Remixed datasets, it seems large zoom has better result than medium zoom. Is there a good explanation for that, considering the original image has no zoom-in? \n\nClarity: The author did a great job on explaining the idea, objective and approach. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9UFNIWsIoUE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a empirical study on the robustness of image classification models to distributions shifts. The authors construct three benchmark datasets that control for effects like artistic renditions of common classes, view-point changes, and geographic shifts (among others). The datasets are then used to test various hypotheses regarding robustness enhancing measures empirically. The authors additionally propose a novel augmentation scheme, that uses deep image processing networks together with random perturbations of their weights to synthesize distorted image samples.\n\n---- Strengths ----\n\nThe paper tackles and important topic. I agree with the authors that robustness is \"multivariate\", i.e. can not be improved by a single factor. The paper makes an effort to disentangle various factors and test them in isolation.\n\nThe paper provides evidence that ImageNet-C can be used to make conclusion about real behavior of models, despite being based on synthetic image transformations.\n\nThe paper is well written and provides comprehensive experiments.\n\n--- Weaknesses ---\n\nThe conclusions that result from the empirical findings are unfortunately not very crisp. Some hypotheses are supported by some datasets, others are not. No clear conclusions that hold across datasets can be drawn. It seems that we can't learn much from the experiments, and that the answer  to the questions \"What improves robustness?\" is still very much \"it depends on what you are testing on\". This is deeply unsatisfying, as true robustness  actually should not depend on the dataset. Table 4 provides a simplified summary of various hypotheses and how they are supported by different datasets. This table doesn't look too surprising: Why should, for example, self-attention help to improve performance on artistic renditions, something that  is very different from a blurred variant of an image? Why would it help classify an image of a pharmacy in France, when the model has only seen pharmacies  in the US (both will look very different, there cannot be a reasonable expectation for such a transfer). On the other hand, data augmentation can strongly abstract away certain image features, it thus is reasonable that it improves performance for ImageNet-R, but is limited for the other two datasets. \n\nThe core issue seems to lie in the construction of the datasets:\n\n1) We know that deep networks have a texture bias, so there is no reasonable expectation for transfer to ImageNet-R. There wouldn't be any expectations to improve this performance for any change, but enlargement of the dataset towards more abstract depictions of the objects. This seems to be confirmed by Table 1, where abstract augmentations (e.g. enlarging the training set with samples that are  in some respect more similar to the test set) clearly improves results, but simple architectural changes or simple augmentations do not help. This would support the hypothesis that to improve robustness, more similar data is necessary (i.e. the training set simply doesn't sufficiently cover the space of images that we expect the model to perform on).\n\n2) The SVSF dataset paints a similar picture. Small shifts (e.g. images taken a year apart) hardly influence performance, whereas a extreme shift (e.g. in location) breaks the model even with the augmentations. This seems reasonable, as the augmentations certainly don't cover the shift that typically happen for building appearances between continents.\n\n3) DFR paints a similar picture: Abstract augmentation slightly helps if it roughly matches the shift. Other simple augmentations or architectural changes do not significantly change results.\n\nMy conclusion from the experiments would be something that is well known:  enlarge the dataset to better cover what you expect your model to do. If this can be done with automatic augmentations  (e.g. for zoom, some augmentations that are closer to artistic renditions) then you can use these augmentations. If not: collect more data. It is thus not clear what the provided analysis provides on top of this. \n\n--- Other ---\n\nSome clarifying questions:\n\n- Why is SVSF limited to augmentations? What does a 30 day retention window mean?\n- Can you clarify you conclusions on DFR? Why don't you see evidence for larger models or pretraining? Both seem to substantially improve performance?\n\n--- Summary ---\n\nI think this work is interesting and is in principle asking the right questions. However, the analysis and conclusions currently do not providing robust and generalizable insights that advance the field.\n\n--- Post rebuttal ---\n\nI'm keeping my initial score. My concern remains (and is apparently reflected by R1): the datasets and results do not allow to draw clear conclusions. The paper overall furthers our understanding on robustness only in a very limited way. The new dataset adds another specialized dataset to the mix. I disagree that the community should first exhaustively and randomly add datasets to the literature without coming up with a definition of robustness or at least try to categorize. The authors in their rebuttal criticize the community that they are looking at robustness and distribution shifts in a too simplistic way, but at the same time the presented work doesn't make an effort to change this either.\n\nTo close the remaining question:\n\n- \"Could you elaborate? We know that humans and primates can generalize to new renditions that they have not seen before (Itakura, 1994; Tanaka, 2006), while some other species cannot. Consequently more than training data matters.\"\n\nWith \"extreme\", I mean distributions shifts that keep semantics, but change appearance strongly. If we go as far looking at biological systems, then yes, it is not only about training data. There is an additional mechanism at play that we don't know and currently can't replicate in ML. Given our current understanding, it is presumptuous to suggest that NN architectures and training approaches as they are covered in this work will be able to do these kinds of generalizations at the level of humans or primates.\n\n- \"The empirical reality does not currently allow for a simple, single-cause characterization of robustness\"\n\nI completely agree to this statement. However, this doesn't mean that characterization of robustness cannot be done by taking into account multiple factors systematically.\n\nI acknowledge that collecting a new dataset is a non-trivial effort and can be useful.  I acknowledge that the paper proposes an additional augmentation technique that seems to improve results in certain cases. All factors together taken together lead to my final score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Go16B-VJl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes three new benchmarks for robustness, named ImageNet-R, StreetView StoreFronts, and DeepFashion Remixed. Also, this paper proposes a new augmentation called DeepAugment.\n\n**Pros**\n\n\\+ A new benchmark could be useful for many researchers in this field.\n\n**Cons**\n\n**[How this paper solve the seven motivations is unclear]**\nTo me, the seven motivations (larger model increases the robustness, self-attention increases the robustness, diverse augmentation increases the robustness, pretrain models increase the robustness, texture bias harms the robustness, IID dataset determines the robustness, synthetic robustness is not helpful for the real-world robustness) *are completely independent to each other*. After I read the paper, it is still remaining as a question of how the seven motivations are related and how they are solved by this paper.\n\nFirst of all, what does \"robustness\" mean in this paper? For instance, adversarial robustness represents the error rate against the worst-case attacks in the L-p ball of the given input. However, in this paper, I feel the terminology \"robustness\" is ill-defined.\n\nSecond, if this paper argues that \"larger model\" or \"self-attention\" increases the \"robustness\", I would expect\n\n- the rigorous definition of the robustness\n- the theoretical guarantee or strong empirical evidence that a larger model or self-attention can increase the robustness against the proposed threat model.\n\nHowever, I cannot find any detail in this paper.\n\nAlso, texture robustness or synthetic robustness is not fully discussed in this paper. Why we have to consider them? And how the proposed benchmarks support the arguments?\n\n\n**[Motivation to a new benchmark is not enough]**\nThere are already many ImageNet benchmarks including ImageNet-C, ImageNet-P [1], ImageNet-A, ImageNet-O [2], ImageNet-V2 [3], clean label [4], stylized ImageNet [5] and other possible benchmarks.\n\n[1] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" ICLR 2019\n[2] Hendrycks, Dan, et al. \"Natural adversarial examples.\" arXiv preprint arXiv:1907.07174 (2019).\n[3] Recht, Benjamin, et al. \"Do imagenet classifiers generalize to imagenet?.\" ICML 2020\n[4] Beyer, Lucas, et al. \"Are we done with ImageNet?.\" arXiv preprint arXiv:2006.07159 (2020).\n[5] Geirhos, Robert, et al. \"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.\" ICLR 2019\n\nI personally cannot find any motivation to use ImageNet-R instead of the above benchmarks to evaluate the robustness.\nEspecially, I feel the first seven motivations are independent to the ImageNet-R.\n\nEven if we consider ImageNet-R as a new ImageNet \"robustness\" benchmark,\nI still wonder why we have to evaluate our models to ImageNet-R instead of ImageNet-C, -P, -A, -V2, and clean labels.\n\nFurthermore, I would like to cite a recent paper on measuring robustness to distribution shift in ImageNet classification (a contemporary work)\n[6] Taori, Rohan, et al. \"Measuring robustness to natural distribution shifts in image classification.\" arXiv preprint arXiv:2007.00644 (2020).\nThis paper shows that natural robustness is completely relying on the true test set accuracy.\nIf we employ a new robustness benchmark, I would expect a rigorous reason why we need a new benchmark.\n\n**[ImageNet-R: Ambiguity on selecting 200 classes and the rendition classes]**\nThe most ambiguous thing in this paper is \"200 sub-classes\" to collect the dataset (as section 3.1). Why the authors use 200 sub-classes instead of the original 1,000 classes?\nHow the authors choose \"200 sub-classes\" from the 1000 classes?\n\nFurthermore, this paper aims to solve the robustness problem (where the ``robustness'' is not defined in this paper). I wonder how the ImageNet-R benchmark can evaluate the robustness of the trained models.\n\nAlso, I wonder how the \"renditions\" are chosen. We always can collect a new ImageNet benchmark by crawling a new dataset from the web.\nIf the renditions in ImageNet-R cannot ensure the whole \"renditions\" in the real-world, we also can suffer from the overfitting issue to the proposed ImageNet-R.\nIt could be problematic because a deep model is known to not be able to generalize to the unseen noises [7]\n\n[7] Geirhos, Robert, et al. \"Generalisation in humans and deep neural networks.\" Advances in neural information processing systems. 2018.\n\n**[Why SVSF and DeepFashion Remixed datasets are required to support the original seven motivations?]**\nAlthough I like to test a new benchmark for testing different methods, it is not clear why SVSF and DeepFashion remixed datasets are required to support the original motivations. Why ImageNet-R is not enough? How ImageNet-R, SVSF, and DeepFashion revisited are related?\nI feel the proposed three benchmarks are not really related.\n\n**[Can not find any criteria to build the proposed dataset]**\nI believe this paper aims to solve a \"robustness\" problem. However, I cannot find any scientific protocol which supports that \"achieving high accuracy on the proposed benchmarks truly solves the robustness problem\". How the datasets are built? How we can trust the benchmark, while other possible benchmarks are not reliable?\n\n**[DeepAugment details are missed in the main paper]**\nIn my opinion, this paper has two contributions (1) a new benchmark, and (2) a new augmentation method to solve (1). However, the augmentation method (DeepAugment) details are not able to understand without reading the appendix.\nEven after I read the appendix, I still cannot understand the motivation of the DeepAugment and the method details.\nWhy do we need to apply layer-wise distortions instead of input-level distortions? How the distortions are chosen? Are the distortions independently chosen from the benchmark distortions? I still have many questions.\nFurthermore, I feel that DeepAugment requires a lot of hyperparameters, especially for choosing the \"distortions\". I wonder how the authors choose the hyperparameters, especially the set of \"distortions\". If the authors directly tune their method on the ImageNet-R, it is not fair and not convincing benchmark to evaluate the robustness.\n\n---\n\n**Final review**\n\nAfter reading the paper, other reviews, and author responses carefully again, I decided to remain on the rejection side because\n\n- I think the proposed dataset does not really guarantee the robustness against \"real-world distribution shifts\" because\n  - This paper did not rigorously define what the \"real-world distribution shifts\" are. In the final response, the authors mentioned that *\"It is clear that temporal, hardware, geographic, and rendition shifts occur in the real world.\"*, but to me, it is not clear whether they are really common and representative in the real-world deployment scenario and really threaten deep models.\n  - Because the real-world distribution shift is not well-defined here, I feel the \"robustness\" is also ill-defined too. According to the author response, robustness is defined as the accuracy gap between \"in-distributed\" samples and \"out-of-distributed\" samples (not critical, but OOD is usually defined as the same data distribution, but unseen class. I think this terminology need to be polished). However, here OOD (distribution shift) is ill-defined, and the robustness test is heavily dependent on the test dataset.\n  - Thus, if we just test the \"robustness against real-world distribution shift\" on the proposed dataset only, it can lead to wrong conclusions, e.g., assume we have a model can be specifically better in a specific shift, e.g., rendition shift, but not generalized to other shifts, then ImageNet-R benchmark cannot measure how this model is vulnerable to the other shifts. It will confuse researchers in this field. Hence, I think this paper needs more justification for the new dataset (e.g., why the chosen shifts? why 200 classes for ImageNet-R? why different three datasets?), and need more human studies (e.g., humans can correctly classify the shifted images and non-shifted images) such as [3, 5, 7, 8].\n- This paper is not clearly presented. After reading the paper, I am still confusing about how to understand the experimental results. To me, the benchmark results cannot answer these questions well. I think R2 has a similar opinion on me in this criterion.\n- It is not mentioned in my previous reviews, so I lower the weights for this part to the final decision, but there are already some datasets benchmarking the dataset distribution shifts, e.g., PACS [9], NICO [10]. It may not be true that this kind of distribution shift is only measurable by the proposed datasets. But, as my first words, I noticed that I did not mention these datasets in my previous reviews, and these datasets will not affect my review a lot.\n  - https://domaingeneralization.github.io/\n  - http://nico.thumedialab.com/\n\n[8] Shankar, Vaishaal, et al. \"Evaluating machine accuracy on imagenet.\" International Conference on Machine Learning. PMLR, 2020.\n[9] Li, Da, et al. \"Deeper, broader and artier domain generalization.\" Proceedings of the IEEE international conference on computer vision. 2017.\n[10] He, Yue, Zheyan Shen, and Peng Cui. \"Towards Non-IID Image Classification: A Dataset and Baselines.\" Pattern Recognition (2020): 107383.\n\nOf course, building a new dataset is a non-trivial effort, and measuring real-world robustness is not an easy task (maybe it even can be an impossible task). However, I think this paper can not clearly present how the proposed benchmark can solve the real-world distribution shifts and how can we move forward in the next directions.\n\nTo sum up, I think this paper is okay, but not enough to be accepted to ICLR main conference paper. However, I will respect all decisions made by AC.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new dataset to evaluate the robustness of image classifiers. The dataset consists of data from three sources: a crowdsourced dataset collected by the authors called ImageNet-Renditions, images from Google street view, and data sampled from DeepFashion2. This new dataset allows the authors to test robustness to different renditions of an object (e.g. artistic depictions of an object category) and robustness to changes in geography and camera type. In addition, they propose a new augmentation strategy called DeepAugment which consists of an encoder/decoder style network that transforms the appearance of the input image by simply applying different random perturbations of the weights of the augment network. Robustness results are presented on the previously described datasets where the proposed augmentation strategy in combination with an existing approach (AugMix) performs best in some cases. However, the results are not convincing and AugMix often outperforms the new method.  

In general, the authors did a good job addressing many of the comments (e.g. they provided more detail about how ImageNet-R was collected), but there were still several lingering concerns. R4 was the most positive about the paper, but unfortunately was one of the least vocal during the discussion. R1 was concerned that the paper did not do a great job of defining what was meant by robustness. This AC doesn't agree fully with their concerns, but does agree that more care could have been taken to position the paper better in light of the existing datasets that are already available (see R1’s comments). As the reviewers and authors note, collecting new datasets is a lot of work so care should be taken to ensure that this is not duplicate effort. The authors addressed these concerns in their response to some extent, but more discussion is needed in the paper. 

There was a lot of discussion between the authors and reviewers about this paper. The new dataset has a lot of merit, but there is some concern that the paper does not do a great job of clearly presenting its findings and conclusions. In addition, the proposed augmentation technique is slightly underwhelming performance wise and not very clearly described in the main paper. R2 sums up the opinion of this AC: “I think this work is interesting and is in principle asking the right questions. However, the analysis and conclusions currently do not provide robust and generalizable insights that advance the field.” There is clearly a lot of promise here, and the current recommendation is a weak reject. The authors are strongly encouraged to take the detailed feedback they have received on board and to revise the paper to further improve it for a future submission. 

Decision: Reject