Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper contributes three new datasets to evaluating seven robustness hypotheses. 

Strengths:

1.The introduced three databases would be valuable to probe the generalization of classifiers in the real world.

2. The deep augmentation method seems neat. It is a unified method to produce a variety of perturbations (despite in a less controllable way). And the performance gap on the proposed databases is noticeably reduced by DeepAugment. 

Weaknesses:

1. The authors may clearly state how they collect human labels for these datasets. In practice, collecting 1 out of 200 possible labels in ImageNet-R is not trivial.

Other comments:
1. What is the intended solution for solving the StreetView StoreFronts dataset? Is it the structure of the store or just text in the image?
2. As shown in Fig. 3, DeepAugment seems to distort the input images. Is there a way to systematically control the distortion levels? Does the architecture of the autoencoder matter?
3. There is another line of research to construct small but adaptive datasets to probe the generalizability of classifiers [C1], published in ICLR2020. The authors may want to be aware of it.

[R1] I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively, https://openreview.net/forum?id=rJehNT4YPr
review_title: An empirical evaluation of natural distributional shifts in image classification

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes three new benchmarks for robustness, named ImageNet-R, StreetView StoreFronts, and DeepFashion Remixed. Also, this paper proposes a new augmentation called DeepAugment.

**Pros**

\+ A new benchmark could be useful for many researchers in this field.

**Cons**

**[How this paper solve the seven motivations is unclear]**
To me, the seven motivations (larger model increases the robustness, self-attention increases the robustness, diverse augmentation increases the robustness, pretrain models increase the robustness, texture bias harms the robustness, IID dataset determines the robustness, synthetic robustness is not helpful for the real-world robustness) *are completely independent to each other*. After I read the paper, it is still remaining as a question of how the seven motivations are related and how they are solved by this paper.

First of all, what does "robustness" mean in this paper? For instance, adversarial robustness represents the error rate against the worst-case attacks in the L-p ball of the given input. However, in this paper, I feel the terminology "robustness" is ill-defined.

Second, if this paper argues that "larger model" or "self-attention" increases the "robustness", I would expect

- the rigorous definition of the robustness
- the theoretical guarantee or strong empirical evidence that a larger model or self-attention can increase the robustness against the proposed threat model.

However, I cannot find any detail in this paper.

Also, texture robustness or synthetic robustness is not fully discussed in this paper. Why we have to consider them? And how the proposed benchmarks support the arguments?


**[Motivation to a new benchmark is not enough]**
There are already many ImageNet benchmarks including ImageNet-C, ImageNet-P [1], ImageNet-A, ImageNet-O [2], ImageNet-V2 [3], clean label [4], stylized ImageNet [5] and other possible benchmarks.

[1] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." ICLR 2019
[2] Hendrycks, Dan, et al. "Natural adversarial examples." arXiv preprint arXiv:1907.07174 (2019).
[3] Recht, Benjamin, et al. "Do imagenet classifiers generalize to imagenet?." ICML 2020
[4] Beyer, Lucas, et al. "Are we done with ImageNet?." arXiv preprint arXiv:2006.07159 (2020).
[5] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." ICLR 2019

I personally cannot find any motivation to use ImageNet-R instead of the above benchmarks to evaluate the robustness.
Especially, I feel the first seven motivations are independent to the ImageNet-R.

Even if we consider ImageNet-R as a new ImageNet "robustness" benchmark,
I still wonder why we have to evaluate our models to ImageNet-R instead of ImageNet-C, -P, -A, -V2, and clean labels.

Furthermore, I would like to cite a recent paper on measuring robustness to distribution shift in ImageNet classification (a contemporary work)
[6] Taori, Rohan, et al. "Measuring robustness to natural distribution shifts in image classification." arXiv preprint arXiv:2007.00644 (2020).
This paper shows that natural robustness is completely relying on the true test set accuracy.
If we employ a new robustness benchmark, I would expect a rigorous reason why we need a new benchmark.

**[ImageNet-R: Ambiguity on selecting 200 classes and the rendition classes]**
The most ambiguous thing in this paper is "200 sub-classes" to collect the dataset (as section 3.1). Why the authors use 200 sub-classes instead of the original 1,000 classes?
How the authors choose "200 sub-classes" from the 1000 classes?

Furthermore, this paper aims to solve the robustness problem (where the ``robustness'' is not defined in this paper). I wonder how the ImageNet-R benchmark can evaluate the robustness of the trained models.

Also, I wonder how the "renditions" are chosen. We always can collect a new ImageNet benchmark by crawling a new dataset from the web.
If the renditions in ImageNet-R cannot ensure the whole "renditions" in the real-world, we also can suffer from the overfitting issue to the proposed ImageNet-R.
It could be problematic because a deep model is known to not be able to generalize to the unseen noises [7]

[7] Geirhos, Robert, et al. "Generalisation in humans and deep neural networks." Advances in neural information processing systems. 2018.

**[Why SVSF and DeepFashion Remixed datasets are required to support the original seven motivations?]**
Although I like to test a new benchmark for testing different methods, it is not clear why SVSF and DeepFashion remixed datasets are required to support the original motivations. Why ImageNet-R is not enough? How ImageNet-R, SVSF, and DeepFashion revisited are related?
I feel the proposed three benchmarks are not really related.

**[Can not find any criteria to build the proposed dataset]**
I believe this paper aims to solve a "robustness" problem. However, I cannot find any scientific protocol which supports that "achieving high accuracy on the proposed benchmarks truly solves the robustness problem". How the datasets are built? How we can trust the benchmark, while other possible benchmarks are not reliable?

**[DeepAugment details are missed in the main paper]**
In my opinion, this paper has two contributions (1) a new benchmark, and (2) a new augmentation method to solve (1). However, the augmentation method (DeepAugment) details are not able to understand without reading the appendix.
Even after I read the appendix, I still cannot understand the motivation of the DeepAugment and the method details.
Why do we need to apply layer-wise distortions instead of input-level distortions? How the distortions are chosen? Are the distortions independently chosen from the benchmark distortions? I still have many questions.
Furthermore, I feel that DeepAugment requires a lot of hyperparameters, especially for choosing the "distortions". I wonder how the authors choose the hyperparameters, especially the set of "distortions". If the authors directly tune their method on the ImageNet-R, it is not fair and not convincing benchmark to evaluate the robustness.

---

**Final review**

After reading the paper, other reviews, and author responses carefully again, I decided to remain on the rejection side because

- I think the proposed dataset does not really guarantee the robustness against "real-world distribution shifts" because
  - This paper did not rigorously define what the "real-world distribution shifts" are. In the final response, the authors mentioned that *"It is clear that temporal, hardware, geographic, and rendition shifts occur in the real world."*, but to me, it is not clear whether they are really common and representative in the real-world deployment scenario and really threaten deep models.
  - Because the real-world distribution shift is not well-defined here, I feel the "robustness" is also ill-defined too. According to the author response, robustness is defined as the accuracy gap between "in-distributed" samples and "out-of-distributed" samples (not critical, but OOD is usually defined as the same data distribution, but unseen class. I think this terminology need to be polished). However, here OOD (distribution shift) is ill-defined, and the robustness test is heavily dependent on the test dataset.
  - Thus, if we just test the "robustness against real-world distribution shift" on the proposed dataset only, it can lead to wrong conclusions, e.g., assume we have a model can be specifically better in a specific shift, e.g., rendition shift, but not generalized to other shifts, then ImageNet-R benchmark cannot measure how this model is vulnerable to the other shifts. It will confuse researchers in this field. Hence, I think this paper needs more justification for the new dataset (e.g., why the chosen shifts? why 200 classes for ImageNet-R? why different three datasets?), and need more human studies (e.g., humans can correctly classify the shifted images and non-shifted images) such as [3, 5, 7, 8].
- This paper is not clearly presented. After reading the paper, I am still confusing about how to understand the experimental results. To me, the benchmark results cannot answer these questions well. I think R2 has a similar opinion on me in this criterion.
- It is not mentioned in my previous reviews, so I lower the weights for this part to the final decision, but there are already some datasets benchmarking the dataset distribution shifts, e.g., PACS [9], NICO [10]. It may not be true that this kind of distribution shift is only measurable by the proposed datasets. But, as my first words, I noticed that I did not mention these datasets in my previous reviews, and these datasets will not affect my review a lot.
  - https://domaingeneralization.github.io/
  - http://nico.thumedialab.com/

[8] Shankar, Vaishaal, et al. "Evaluating machine accuracy on imagenet." International Conference on Machine Learning. PMLR, 2020.
[9] Li, Da, et al. "Deeper, broader and artier domain generalization." Proceedings of the IEEE international conference on computer vision. 2017.
[10] He, Yue, Zheyan Shen, and Peng Cui. "Towards Non-IID Image Classification: A Dataset and Baselines." Pattern Recognition (2020): 107383.

Of course, building a new dataset is a non-trivial effort, and measuring real-world robustness is not an easy task (maybe it even can be an impossible task). However, I think this paper can not clearly present how the proposed benchmark can solve the real-world distribution shifts and how can we move forward in the next directions.

To sum up, I think this paper is okay, but not enough to be accepted to ICLR main conference paper. However, I will respect all decisions made by AC.
review_title: The unclear motivation for three different benchmarks and DeepAugment

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper provides a empirical study on the robustness of image classification models to distributions shifts. The authors construct three benchmark datasets that control for effects like artistic renditions of common classes, view-point changes, and geographic shifts (among others). The datasets are then used to test various hypotheses regarding robustness enhancing measures empirically. The authors additionally propose a novel augmentation scheme, that uses deep image processing networks together with random perturbations of their weights to synthesize distorted image samples.

---- Strengths ----

The paper tackles and important topic. I agree with the authors that robustness is "multivariate", i.e. can not be improved by a single factor. The paper makes an effort to disentangle various factors and test them in isolation.

The paper provides evidence that ImageNet-C can be used to make conclusion about real behavior of models, despite being based on synthetic image transformations.

The paper is well written and provides comprehensive experiments.

--- Weaknesses ---

The conclusions that result from the empirical findings are unfortunately not very crisp. Some hypotheses are supported by some datasets, others are not. No clear conclusions that hold across datasets can be drawn. It seems that we can't learn much from the experiments, and that the answer  to the questions "What improves robustness?" is still very much "it depends on what you are testing on". This is deeply unsatisfying, as true robustness  actually should not depend on the dataset. Table 4 provides a simplified summary of various hypotheses and how they are supported by different datasets. This table doesn't look too surprising: Why should, for example, self-attention help to improve performance on artistic renditions, something that  is very different from a blurred variant of an image? Why would it help classify an image of a pharmacy in France, when the model has only seen pharmacies  in the US (both will look very different, there cannot be a reasonable expectation for such a transfer). On the other hand, data augmentation can strongly abstract away certain image features, it thus is reasonable that it improves performance for ImageNet-R, but is limited for the other two datasets. 

The core issue seems to lie in the construction of the datasets:

1) We know that deep networks have a texture bias, so there is no reasonable expectation for transfer to ImageNet-R. There wouldn't be any expectations to improve this performance for any change, but enlargement of the dataset towards more abstract depictions of the objects. This seems to be confirmed by Table 1, where abstract augmentations (e.g. enlarging the training set with samples that are  in some respect more similar to the test set) clearly improves results, but simple architectural changes or simple augmentations do not help. This would support the hypothesis that to improve robustness, more similar data is necessary (i.e. the training set simply doesn't sufficiently cover the space of images that we expect the model to perform on).

2) The SVSF dataset paints a similar picture. Small shifts (e.g. images taken a year apart) hardly influence performance, whereas a extreme shift (e.g. in location) breaks the model even with the augmentations. This seems reasonable, as the augmentations certainly don't cover the shift that typically happen for building appearances between continents.

3) DFR paints a similar picture: Abstract augmentation slightly helps if it roughly matches the shift. Other simple augmentations or architectural changes do not significantly change results.

My conclusion from the experiments would be something that is well known:  enlarge the dataset to better cover what you expect your model to do. If this can be done with automatic augmentations  (e.g. for zoom, some augmentations that are closer to artistic renditions) then you can use these augmentations. If not: collect more data. It is thus not clear what the provided analysis provides on top of this. 

--- Other ---

Some clarifying questions:

- Why is SVSF limited to augmentations? What does a 30 day retention window mean?
- Can you clarify you conclusions on DFR? Why don't you see evidence for larger models or pretraining? Both seem to substantially improve performance?

--- Summary ---

I think this work is interesting and is in principle asking the right questions. However, the analysis and conclusions currently do not providing robust and generalizable insights that advance the field.

--- Post rebuttal ---

I'm keeping my initial score. My concern remains (and is apparently reflected by R1): the datasets and results do not allow to draw clear conclusions. The paper overall furthers our understanding on robustness only in a very limited way. The new dataset adds another specialized dataset to the mix. I disagree that the community should first exhaustively and randomly add datasets to the literature without coming up with a definition of robustness or at least try to categorize. The authors in their rebuttal criticize the community that they are looking at robustness and distribution shifts in a too simplistic way, but at the same time the presented work doesn't make an effort to change this either.

To close the remaining question:

- "Could you elaborate? We know that humans and primates can generalize to new renditions that they have not seen before (Itakura, 1994; Tanaka, 2006), while some other species cannot. Consequently more than training data matters."

With "extreme", I mean distributions shifts that keep semantics, but change appearance strongly. If we go as far looking at biological systems, then yes, it is not only about training data. There is an additional mechanism at play that we don't know and currently can't replicate in ML. Given our current understanding, it is presumptuous to suggest that NN architectures and training approaches as they are covered in this work will be able to do these kinds of generalizations at the level of humans or primates.

- "The empirical reality does not currently allow for a simple, single-cause characterization of robustness"

I completely agree to this statement. However, this doesn't mean that characterization of robustness cannot be done by taking into account multiple factors systematically.

I acknowledge that collecting a new dataset is a non-trivial effort and can be useful.  I acknowledge that the paper proposes an additional augmentation technique that seems to improve results in certain cases. All factors together taken together lead to my final score.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This paper investigates the robustness problem of computer vision model. To study the model robustness in a controlled setting, the author introduces three new robustness benchmarks: ImageNet-R, StreetView StoreFronts and DeepFashion Remixed. Each of them address different aspects of distribution drift in the real world. The author evaluates seven popular hypotheses on model robustness in the community on the three new datasets and has found counter-example for most of them. Based on those new results, the author concluded that model robustness problem is multi-variate in nature: no single solution could handle all aspects yet. And future work should be tested on multiple datasets to prove robustness. Moreover, the author also proposes a new data augmentation method using perturbed image-to-image deep learning model to generate visually diverse augmentations.


Significance: This paper is a solid work on the robustness problem. It systematically evaluated common hypotheses and successfully found counter-example on all of them except Texture Bias. The analysis is insightful and supported by the experimental results. The authors also provides three new carefully designed datasets for future work evaluation. While the study of using deep neural network to generate training image is not new, DeepAugmentation is still an innovative and practical way for data augmentation purpose.

Question: On DeepFashion Remixed datasets, it seems large zoom has better result than medium zoom. Is there a good explanation for that, considering the original image has no zoom-in? 

Clarity: The author did a great job on explaining the idea, objective and approach. 
review_title: Solid work!

Meta-review: The authors propose a new dataset to evaluate the robustness of image classifiers. The dataset consists of data from three sources: a crowdsourced dataset collected by the authors called ImageNet-Renditions, images from Google street view, and data sampled from DeepFashion2. This new dataset allows the authors to test robustness to different renditions of an object (e.g. artistic depictions of an object category) and robustness to changes in geography and camera type. In addition, they propose a new augmentation strategy called DeepAugment which consists of an encoder/decoder style network that transforms the appearance of the input image by simply applying different random perturbations of the weights of the augment network. Robustness results are presented on the previously described datasets where the proposed augmentation strategy in combination with an existing approach (AugMix) performs best in some cases. However, the results are not convincing and AugMix often outperforms the new method.  

In general, the authors did a good job addressing many of the comments (e.g. they provided more detail about how ImageNet-R was collected), but there were still several lingering concerns. R4 was the most positive about the paper, but unfortunately was one of the least vocal during the discussion. R1 was concerned that the paper did not do a great job of defining what was meant by robustness. This AC doesn't agree fully with their concerns, but does agree that more care could have been taken to position the paper better in light of the existing datasets that are already available (see R1’s comments). As the reviewers and authors note, collecting new datasets is a lot of work so care should be taken to ensure that this is not duplicate effort. The authors addressed these concerns in their response to some extent, but more discussion is needed in the paper. 

There was a lot of discussion between the authors and reviewers about this paper. The new dataset has a lot of merit, but there is some concern that the paper does not do a great job of clearly presenting its findings and conclusions. In addition, the proposed augmentation technique is slightly underwhelming performance wise and not very clearly described in the main paper. R2 sums up the opinion of this AC: “I think this work is interesting and is in principle asking the right questions. However, the analysis and conclusions currently do not provide robust and generalizable insights that advance the field.” There is clearly a lot of promise here, and the current recommendation is a weak reject. The authors are strongly encouraged to take the detailed feedback they have received on board and to revise the paper to further improve it for a future submission. 

Decision: Reject