Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new dataset for estimating robustness to distribution shift, in particular corruption robustness. They accomplish this by proposing an alternative to ImageNet-C, ImageNet-NOC, which uses different corruptions. They consider corruptions not in ImageNet-C, and they argue that their dataset is superior because they have more "balance and coverage." They select corruptions that are "decorrelated" in a specific sense.

I appreciate the analysis in this paper and provides useful comparisons to ImageNet-C. Unfortunately experimentation is not thorough. They don't show results with AugMix nor DeepAugment, so the analysis only uses SIN and ANT.

The paper is missing key qualifiers.
"In other words, the expression (1) estimates if increasing the robustness to c2 implies an increase of robustness to c1 and vice versa."
... provided that the model fits the exact corruption c2.
"Then, the higher (1) is, the more the robustnesses to c1 and c2 are correlated, i.e. the more c1 and c2 overlap."
... provided the models are robustified through exactly training on c1 or c2.
Without these qualifications, the claims are far too general.
Contrast and fog can be negatively correlated with the rest of the corruptions under some robustness interventions. For example, adversarial training really harms contrast and fog corruption robustness, though it might help other corruptions like noise (and a less expansive testbed might not catch this). This demonstrates that their robustness correlations require qualification and are less predictive that this paper suggests.

This paper is fairly similar to Is Robustness Robust? On the interaction between augmentations and corruptions (submission #704). If these papers have very different ratings, then there's a problem with this review process.

Nitpicks:

The bibtex for the citations are messed up. For example, we see "(Evgenia Rusak, 2020)" instead of "(Rusak et al., 2020)". Author ordering for papers in the bibtex is often scrambled, some are separated by commas, and some are not. In general this paper's formatting is a little unrefined.

"weight decay set to 10e-4." -> "weight decay set to $10^{-4}$."

"For instance, a benchmark that contains a motion blur corruption covers the defocus blur corruption, because the robustnesses [sic] towards these two corruptions are correlated (Vasiljevic et al., 2016)."
Vasiljevic et al. (https://arxiv.org/pdf/1611.05760v1.pdf) show the opposite. Fine-tuning on defocus blur did made models slightly worse on horizontal motion blur (Figure 2).
review_title: Modification to ImageNet-C too marginal

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Paper summary
The paper considers the problem of measuring the robustness of image classification models to common image perturbations. Datasets of corrupted images, such as ImageNet-C, have been created for this purpose. However, these datasets have been created from an ad-hoc, heuristic selection of perturbations. The present paper proposes a systematic approach to select types of perturbations in a way that spans a large variety of perturbations and assigns similar importance to each perturbation. Similarity of perturbations is measured based on how much training on one perturbations confers robustness against another perturbation. The paper provides an algorithm for selecting perturbations to include, and uses the algorithm to create a variant of ImageNet-C with improved coverage and balance.


## Arguments for acceptance
1. Robustness of computer vision models to image perturbations is a topic of great interest, but methods to measure robustness are either highly artificial (adversarial robustness) or ad hoc heuristics. The present paper takes an important step in the direction of making evaluation of robustness to non-adversarial corruptions more principled.
2. The paper is written clearly.
3. The overlapping score is based on a practically relevant quantity, namely the performance of a network on the corrupted data.
4. A simple algorithm is provided for selecting perturbations for new datasets.
5. A new alternative to ImageNet-C is created with the proposed algorithm. The new dataset has improved coverage and balance properties. Some evidence is provided that these improvements can affect the results of robustness evaluations.

## Arguments against acceptance
6. The computational cost of the method is high, and not stated or discussed. As far as I can tell, at least one neural network needs to be trained for each candidate dataset. Although the authors state that results transfer across architectures, such that a small architecture can be used, it is known that robustness properties depend significantly on model size and training procedure (e.g. see https://arxiv.org/abs/2007.08558). This should be addressed further.
7. This method only really works for synthetic corruptions for which many new examples can be generated. Otherwise, it may be difficult to obtain enough examples to train the network used for computing the overlapping score.
8. It is not discussed whether Algorithm 1 is guaranteed to provide the optimal combination of datasets (in terms of overlap and balance).
9. It is not entirely clear if the improvements provided by ImageNet-NOC make a significant difference in practice. Table 3 starts to address this question, but it would be useful to compare ImageNet-C and ImageNet-NOC mCE across a wider range of models (e.g. pretrained models available online). What is the rank correlation between ImageNet-C and ImageNet-NOC mCE? 

## Conclusion and suggestions
This is a borderline submission. Because of the principled approach to an important and topical problem, I tend towards accepting it.

Suggestions for improvement:
10. Discuss the computational cost of the method.
11. Discuss the optimality of Algorithm 1.
12. Compare ImageNet-C and ImageNet-NOC on a wider range of models.
review_title: A principled approach towards robustness evaluation with some shortcomings

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##################################################################

Summary:

This paper proposes an algorithm to build a benchmark of Non-Overlapping Corruptions. The proposed ImageNet-NOC dataset is balanced and also covers a wider range of corruptions. 

##################################################################

Pros:

(1) This paper first provides detailed analysis of dataset coverage and balance. 
(2) This paper develops an algorithm to construct a robust benchmark that is balanced and non-overlapping, which means a lot for future work.
(3) This paper is well-organized relatively.
(4) The experimental results are relatively consistent with the mentioned features.

##################################################################

Cons:

(1) The coverage (Sec 5.3) is a little bit confusing. It reads “We observe that the mINOC model is 【significantly】 more robust than the standard model to all the ImageNet-C corruptions … … ImageNet-NOC covers all the corruptions of ImageNet-C”The reviewer noticed that gaps of some corruptions are not sufficiently significant, for example, Elastic (67 vs 63), Pixel (78 vs 71) and Jpeg (56 vs 53). Although marginal gaps are observed, can we conclude that ImageNet-NOC already covers that of ImageNet-C? The reviewer suggests adding the following experiments: Reporting both the results of m_IC evaluated on ImageNet-C & ImageNet-NOC, and that of m_INOC evaluated on ImageNet-C & ImageNet-NOC. Comparing “m_IC on ImageNet-C” and “m_INOC on ImageNet-C”. If these two results are similar, then we can conclude that ImageNet-NOC covers ImageNet-C. 

(2) The experiments in Sec 5.4 show that SIN+IN is more robust than ANT on ImageNet-NOC. But on ImageNet-C, ANT is more robust. Is this phenomenon caused by imbalanced corruptions of ImageNet-C? Could the authors explain this? (Maybe based on the proposed overlapping scores between corruptions.)


##################################################################
[Edit]
After reading the authors' response and other reviewers' comments, I have lowered the scored to 5. The reasons are bellow.

This paper provides good analysis of the coverage and balance of robustness benchmarks. Based on the analysis, the paper presents a method to construct a benchmark of Non-Overlapping Corruptions. I think such analysis is valuable and interesting, despite some concerns about the coverage comparisons (which are only partially addressed by the authors' response). 

I agree with Reviewer#1 that the comparisons between ImageNet-C and ImageNet-NOC are not solid. And that "the paper should have done an equivalent evaluation, comparing ImageNet-C as a whole vs. ImageNet-NOC as a whole." Since the core idea of this paper is to propose a better robustness benchmark, solid comparisons between ImageNet-C and ImageNet-NOC are critical.

I also agree with Reviewer #3 and Reviewer #4 that, the improvements provided by ImageNet-NOC does not make a significant difference. 

The reviewer appreciates the analysis of the coverage and balance of the robustness benchmarks, which in my opinion, is valuable. However, since this paper focuses on proposing a better benchmark, solid evaluations between these two benchmarks are critical. 

review_title: Official Blind Review2 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper points out that ImageNet-C, the de facto standard for measuring robustness to natural corruptions for ImageNet classification models, contains correlated corruptions, so a mean robustness score over the ImageNet-C corruptions is biased in favor of certain classes of corruptions. It proposes two metrics: robustness score and overlapping score, and uses them to specify desirable characteristics of a robustness benchmark: coverage and balance. It specifies an algorithm for selecting a set of corruptions that optimize for these characteristics, and introduces a new benchmark (ImageNet-NOC) built using this algorithm and evaluates some models that performed well on ImageNet-C on ImageNet-NOC to see if they still perform well.

Positives:
* I think the authors are correct that some corruptions in ImageNet-C are correlated and therefore, using a simple mean over corruptions is going to bias towards models that improve on the correlated corruptions.
* Results like Section 5.4 seem valuable, since they show that existing ImageNet-C results may not generalize to robustness to a broader set of corruptions.

Concerns:
* The ImageNet-C paper specifically says "networks should not be trained on these images," for good reason, since training on them makes it too easy to optimize for the metric, but miss the overall goal. This paper specifically trains on it. Maybe it's ok for this particular case of analyzing the benchmark itself, but that should be called out very clearly. The entire benchmark becomes useless if people start training on it.
* There is an assumption made throughout the paper that using a corruption as data augmentation during training imparts robustness to that corruption on the trained model. This needs to be established. I have seen empirically that it isn't always the case.
* I worry that Algorithm 1 is itself not robust to changes in architecture or training methodology. You do present results showing that the corruptions selected using ResNet18 also have low overlap when measured on DenseNet, but I would have also liked to see whether running Algorithm 1 with the same set of candidate corruptions, but using DenseNet, would result in the same set of corruptions being selected.
* The claims in section 5.2 seem entirely misleading. The results you show are purely a result of the initial set of candidate corruptions you chose, and you provided no methodology for how to perform that step. If your initial set had been exactly ImageNet-C or even a subset of ImageNet-C then the results would have looked entirely different. Similarly, I could introduce a new corruption that isn't in your candidate set and suddenly ImageNet-NOC would look like it has low coverage. Given that the space of possible corruptions is unbounded, this notion of coverage doesn't seem at all valuable to me.
* On the whole, I think this paper would be a lot stronger if the proposal was just a set of weights for each ImageNet-C corruption, such that the top-line robustness metric used would be a weighted average that is less biased towards correlated corruptions.

Minor details:
* It's kind of weird to have section 2.3 placed where it is. It would make more sense at the beginning of section 4.
* In section 3.1, the statement that R_c^m \in [0, 1] isn't strictly true. It's possible that A_c > A_clean.
* It's worth mentioning explicitly that the robustness score, as you've defined it, is only valuable when presented along with A_clean. If A_c == A_clean because both are at chance, this has optimal robustness but isn't interesting.
* R^standard is used in Eq 1 but it isn't defined until a few paragraphs later.
* Algorithm 1 is very verbose and complicated-sounding. Steps 2-6 could just have been written as: "Pick the largest subset with overlap score under the threshold and break ties by selecting the lowest overlap score."
* There are a number of misspellings and grammatical issues throughout.

Reasons for score:
The critique of ImageNet-C is valid, but I don't think the proposed solution really addresses it. A good solution needs to be totally agnostic to architecture and training algorithm, which this method is not. There also needs to be a lot less dependence on the manually selected list of initial corruption candidates. I think ImageNet-C made a conscious choice not to involve model training when building their benchmark dataset and they had good reasons for doing so. Those reasons (laid out in the "Concerns" above) are why I don't think this paper is worthy of acceptance.
review_title: ImageNet-NOC

Meta-review: The authors propose a new dataset, namely ImageNet-NOC, for evaluating robustness of image classifiers to corruptions. The dataset may be viewed as an alternative to ImageNet-C which uses a different set of corruptions. To derive this set of corruptions, the authors first develop a notion of similarity between two corruptions, and then propose an iterative algorithm to build a set of corruptions which, intuitively, is sufficient to cover the larger set of corruptions (i.e., enjoys *high coverage*), and assigns a similar importance to each such corruption (i.e., is *balanced*). Then, the authors argue that ImageNet-NOC is superior to ImageNet-C as it achieves a higher degree of balance and coverage.

The reviewers found this to be a borderline paper. The reviewers appreciated the introduced metric and agree that there is no point in evaluating on corruptions which are perfectly correlated. In addition, the systematic approach for generating a set of relevant corruptions is seen as a step in the right direction. The reviewers appreciated the author response and were engaged in the discussion. As it currently stands the reviewers are not convinced that the paper is ready for acceptance. To improve the manuscript the authors could extend Tables 3 and 4 with a wider range of models and investigate qualitative differences between models robust on one dataset, but not on the other. Furthermore, there should be a more detailed discussion of stability and computational properties of algorithm 1. In addition, the authors should provide strong arguments as to why is it not sufficient to add additional corruptions to ImageNet-C and compute a weighted score instead. The latter suggestion could lead to an iterative improvement of the current set of benchmarks and place more emphasis on the methodology. I suggest the authors to incorporate the reviewer's feedback and place more emphasis on the methodology around algorithm 1, rather then on introducing another dataset which is likely to be superseded as soon as we add a couple more corruptions in the mix. 
Decision: Reject