Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose an exciting perspective to rethink the interaction between augmentation and corruption. The perceptually similar augmented training images could be helpful for better performance on common natural corrupted test-time images. The creation of CIFAR/ImageNet-$\overline{C}$ and the related discussion (section 6) indicates the future direction of data augmentation for corruption robustness with enough care in evaluation protocol.
main_review: - The proposed method is easy to follow. The overall writing looks well-organized to me. Both big picture and implementation details are discussed intensively. I believe the readers could catch up with the most critical observations in an easy way. 
- The illustrations are helpful for the readers to realize the main motivations and contributions of the proposed method. Specifically, the correlation plots between minimum sample distance (MSD) and corruption error look well-structured to highlight the key points.
- The contributions look solid to me: 1) perceptual similarity measurement, minimal sample distance (MSD); 2) new dataset, CIFAR/ImageNet-$\overline{C}$, perceptually dissimilar training samples; 3) understanding and analysis on the interaction and the discussion on the future direction.
limitations_and_societal_impact: Besides train-time augmentation, there are a bunch of methods that leverage the test-time/source-free adaptation[1][2][3] techniques to update the model during the inference phase. I appreciate the authors' efforts in the existing evaluation. I wonder if these methods could also be evaluated as the representative approaches for the group of test-time dynamic models.

- [1] Sun, Yu, et al. "Test-time training with self-supervision for generalization under distribution shifts." International Conference on Machine Learning. PMLR, 2020.
- [2] Liang, Jian, Dapeng Hu, and Jiashi Feng. "Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation." International Conference on Machine Learning. PMLR, 2020.
- [3] Wang, Dequan, et al. "Tent: Fully Test-Time Adaptation by Entropy Minimization." International Conference on Learning Representations. 2020.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a metric, the Minimal Sample Distance (MDS), that quantifies the distance between the distribution of samples augmented with a certain augmentation and samples affected by a certain corruption. The authors show a correlation between the MDS and the corruption error, and use this metric to show that existing corruption benchmarks, e.g. ImageNet-C, are not broad enough to properly evaluate the robustness of image classifiers. Thus, they expand ImageNet-C with further corruptions that are not perceptually similar to those already available.
main_review: The paper addresses an important problem of Computer Vision, namely the robustness of image classifiers to common test-time corruptions that are not seen during training. This links with the generalization capabilities of classifiers, and their applicability to real-world problems.

Strengths:
- Extension of the ImageNet-C benchmark with perceptually-dissimilar corruptions
- A metric (MSD) that measures the relation between augmentation and test-time corruptions
- Demonstration that the robustness of computer vision models needs to be addressed with a broader approach 


Concerns:
- By the definition of the MSD, it tries to measure how much a certain augmentation is contained in the distribution of a given corruption, by computing the distance between the nearest sampled point. I have the impression that such evaluation might not be stable, having some flaws due to the randomness of agumentation and corruptions. It seems to me that the overlapping area between the two distributions might be a better option. I did not find a possible intepretation of this type, and would like the authors to comment about.
- 'perceptual' similarity is not defined precisely. This opens room for doubts and interpretations. When, precisely, two corruptions can be said perceptually similar? and what are the conditions that determine that two augmentations/corruptions are not similar?
- I have the impression that a lot of the similarity/dissimilarity can be explained by looking at the spatial frequency components that certain agumentations and corruptions affect. Using similar agumentations (likely meaning that they inject or modify similar spatial frequency statistics) is clearly not much beneficial, especially when compared to using augmentations that affect a larger portion of the spectrum, thus more dissimilar. More concrete explanations of the meaning of perceptual similarity/dissimilarity should be provided, in terms of frequency analysis or other approaches that the authors might find more suitable.
- the paper and results discussion rely too much on the supplementary material. The paper is thus not self-contained and the observation and delivered messages are not strong enough without looking into extra materials.

I think this paper addresses an important problem and proposes data and metrics to stimulate advancements and new solutions in the field. However, some results and interpretations need a little clarrification.
limitations_and_societal_impact: The author discuss limitations and potential societal impact in a broad sense, being the work addressing a general problem with large potential impact. However, I did not see explained how this work can be used to prevent issues and failure cases, as the authors only mention this possibilities briefly.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper study the similarities between the augmentations/augmentation schemes with the corruptions and reason that the perceptual similarity between them is a strong predictor for the robustness improvements. The authors propose a new metric, Minimal Sample Distance (MSD), to evaluate the perceptual similarity between a corruption and an augmentation statistically. In addition, authors construct a new corruption benchmark ImageNet-C bar that is perceptually dissimilar to ImageNet-C. Results on this benchmark show that recent robust models with different augmentations schemes have reduced performance than ImageNet-C, suggesting that these augmentation schemes share perceptual similarity to ImageNet-C.
main_review: Originality: 
The paper provides a new interesting finding and the metric proposed to evaluate the perceptual similarity between corruptions and augmentations is novel. Moreover, the paper also proposes a new corruption benchmark ImageNet-C bar.

Quality: 
The paper is technically sound and the claims like “Augmentation-corruption perceptual similarity is a strong predictor of corruption error” is moderately supported based on the results from Table 1.

Clarity: 
The paper is clearly written, well organized and included important details to avoid possible misunderstandings.

Significance: 
The paper studies an important relationship between the recent data augmentation schemes and the robustness of common corruptions. Future works would benefit from the interesting findings and the benchmark proposed in this paper. To my knowledge, this is the first paper that proposes a statistical measure to find the correlation between augmentations and the corruptions.
Results from Table 1 show that recent robust models drop few percent on ImageNet-C bar compared to ImageNet-C. It is not trivial to conclude or not easy to support the authors claim “Augmentation-corruption perceptual similarity is a strong predictor of corruption error” based on the few percent drop from this table. However, individual corruption errors reveals that performance on some of the corruptions are worse than the others.

ImageNet-C bar is constructed to be perceptually dissimilar to ImageNet-C in the transform feature space. Are any of the ImageNet-C bar corruptions perceptual to existing data augmentation schemes?  For example, any lower MSD between augmentations from Augmix (or other augmentation schemes) and ImageNet-C bar? Does that explain augmentation schemes improvements on some of the corruptions in ImageNet-C bar? Illustration similar to Figure 5 on ImageNet-C bar would be helpful here.

Does AugMix tested with Jensen-Shannon divergence loss? If so, it would be interesting to check the AugMix results with and without divergence loss on ImageNet-C bar.

How does the augmentations generated by a deep network DeepAugment (Hendrycks et al. [15]) would correlate with the ImageNet-C and ImageNet-C bar corruptions? How does DeepAugment, DeepAugment+Augmix would work on ImageNet-C bar?

Overall, I find this paper interesting and accept it as Good paper. However, I would reconsider my rating based on other reviews and author responses.

-------------------------------------------------------
Post rebuttal remarks:

I read the other reviews and authors responses. I thank the authors for providing additional insights. Given the rebuttal time, I understand that some of posed questions might not be completely addressed. However, I highly encourage the authors to address those questions in their final version, as it is beneficial to the community. Finally, I believe that this work is a significant contribution to the community and tend to keep my rating to accept it.

limitations_and_societal_impact: I appreciate the authors for addressing the limitations and societal impacts of their work..
review_title: null

Meta-review: This is an interesting paper, and there was clear agreement among the reviewers that this marks an interesting contribution. 
Decision: Accept (Poster)