Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studied an interesting question, whether the gain of robustness from synthetic distribution shifts can be transferred/generalized to the robustness under natural distribution shifts. It was shown that in the context of natural distribution shifts, no current robustness intervention can really outperform standard models without a robustness intervention. The main strength of this paper is its extensive experimental study. However, I still have concerns on this work.

1)  The authors mentioned "an implicit assumption underlying this research direction is that robustness to such synthetic distribution shifts will lead to models that also perform more reliably on natural distribution shifts." However, I am uncertain about this point. Let us take adversarial robustness as an example, I am NOT surprising that the robustness on crafted adversarial examples cannot be generalized to the robustness over natural distribution shifts. Thus, I do not think that adversarial robustness obeys the 'implicit assumption'. In other words, the studied problem should be better connected to adversarial robustness.

2) The significance of 'effective robustness' is not clear. It seems that most of insights were learnt from Figure 1, 3 and 4. Do they rely on the metric 'effective robustness'? 

3) In Figure 1, 'Trained with more' -> 'Trained with more data'

############# Post-feedback ################
Thanks for the response. I am still not fully convinced by the significance of the findings in this work.
In the paper, the authors highlighted that "our results show that current robustness gains on synthetic distribution shifts do not transfer to improved robustness on the natural distribution shifts presently available as test sets."  I am not surprising at this point, since the synthetic shift, e.g., introduced from adversarial examples, may only characterize the short-cut shift for misclassification. Thus, robustness learnt from this synthetic distribution shift might not transfer to the natural distribution shift. 

I decide to keep my original score. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
============================================ Update after rebuttal =========================================

I thank the authors for their detailed rebuttal. I'm in general satisfied with the authors' response to my concerns, so as promised, I'm happy to increase my score.

========================================================================================================

This paper considers the relationship between various measures of synthetic robustness and two distinct measures of natural robustness in large scale image classification models. The authors argue that the synthetic robustness measures considered in this paper are not predictive of natural robustness when the effect of baseline accuracy is subtracted. I think, if true, this is an important message that researchers in this area need to be aware of. However, I have a number of questions and concerns about the results. I would be happy to increase my score if the authors could address some of these issues:

1) Another important recent benchmark not mentioned in the paper is ImageNet-A (https://github.com/hendrycks/natural-adv-examples). I would encourage the authors to include this benchmark among their natural robustness measures (in addition to ImageNetV2 and ImageNetVidRobust). The advantage of this dataset is that because it samples from the error distribution of a high-performing ImageNet model, to a large extent, it already comes with the baseline subtracted, so it essentially obviates the need for the indirect effective robustness measure introduced here. The raw accuracies on ImageNet-A would be directly interpretable and they would also answer the question “why should we care?” in a more visceral way, because even the Instagram trained state-of-the-art ImageNet models seem to achieve a mere 17% accuracy on this benchmark: https://arxiv.org/abs/1907.07640

2) There seems to be a direct conflict between the main conclusion of this paper (that synthetic robustness measures do not predict natural robustness) and an opposite conclusion reached in an earlier paper (https://arxiv.org/abs/1904.10076) where the authors claim that robustness against synthetic perturbations like translation, hue, and saturation are actually highly predictive of video robustness (not sure if this would generalize to ImageNetV2). As far as I can see, these particular perturbation types are not included among the synthetic perturbations considered in this paper. Can you please clarify this discrepancy? 

3) Relatedly, looking at the scatter plots of effective robustness vs. robustness against individual perturbations in the appendix, especially for the video robustness measure, some of the correlations seem to be pretty significant (for example, video robustness vs. jpeg compression robustness, p. 19). So, I am wondering to what extent the main conclusion of this paper might just be driven by the averaging of a large number of non-predictive perturbations and a smaller number of more predictive perturbations. 

4) Also, ImageNet-P perturbations are not included in the paper. If the authors want to make their claims more reliable, I would encourage them to consider these among their synthetic perturbations as well. The translation perturbation in ImageNet-P, in particular, would be particularly important to consider for video robustness given the results from the arxiv pre-print mentioned in 2) above.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary: This paper tries to evaluate whether robustness ‘interventions’ such as robustness to adversarial examples and other synthetic distribution shifts on natural distribution shifts. 
Overall this paper is generally well written and focuses on an important direction of understanding when the toy/synthetic settings used for convenience actually transfer to the real world. However, I have some concerns with the current version of this work. 

Decision: I vote for rejecting this paper. My main concerns are regarding both the motivation of this work and the quality and soundness of the claims. 

— Soundness: My major concern is with the datasets that are used for “natural” distribution shift. ImageNetV2 is essentially from the same distribution as the original Imagenet. It’s unclear what the distribution shift is and whether there is a distribution shift. In fact, the paper that produces Imagenet V2 tries very hard to minimize the distribution shift. Hence it seems unreasonable to use this dataset for a shift. Similarly, the other dataset used is from Imagenet Videos. Here again, it’s unclear what the shift actually is, and hence using these datasets to measure performance to natural distribution shifts seems questionable. 

— Motivation: I am a little confused about the motivation of this work as presented. It does not seem unreasonable that robustness to one kind of shift doesn’t transfer to another. It seems like the right questions to ask would be the following: Do the synthetic robustness goals seem like they would occur in natural settings? Does robustness on synthetic datasets and synthetic generations of such perturbations transfer to perturbations (of the same kind) but occurring in natural world. 

Finally, it seems like the novelty/contribution of this work is very small. The paper uses existing datasets and existing robustness measures. 
review_title: Official Blind Review #2

Meta-review: The authors show that models trained to satisfy adversarial robustness properties do not possess robustness to naturally occuring distribution shifts. The majority of the reviewers agree that this is not a surprising result especially for the choice of natural distribution shifts chosen by the authors (for instance it would be better if the authors compare to natural distribution shifts that look similar to the adversarial corruptions). Moreover, this is a survey study and no novel algorithms are presented, so the paper cannot be accepted on that merit either.
Decision: Reject