Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: **High level goal:** The authors are interested in an empirical study of models that display *effective robustness* (those that are "above the line" that's traced by in and out-of-distribution accuracies of models that don't display effective robustness) in the image domain. 

**Gap in literature that the paper contributes in addressing:** The conditions (on OOD data, model and training techniques) under which effective generalization is achieved is not well understood. 

**Main contributions:**
  * Using previously proposed metrics that are known to correlate with OOD generalization (including some new metrics based on the Fourier spectra of the images and models), the authors study how different types of pruning, data augmentation and ensembling affect effective generalization. 
  * The authors release the RobustNets suite of models that display effective robustness to facilitate further research in this space. 
  * The Fourier-based metrics proposed by the authors correlate well with effective generalization of models form the CLIP family. It's also interesting that these metrics don't correlate particularly strongly when other (mostly convnet based) models are used, which is useful in gaining indirect knowledge about the types of functions represented by these networks. 






strengths_and_weaknesses: STRENGTHS:
* **Relevance:** The focus of the paper is well within the scope of NeurIPS, especially given the recent interest in gaining a deeper understanding of OOD generalization. 
* **Comprehensive evaluation:** The experiments are quite extensive, providing a plethora of useful data for understanding the conditions under which effective generalization occurs. 
* **Usefulness of the spectral characteristics of in and OOD data:** It turns out that the spectral characteristics of in and OOD data gives useful information regarding on which distributions shifts are more likely to induce "accuracy-on-the-line' style phenomenon. 
* **Evidence that each distribution shift is "unique"** The authors make a compelling case that each distribution shift has it's own problem structure, and there's not single metric that correlates well with generalization across all shifts (at least among the ones explored in the paper)


WEAKNESS:
* **Lack of causal explanations:** Even though the authors use words that imply the results provide a causal understanding of why effective generalization happens, all of the results are correlational. It's misleading to claim that the experiments can answer "why" questions. I believe it's very important that this is fixed before the camera-ready version, should the paper be accepted. (I'm happy to hear arguments against this criticism) In my opinion, this is an important limitation that can be fixed simply by removing any causal claims from the paper - the existing results are still worthwhile. 
* **No emphasis on architecture:** It'd be interesting to see a discussion of whether model architecture has any impact on effective generalization. The authors likely have enough empirical data to address this - a discussion of this would have made the paper stronger. 
**Lack of mechanistic understanding:** Despite a plethora of empirical data, I don't think I've acquired a deeper (i.e. mechanistic) understanding of why/how effective robustness occurs. I also am not sure if a practitioner also leaves the paper with a solid understanding of which techniques to use to achieve effective robustness, beyond using methods from already existing literature. 
  * This is not to say that the presented results are non meaningful - perhaps follow-up works will fill these gaps. 

================
**Post rebuttal update**
I thank the authors for their response. Since some of my important concerns are addressed, which I've reflected on my review by increasing the score. 
questions: * Could you explain which models you evaluated on ImageNet? The description (between lines 109 and 112) is a bit vague. Which architectures are included, for example? 
* Line 153: Could you clarify what you mean by "we interpolate the lowest 40% of image frequencies"? Are these the frequencies that occupy 40% of the energy? Or perhaps something else? Also, do you leave the rest of the frequencies untouched?
* Line 280: Are the 11 CLIP models obtained by interpolating the same pretrained and finetuned versions of CLIP? 


Nitpick: It perhaps could be interesting to add a plot in the Appendix that supplements Figure 1 in that instead of tracing amplitude or phase interpolations along a single axis, a grid of images can be provided where on the left top and right bottom, we have the unperturbed images, and in the middle we have various (mixed) interpolations. 
limitations: **Only image-based distribution shifts:** The proposed metrics (and the analysis) is only concerned with image data. 
**Experiments have no causal power:** Despite authors' claims, the experiments are fully correlational, and this should be acknowledged so. (see weaknesses for above)
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper carries out a comprehensive study of the Distribution Shift Robustness through a Fourier lens and designs new 'Fourier sensitivity metrics' that capture Fourier sensitivity of models as test data moves farther away from the training data manifold. It provides the theoretical analysis of the OOD robustness on pruning, data augmentation, and weight ensembling.
strengths_and_weaknesses: Strength: 
(1) This paper considers the OOD robustness on pruning, data augmentation, and weight ensembling.
(2) This paper designs new metrics that capture the Fourier sensitivity of models.

Weaknesses:
(1) To the best of the reviewer's knowledge, this is one existing work [1*] considering both the amplitude and phase on the robustness in the frequency, while [1*] is not in this paper's reference list. The design difference is needed, at least.

(2) The Power spectral densities in Figure 2 are nearly the same as those in [36]. A citation of [36] is needed in Sec. 4.1.

(3)  The experiment setting is a little strange. The experiments in Secs. 4.2 and 4.3 are conducted on CIFAR-10, while ImageNet is used in Sec. 4.4. From the reviewer's view, it is better to conduct the experiments on one consistent dataset or on both datasets.

(4) Since all the experiments are conducted on corruption benchmarks (CIFAR10-C and ImageNet-C), the separated experimental results on each type of corruption are suggested to add.

[1*] Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain. ICCV 2021


-------------------------------------------------------
Post-rebuttal reviews:
My main concerns are on the experimental settings, which have been well-addressed in the rebuttal. I will raise the recommendation to 'weak accept'.

questions: The questions are listed in the pros and cons.
limitations: N.A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper takes metrics which predict OOD robustness (such as ID accuracy, model jacobian norm, etc.) and then looks at the correlation between these metrics and OOD accuracy for methods which can produce robust models. This paper observes that while ID accuracy is a good predictor of robustness for natural distribution shifts (ImageNet -> ImageNetV2), for some synthetic distribution shifts there are better metrics. In general they find "no single metric rules them all". The paper also introduces new metrics which correlate will with the robustness of ensembled CLIP models.
strengths_and_weaknesses: Strengths:
Overall the results are very interesting and the empirical findings are likely useful to the community. Moreover, the proposed metrics seem that they may be potentially useful in further understanding CLIPs robustness. Finally, robustnets could be a useful resource.

Weaknesses:
My thoughts are that the main weakness of this paper was the lack of distribution shifts, in particular natural distribution shifts which are more likely to occur in the world. I would have appreciated results for additional distribution shifts including ImageNetR or ObjectNet. In addition, it's not clear if there is a takeaway for designing better models. Finally, while a conclusion of this paper is that there is no one metric to rule them all, there is only one column of plots without ID accuracy on the x-axis. Consider showing other metrics on the x-axis in the appendix.
questions: - Why do the authors think that proposed metrics correlate well for ensembled CLIP models?
- Can any of the proposed metrics be used as objectives to train better models?
- What happens on additional natural distribution shifts such as ImageNetR or ObjectNet or a shift from WILDS?

limitations: Yes
review_title: null

Meta-review: metareview: All reviewers noted the relevance of the proposed study for the NeurIPS community. They all agreed that the paper is well-motivated, sound, and that the proposed Fourier interpolation is novel. While some reviewers had initial concerns regarding the experimental evaluation, the authors did a great job at improving their experiments and reply to the reviewers's concerns. There, we recommend acceptance. 
confidence: Certain
Decision: Accept