Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies on whether pruning is able to improve the model robustness. This paper first show that with a proper pruning algorithm, the robustness can be achieved. Then a spectrum analysis is proposed to visualize what kind of heat map is suitable for robustness. Thirdly, a test-time augmentation approached is proposed to further improve the robustness.
main_review: The direction this paper studied is interesting while there are several important points regarding the significancy of the result needs discussions.

1. To draw the main conclusion on what kind of pruning algorithm can improve the robustness, only one kind of bench mark (Cifar-10 on label corruption) is tested. This makes the conclusion less convincing. How about adversarial robustness? OOD detection? Calibration? The robustness so far is quite limited and the dataset test is also quite limited.

2. I appreciate the analysis through the heat map, which is very interesting. However, despite that it is a successful visualization, it still didn't give answer to the most important question: why pruning is helpful for robustness. For example, if we train the network use some standard approach for robustness, is the heat map of pruning close to the network using robust training? If so, why this happens. I understand it is hard to really give a very comprehensive explanation but to me, the current analysis is more close to listing the observations.

3. Although pruning can improve the robustness, but more important experiment is to show, combined with the training algorithm that improves the robustness, whether pruning can further enhance the robustness above that. To seriously achieve the robustness, people will use non-standard training approach. How's the pruned network's robustness performance compared with that? How much improve you can further achieve above that?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper performs an empirical study on the OOD robustness of compact models and shows the existence of compact, accurate, and robust DNNs (CARDs). It gives a frequency-domain analysis with the Fourier sensitivity method to explain why different pruning methods lead to compact models with varying OOD robustness. Then, the paper proposes CARD-Deck, a domain-adaptive test-time ensembling approach, to dynamically choose an appropriate CARD for each test sample using a spectral-similarity metric and presents some experimental results with CARD-Decks. Finally, it provides some theoretical analysis to support the existence of CARD and the ability to find a CARD with CARD-Deck.
main_review: **Originality**:

The paper tries to show that compact, accurate, and robust deep neural networks (CARDs) exist and to develop efficient algorithms to design CARDs that give comparable or better OOD robustness compared to the SOTA dense DNNs. These problems are important and it is novel that the paper points out compression can improve OOD robustness if done properly. Also, this work uses the existing Fourier sensitivity method to study the OOD robustness with different pruning approaches, which is valuable.

**Quality**:

1. The first part of the paper (sections 2 & 3), which discusses the observations on model OOD robustness with different pruning approaches and the Fourier sensitivity studies, is sound and the presented results support most of the claims in the paper.

    * Here is a question regarding FT and GMP results in Figure 1 and Figure 3: in Figure 1, compared to the dense baseline, FT and GMP have higher Top-1 accuracy on CIFAR-10 and lower Top-1 accuracy on CIFAR-10-C. However, in Figure 3, the center of the heatmaps for FT and GMP indicates lower accuracy on clean data and higher robustness to the OOD data following the logic presented in lines 186-191. The conclusions given by the two figures are contradictory. How to explain this?

2.  The second part (section 4) talking about CARD-Deck is very confusing. I have several questions regarding it:
    * What is the difference between CARD-Decks of size 2, 4, and 6?
    * Given the first part of the paper talking about different pruning approaches leads to different OOD robustness, I expected to see a CARD-Deck containing models compressed with different pruning approaches and adaptively selecting models with an approach with the highest OOD robustness. However, according to the results in Table 2, it seems that a CARD-Deck only containing models trained with 2 augmentation schemes and compressed with a single pruning approach. Is there any reason for it? A CARD-Deck containing models compressed with only a single pruning approach is less attractive.
    * It is not clear how CARD-Deck achieves the clean and robust accuracies presented in Table 2. In Table 1, the highest clean accuracy and robust accuracy with LRR is 96.1 and 89.8 given by Augmix. Why with CARD-Deck, LRR can achieve a clean accuracy of 96.6 and robust accuracy of 92? It may be worth spending more space to explain the experiment setups and results.
    * Given the results in Table 2, the improvement brought by Adaptive CARD-Deck is not significant compared to the Agnostic counterparts. Therefore, what is the benefit of CARD-Deck? It would be better to clearly point it out in the paper.

**Clarity**:

The paper is not well organized, and the presentation of the second half (starting from section 4) needs improvement. The description of experimental results is too brief so that it is hard to tell what is the benefit brought by CARD-Deck. The theoretical justifications are also vague. It is not clear what insight or guidance they can provide. I don’t see a strong connection between the CARD approximation theorem and the empirical observations in the paper. It cannot guarantee the existence of CARDs. Also, it cannot explain why “lottery ticket-style” pruning approaches are able to improve the OOD robustness over dense models. Section 5.2 is also unclear. I would recommend putting some formal analysis to the main paper instead of leaving everything in the appendix.

**Significance**:

The idea of studying the existence of CARDs and comparing how different pruning methods affect the OOD robustness of compressed models are valuable and advance our understanding of the connection between model compactness, pruning methods, and OOD robustness. Also, the idea of selecting CARD given the spectral similarity metric is likely to be useful.

**Overall recommendation**:

My overall score for this paper is 4. As stated above, the paper asks important questions and provides valuable observations. However, the presentation needs improvement; the experiments need better design and explanation, and it is better to make the theoretical analysis more relevant to the main ideas of the paper.


**Updates to the Review post rebuttal**

I appreciate the authors' detailed responses to all the reviews. These responses addressed most of my questions and concerns. I have raised my rating.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper explores the possibility and practibility to build compact, accurate, and robust deep neural networks (CARDs). The authors first conduct extensive empirical studies to verify the existence of CARDs. Meanwhile, to better understand CARDs, authors further analyze the Fourier sensitivity in the spectral domain. Based on the empirical observations, a domain-adaptive ensembling approach (CARDs-Deck) is proposed as a practical algorithm to utilize the CARDs models, which is able to achieve new SOTA results for robustness and accuracy on CIFAR-10. The paper is quite informative and solid throughout the text, and the claims are well supported by empirical observations. There are also theoretical analysis that justifies CARDs in depth. 
main_review: 
Strengths:

* The paper explores an important problem with promising conclusions: the compact, accurate and robust deep neural networks do empirically exist.

* The authors not only answer the existence of CARDs, but also propose a practical approach to utilize CARDs to improve the accuracy and OOD robustness, which establish new SOTA performance with smaller model sizes on CIFAR-10-C.

* The methodologies involved in both empirical studies (e.g., Fourier sensitivity analysis) and the practical algorithm (domain-adaptive ensembling) are inspiring and novel.  


Weakness:

* While extensive experiments and analysis are conducted on CIFAR-10 and CIFAR-10-C, it remains unknown if CARD still exists on large scale datasets such as ImageNet and ImageNet-c. 

* It seems the theoretical results (e.g., Theorem 1 and Theorem 3) are less related to pruning, but are general to analyze the approximation and OOD robustness. It may be inspiring to show in what way the compactness (sparsity of network parameters) affect the CARD approximation or the upper bound of OOD robustness.


Detailed comments:

* How about the standard deviation of different methods in Figure 1 and 2? Some lines are highly over-lapped, which could be hard to draw firm conclusions. For example, it is not very clear that layerwise pruned models consistently achieve the maximum accuracy as mentioned in L157. 

* The improvement of domain-adaptive CARD-DECKs seems to be incremental, especially when more models are present, according to Table 2.

* In Table 1, there is mis-alignment of the header. Meanwhile, it is kind of misleading that baseline models have the same performance and memory under different pruning ratios in both Table 1 and Table 2.

* The authors can make the paper more self-contained when an additional page is allowed in the revision, especially for theoretical analysis Section 5.

limitations_and_societal_impact: Yes the authors have listed the limitations and potential societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors evaluate several pruning techniques and their impact on accuracy and robustness on CIFAR10. They perform a spectral analysis of Fourier domain errors and attempt to link those to OOD robustness. They propose an ensembling approach based on the spectral properties of unlabeled test images.
main_review: ### Pros:

I think the question of achieving robustness and efficiency at the same time is interesting and very relevant. I also think that this is an under-explored question and agree with the authors’ statement in line 53: “Perhaps surprisingly, the current solution in the robust ML community to improve the OOD robustness (and accuracy) is to increase the model size (e.g., [21, 13] and [6]).” It would be great if we had small and efficient models that are robust and accurate.

### Cons:

The authors oversell their results in all performed experiments. They speak of “drastic” gains, “dramatically more robust” models etc. Looking at the results, the robustness gains are on the order of 6-8 percent points for the smaller models and on the order of 1-2 percent points for the larger and thus, more interesting models. Given that their findings do not produce significant gains for their best model, I doubt that this analysis would actually scale to ImageNet. Since their small models with the larger gains have around 88% clean accuracy on CIFAR10, I do not find an analysis on those models particularly interesting, because the clean result is very far from the state of the art on CIFAR10, and the ultimate goal of the authors is to show a new state of the art on CIFAR10. Thus, they maybe should focus on using better models from the beginning.

Similar to the overselling of results, the authors seemed to have an “agenda”, a “hypothesis” of what they wanted to show. Irrespective of experimental evidence, they still claimed to see the desired outcome in the experiment, even if the experiment showed the exact opposite, which is a very dangerous and unscientific practice. 
Examples:

Line 156: “While layerwise pruned models consistently achieved the maximum accuracy (across prune percentages), the globally pruned models consistently provided significant robustness gains over the layerwise models. Furthermore, the globally-pruned models typically achieve higher or comparable accuracy at higher sparsity levels, indicating that using initialization methods with global pruning can yield CARDs with better accuracy and robustness at the same level of compactness.” I am confused. 1. The layerwise pruning methods (yellow) lose quite a lot of clean accuracy in the conv models (up to 10-12 percent points) while the global ones degrade the clean accuracy less. Thus, the first part of the first sentence “While layerwise pruned models consistently achieved the maximum accuracy (across prune percentages)” is wrong. 2. On both VGG and ResNet18, the global pruning methods show the same loss in clean accuracy as their robustness gain. Thus, the models are not “CARDS”, because they are less accurate than the baseline *for all considered pruning percentages*! Thus, the statement that “initialization methods with global pruning can yield CARDs with better accuracy and robustness at the same level of compactness” is incorrect. Furthermore, for the ResNet18, the gains/losses are on the order of 1-2 percent points which I would not call significant in the first place. This symmetry in losing clean accuracy and gaining the “same amount” in robustness makes me wonder whether there is some weird robustness-accuracy trade-off at play?

Line 186: “In particular, by comparing the center of the heatmaps for layerwise and global methods, it follows that initialization methods leveraging layerwise pruning are more robust to low frequency perturbations, which translates to a higher accuracy on the clean data.” -> This is plain wrong. We look at conv8 with 80% weights pruned. Conv8 at 80% pruned weights has a 2% lower clean accuracy for Edgepopup and maybe a 0.5% increase in clean accuracy for Biprop (both initialization methods). Virtually all the remaining models have lower clean accuracy compared to baseline at all pruning levels for layerwise pruning.


Line 189: “On the other hand, global pruning in initialization methods yields CARDs that are less sensitive to mid to high frequency perturbations when compared to layerwise pruned CARDs, which translates to a
higher robustness to the OOD data.” To me, the layerwise pruning heat maps definitely have higher robustness values in the mid-to-high range frequencies compared to the global pruning methods. Following the authors’ argumentation, layerwise pruning methods should then have higher OOD robustness compared to layerwise pruning methods which they don’t.

Line 216: “Our analysis in Section 3 highlights that the robustness of the compressed model on frequency 
perturbed data is crucially dependent on the interaction between the frequencies the model is biased towards and the frequencies on which the test-time corruption is concentrated.” This is wrong. The authors did not evaluate the individual corruptions in CIFAR-C, but only differentiated between clean (low frequency) and corrupted (high frequency) corruptions. Considering that CIFAR-C corruptions have very different frequencies, I find the analysis too coarse and the statement here far too bold.

The results in Table 1 look very strange. The vanilla trained model has the highest baseline accuracy which is significantly degraded by Gaussian noise training and Augmix. There must some optimization issue. Since the authors use an adaptive method, I would suggest comparing to TENT (https://arxiv.org/abs/2006.10726v1) who also use an adaptive technique at test time and show results on CIFAR-C. I would then either use the same architecture as them or reimplement their method on a ResNet50.

The results are only done on CIFAR10. I would suggest performing the analysis on ImageNet since this is the dataset robustness researchers, i.e. the target audience for this paper, are most interested in. I doubt robustness researchers would pay attention to this work if the analysis is only done on CIFAR10.
In general, the used models are very far from state-of-the-art models on CIFAR10. Checking the CIFAR10 leaderboard (https://paperswithcode.com/sota/image-classification-on-cifar-10), models with a clean accuracy of 94.4% are around place 104 in the leaderboard. The VGG net has a top-1 accuracy of 93.4%. Given that CIFAR10 is a very small dataset and thus, does not impose huge computation requirements, I would suggest using a state-of-the-art model. Even larger state-of-the-art CIFAR10 models can still be run on one GPU with a reasonable computation time.
Line 262: Gaussian noise is part of CIFAR-C. The authors overfit to the test set rendering their results meaningless. In any case, the performance gains over the baseline are moderate at best (2 percent points max).

This was a rough summary of my impression of the paper and a justification of my rating. Please see the general comments section for further comments.

### General comments:

Line 96ff. “F1 represents the function to find the masks, F2 represents the function to train the weights after pruning, ki is the earliest training iteration that Fi requires information from”, “Rewinding-based Lottery Tickets: k1 = T and k2 = r; r < T;” I am confused by this definition. I understand F2 to be the function to be trained after pruning. This means, F2 needs to know F1 (i.e. the masks). Then, the definition k1=T, k2=r; r<T does not make sense because this would lead to the opposite: We would first train the already pruned function at k2=r and then find the pruning masks at k1=T. Please rephrase this paragraph for clarity.
Which heuristic is used to determine the final network? Is it the accuracy on the clean training set?

Line 116: „For a reliable comparison, we make use of convolutional [9], VGG [39] and ResNet [16] style architectures of varying size and train them on CIFAR-10 [24]“. The distinction is unclear: VGG and ResNet architectures also involve convolutions. 

Line 125: “For each pruning method, we plot the mean accuracy or robustness value over all realizations relative to the mean of the dense baseline realizations with error bars extending to the minimum and maximum realization value at each of the aforementioned sparsity levels.” This sentence is very hard to parse, please rephrase. Also: unclear which “realizations”.
Does the y-axis of Figure 1 show percent points?

Line 134: “However, we find that rewinding and initialization based pruning approaches consistently produce dramatic gains in robustness relative to dense baselines while matching (and sometimes surpassing) the accuracy of the dense baseline.” -> “consistently dramatic gains” is a strong claim and for the ResNet18 which is a popular model for CIFAR10 and which is the one with the highest clean accuracy of 94.4% here, the gain seems to be at 2 percent points which is rather small.

Line 137: “In particular, the rewinding class of methods provide a consistent, moderate improvement to both accuracy and robustness while the initialization class provides more substantial gains in robustness even when the accuracy is slightly below the baseline accuracy.” -> The second statement about the initialization class strongly oversells: For the ResNet18, the drop in clean accuracy is actually larger than the gains in robustness! For the VGG network, we see a loss of 3% in clean accuracy and a gain of 6% in robustness.

Line 139: “The significance of overparameterization to finding highly compact CARDs using initialization methods is evident for all architecture types, as the robustness of these models in higher pruning regimes improves at increasing levels of parameterization.” I am confused by this sentence. A ResNet18 has more parameters than a Conv4, but sees fewer gains from pruning? Additionally, the gains diminish for all models in higher pruning regimes. Please rephrase/ explain.

Line 162: “drastically different levels of OOD” -> very bold statement. I would not call a gain of 2 percent points for the best and thus most interesting model drastic.

Line 173: “usually corresponding to clean data” -> what does “usually” mean here? Very vague, be more specific please.

Line 180: “Figure 3 illustrate how rewinding and initialization pruning methods reduce the error rate across the whole frequency spectrum of perturbations when compared to the dense model.” I do not see major differences for the rewinding based approaches. I would highly suggest to the authors to show ResNet18 results here since it is their best and therefore, it is the most interesting and relevant model. Looking at Figs. 8 and 9 and the ResNet18 heat maps, I do not see any differences between the dense heat map and the rewinding methods’ heat maps. -> Maybe it would be helpful if the authors also reported an aggregated value because a qualitative comparison merely by looking at the heat maps can be ambiguous.


line 191: “Finally, traditional pruning methods yield a Fourier heatmap with a slightly altered structure when compared to the dense baseline and it slightly resembles a higher error rate version of heatmaps for rewinding methods.” Very vague, using “slightly” twice in this sentence and I don’t agree. I don’t see structure differences I would consider significant here. As discussed above, I interpret this sentence as authors wanting to follow their agenda and over-interpreting their results.

I don’t understand what Figure4 is supposed to tell me. I don’t see significant differences between the different columns.

The results in Table 1 seem wrong: In the first column, “clean” training leads to a robust accuracy of 89.2% while Gaussian noise training only leads to 73.7%. Given that the authors overfit to Gaussian noise on CIFAR-C, the accuracy with Gaussian noise training should be much higher than with clean training. It also seems wrong that Augmix seems to degrade accuracy over clean training. In addition, the high accuracy of 89.2% achieved with clean training is surpassed by not even one percent point when using the ensembling strategy. I would suggest to the authors to mark the best value in bold to make parsing the table easier.

Section 5.1. I do not understand how Theorem 1 relates to the experiments/ the rest of the paper. None of the pruning methods are about binary networks, yet Theorem 1 considers binary networks. In addition, in theorem 1, the pruned network is twice as deep as the target non-compressed network which also doesn’t hold for any of the pruning approaches above. The theoretical considerations seem like an after-thought and did not get how they supported the other insights from the paper.

### Updates to the Review post rebuttal

I think the authors did a great job addressing my comments and participating in the discussion with me. I specifically appreciate them removing and modifying instances where their results are "oversold". I think their last response to my question as to how the Fourier analysis and the OOD robustness analysis are connected is very rigorous and should be reflected in the paper as was presented in their comment.  Due to their last comment, I now understand the connection much better and it does make sense to me to include the Fourier analysis even though a direct correspondence to CIFAR10-C is not given. I urge the authors to include all points from their last comment in the manuscript to avoid confusion similar to what I experienced.
I appreciate that they included results from the robustbench benchmark and that they show that they actually get SotA results with their method. As stated before, I think the main research question is interesting and important, that is how to design compact DNNs that are accurate and robust at the same time.

I really like the current rebuttal setting of NeurIPS and I think that the submission has been improved a lot due to the discussion. I wish it was possible to upload a new version of the paper. I would like to reread it with the new changes, as it is hard to judge whether any incosistencies remain. 

I have raised my rating.
limitations_and_societal_impact: The authors discussed limitations of their work. They did not discuss potential negative societal impacts of their work, but I would not expect this paper to lead to such.
review_title: null

Meta-review: This work studies whether or not model pruning can be leveraged to improve the out-of-distribution robustness of image models. The work notes that the benefits of pruning strongly depend on details on how pruning is applied. This observation is valuable in light of past literature which were largely unable to improve robustness via model pruning. Reviewers initially had a number of criticisms of the work. For example the question of why model compression sometimes improves robustness remained unanswered and also the work only provides experiments on variants of CIFAR-10 and CIFAR-100. However, the authors had several extended back and forth discussions with reviewers during the rebuttal period and made several improvements to the work while addressing a number of concerns that reviewers have. Although I am quite hesitant to accept a primarily empirical work with experiments only on CIFAR, I overall agree with reviewers that the provided experiments are quite through and recommend accepting the work.
Decision: Accept (Poster)