Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents an empirical study on the effect of pruning to the model performance on each class and example, which leads to a novel finding that it has disparate effects to each sample. Specifically, the authors have found out that examples that are affected the most by pruning are more difficult to classify even for the non-pruned network, due to low image quality, mislabeling, or being atypical from the class prototype, and performed a further human study to analyze the source of difficulty. Moreover, the authors performed an additional experiment, which shows that the sparse models are brittle against natural adversarial examples.

Pros
- The paper provides a novel insight on the effect of pruning at the class and the example level, which could lead to a more effective pruning approach that exploit this findings. 

Cons

- The paper only provides a novel finding but not the solution on how to tackle this problem, and thus the paper looks incomplete. After section 3.3, I was expecting to see some approaches to tackle this problem but the paper abruptly ended. 

- The effect of pruning could largely differ from one method to another, but the authors do not experimentally compare the effects of different pruning methods. Also, it is highly likely that the findings discussed in the paper may be only true for input-independent pruning approaches, and may not generalize to input-dependent pruning method. The authors need to perform extensive study of both input-dependent and input-independent pruning approaches to validate their points.

In sum, the paper provides a novel insight on how pruning affects the performance at the example level, but does not provide a solution, and the current set of experiments is insufficient to validate that the empirical findings that the authors report generalize to other types of pruning approaches, such as input-dependent pruning. Thus I believe that the paper is not ready for publication yet, and vote for rejecting this paper in its current form.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper study how the pruning impacts the images/classes and observes which images/classes do not generalize in the pruned networks under ResNet-50 with ImageNet dataset. The authors first observe whether there exist classes exhibiting higher/lower accuracy in pruned networks compared to unpruned ones. Then, the authors define images (PIE) that generate inconsistent predictions for unpruned/pruned networks and observe the properties of PIE (e.g., whether corrupted, contains multiple labels, etc.). In particular, PIE images could be considered as `hard examples’ since even the unpruned networks misclassify them w.h.p. The authors also verify the properties of PIE and conclude the paper.

Overall, I think the paper contains some interesting observations but not enough for publication yet, due to the following reasons. First, it is well known that hard (noisy or corrupted) images are harder to be correctly classified in networks of smaller model power (Hendrycks & Dietterich, 2019). That is, results in Figure 4 and Figure 5 are not very surprising. Moreover, there is no further discussion on the hypothesis test after presenting Table 1 and Figure 3. 

The paper is generally well written even it contains few typos and inconsistent presentations (e.g., legends in images are inconsistent. Both underbar and space are mixed.)

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper claims that neural network pruning methods have different impact on accuracy in class-wise and sample-wise. To this end, the paper performs statistical tests to identify such classes (and samples) from a population of neural network models of different levels of pruning. In particular, the identified samples, called PIEs, are shown to have significantly lower test accuracy, i.e., the PIEs are much harder to classify, and a user study is performed to support this claim. The paper also demonstrates that pruned models tend to have lower accuracy against adversarial attacks or common corruptions.  

In overall, the paper addresses an important problem of investigating the effects of pruning. The experiments performed here seems fairly extensive. One of my primary concerns, however, is that I am still not convinced whether the empirical findings presented here is indeed significant, as some reader might feel those results are not that surprising. I personally feel that it is much more likely that pruning gives heterogeneous effects over the classes, since current pruning schemes do not enforce the "uniform" brain damage, i.e., they simply aim to minimize total accuracy drops. Also, the accuracy differences in Table 1 do not appear to be that large in my opinion. The questions in what follows may help to resolve such concerns: 

- Section 2.1: For ImageNet dataset, data imbalance in training set might be the reason of such disparate impact? As CIFAR-10 dataset is balanced, I wonder if CIFAR-10 results could be also presented in more details.
- What if the whole training and pruning is re-performed after excluding the classes which the accuracy is decreased more by pruning? Would it lead to better pruning results, or better overall accuracy?
- I feel there should be more explanation about the actual statistical implication of the use of Welch's t-test: Is it ok that S^c_t may not be independent to S^c_0? What is the key benefits of not just picking outlier classes among S^c_t with respect to c?
- What would happens if we exclude the PIEs in training set, and performs training & pruning from scratch? The overall claims may be strenghten if this could improve the pruning efficiency.
- Figure 7: I think the pure ImageNet top-1 accuracy should be also presented in each plot as a baseline for fair comparison, as this accuracy will be decreased with respect to the sparsity as well.
review_title: Official Blind Review #3

Meta-review: This work investigates neural network pruning through the lens of its influence over specific exemplars (which are found to often be lower quality or mislabelled images) and how removing them greatly helps metrics.
The insight from the paper is interesting, as recognized by reviewers. However, experiments do not suggest that the findings shown in the paper would generalize to more pruning methods. Nor do the authors give directions for tackling the "hard exemplar" problem. Authors' response did provide justifications and clarifications, however the core of the concern remains.
Therefore, we recommend rejection.
Decision: Reject