Selective Brain Damage: Measuring the Disparate Impact of Model Pruning
Authors: Sara Hooker
Yann Dauphin
Aaron Courville
Andrea Frome
Venue:  (2020)
Keywords: machine learning

Abstract: Neural network pruning techniques have demonstrated it is possible to remove the majority of weights in a network with surprisingly little degradation to top-1 test set accuracy. However, this measure of performance conceals significant differences in how different classes and images are impacted by pruning. We find that certain individual data points, which we term pruning identified exemplars (PIEs), and classes are systematically more impacted by the introduction of sparsity. Removing PIE images from the test-set greatly improves top-1 accuracy for both sparse and non-sparse models. These hard-to-generalize-to images tend to be of lower image quality, mislabelled, entail abstract representations, require fine-grained classification or depict atypical class examples.

Under review as a conference paper at ICLR 2020
SELECTIVE BRAIN DAMAGE: MEASURING THE DIS- PARATE IMPACT OF MODEL PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network pruning techniques have demonstrated possible to remove the majority of weights in network with surprisingly little degradation to top- test set accuracy. However, this measure of performance conceals significant differences in how different classes and images impacted by pruning. We find that certain examples, which we term pruming identified exemplars (PIEs), and classes systematically more impacted by the introduction of sparsity. Removing PIE images from the test-set greatly improves top-1 accuracy for both sparse and non-sparse models. These toget-r-reeralzztttoo images tend to be mislabelled, of lower image quality, entail abstract representations, atypical examples or require fine- fme-grained classification.
INTRODUCTION
Between infancy and adulthood, the number of synapses in our brain first grow and then fall. Synaptic pruning improves efficiency by removing neurons that are redundant and strengthening synaptic connections that are useful for the environment (Rakic et al. 1994). Despite losing 50% of all synapses between age two and ten, the brain continues to function (Kolb & Whishaw 2009: (Sowell jet al. (2004). "Use it or lose it" is frequently used to describe the environmental influence of the learning process on synaptic pruning, however there is little scientific consensus what exactly is lost (Casey et al. 2000).
In this work, ask what is lost when we prune deep neural network. Work pruning deep neural networks has demonstrated a remarkable ability to sparsify model to fraction of the original
weights while giving up minimal test-set accuracy (Cun al. 1990, Hassibi al. 1993b Han et al.
2015;
Ullrich et al. 2017 Liu et al. 2017 Louizos et al. 2017]Colling & Kohli 2014 Weigend Hinton 1992 Lee et al. 2018b). ). Gale al. (2019) show is possible to prune 90% of all weights in ResNet-50 network (He et al. 2015) trained on ImageNet (Deng et al.
al.
1991 1991] Nowlan
2009) and only lose less than 3% absolute in top-1 test set accuracy. The ability to prune networks
with seemingly so little degradation to accuracy is puzzling. In this work, we address the following questions:
Are certain types of examples classes disproportionately affected by pruning?
How does pruning impact robustness such sensitivity image corruptions (blur, noise, contrast) and adversarial examples?
Answers for these question can provide intuition into the role of additional capacity in deep neural networks and, perhaps more important, provide a principled framework for articulating the trade- offs incurred by compressing deep neural networks. Many of the most promising use cases for compressed models occur in sensitive domains, such as improving access to health care by using machine learning driven diagnostics on mobile phones (Esteva al., 2017). Pruned or compressed models are frequently favored for deploying deep neural networks onto devices because reducing the number of network weights lowers energy consumption, memory footprint, and latency (Reagen) et al.) 2016 Chen et al. 2016 Theis et al. 2018 Kalchbrenner et al. 2018 Valin & Skoglund 2018). 8). For tasks where incorrect predictions can harm human welfare, it is critical that we understand when machine learning model is qualified to make decisions real world inputs. To knowledge. this the first work to shed light on the trade-offs pruning incurs by considering new measures beyond
test accuracy.

---

Under review as a conference paper at ICLR 2020
The primary findings of our work can be summarized follows:
1. In some settings, pruning has non-uniform impact across classes; a fraction of classes are disproportionately and systematically impacted by the introduction of sparsity.
2. The examples most impacted by pruning, which we term Pruning Identified Exemplars (PIEs), are more challenging for both sparse and non-sparse models to classify.
3. Pruning makes deep neural networks less robust to natural adversarial examples and common image perturbations (such as blur, fog and contrast).
For (T) and (2). we establish consistent findings for different standard architectures on CIFAR-10 10 (Krizhevsky) (2009) and ImageNet (Deng et al. (009). We identify Pruming Identified Exemplars (PIEs) as images where the modal label differs between a set of sparse and non-sparse models. We find that removing PIEs from the test-set improves top-1 accuracy for both a fully parameterized non-sparse model as well as a sparse model (Figure 4
Toward finding 3 we measure changes to model sensitivity to both common image corruptions and natural adversarial examples using two open source robustness benchmarks: ImageNet-C (Hendrycks) & Dietterich 2019) and ImageNet-A (Hendrycks et al. (211)). We find that both sparse and non-sparse models are brittle and sensitive to ImageNet-A and C, and that this brittleness is amplified at highers level of sparsity (Table
Our findings provide important insights about when pruned models are qualified to make decisions on real world inputs. Our PIE methodology identifies tractable subset of images which are more challenging for sparse and non-sparse models. Our work suggest that PIE could be useful for tasks where it important choose not classify certain examples when the model is uncertain (Bartlet)) & Wegkamp 2008 Cortes et al. 201 a Cortes et al. 2017), 7 to aid interpretability as a case based reasoning tool explain model behavior (Kim al. [0116] Gurumoorthy et al.) 2017 Caruana 2000 Hooker et al. 2018) or to surface atypical examples for further human inspection (eibiig et al. 2017)
A COMPARISON OF SPARSE AND DENSE MODELS
We consider a supervised classification problem where a deep neural network is trained to approximate the function F that maps input variable X to an output variable Y, formally F X + Y. The model is trained on a training set of N images = {(x; ) and at test time makes a prediction yf for each image in the test set. The true labels yı are each assumed to be one of C' classes, such that yi = [l,.....,,, CJ.
A reasonable response to our desire for more compact representations is to simply train network with fewer weights. However, as of yet, starting out with a compact dense model has not yielded competitive test-set performance. Instead, research has centered on training strategies where models are initialized with "excess capacity" which is then subsequently removed through a pruning process. A pruning method yP identifies the subset of weights to remove (i.e. set to zero). A sparse model function, fP. is one where fraction E [0, 1] of all model weights are to zero. Equating weight value to zero effectively removes the contribution of a weight as multiplication with inputs no longer contributes to the activation. A non-sparse model function, Fo. is one where all weights are trainable.
ImageNet and CIFAR-10 Setup We consider two classification tasks and models; a wide ResNet model ZZagoruyko& & Komodakis 2016) trained on CIFAR- 10 and a ResNet-50 model (He et al. 2015) trained on ImageNet, both using batch normalization (loffe & Szegedy 2015). A key goal of our analysis is to produce findings that are not anecdotal as would be the case when analyzing one trained model of each type. Instead we independently train a population of 30 models for each experimental setting. We train for 32, 000 steps (approximately 90 epochs) ImageNet with a batch size of 1024 images and 80, 000 steps with a batch size of 128 for CIFAR-10 10. For ImageNet, the baseline non-sparse model obtains mean top-1 accuracy of 76.68% and mean top-5 accuracy of 93.25% across 30 models. For CIFAR-10, 10, mean baseline top-1 1 accuracy is 94.35%.
We prune over the course of training to obtain target end sparsity level (0.0, 0.1,0.11..33 0.3, 0.5, 0.7, 0.9). For example, = 0.9 indicates that 90% of model weights are removed by pruning, leaving a maximum of 10% non-zero weights.
2

---

Under review as a conference paper at ICLR 2020
50%
50%
5276
10%
50%
370 675 834
0.70 0.70
332 0.92
Imageeett Top-1 Accuracy
Imageltet top-5 wecurary
CIFAR-10 top-1 acturaty
Figure I: Distributions of top-1 and top-5 model accuracy for populations of independently trained sparse and non sparse models on ImageNet and CIFAR-I0. The distributions for CIFAR-10 top-5 accuracy (not shown) are tightly clustered and overlapping in [0.9965. 1.0]. The distributions are fairly tight with one outlier for the ImageNet baseline model.
true label:
dog
airplane
airplane
automabile
truck
harse
non-sparse:
dog
airplane
airplane
automobile
automobile
cat
sparse:
ship
ship
truck
horse
Figure Visualization pruming identified exemplars (PIE30) for the CIFAR- 10 dataset. This subset of impacted images is identified by considering set of 30 non-sparse wide ResNet models and 30 models trained 30% sparsity.
Across all experiments, the weights to remove. Magnitude pruning is a simple rule-based method that thresholds weights at zero that fall below a certain absolute magnitude. It has been shown outperform more sophisticated Bayesian pruning methods and is considered state-of-the-art across both computer vision and language models (Gale al. 2019). The choice of magnitude pruning also allowed us specify and precisely vary the final model sparsity for purposes of our analysis, unlike regularizer approaches that allow optimization process itself to determine the final level of sparsity (Liu et al. 2017 Louizos et al., 2017 Collins & Kohli 2014 Wen et al.) 2016 Weigend et al. 1991; Nowlan & Hinton 1992). Although the ability to precisely vary sparsity is required for this experimental framework, we note that methodology be extended to other methods. Figure I shows the distributions of model accuracy across model populations for the non-sparse and sparse models for ImageNet and CIFAR- 10.
magnitude pruning proposed by Zhu & Gupta (2017) identify
2.1 CLASS LEVEL PERFORMANCE
Comparisons of the effects of different pruning algorithms levels of sparsity classification tasks such as ImageNet have largely centered on top-line metrics such as top-1 or top-5 test-set accuracy averaged across classes, but reliance on these measures hides detail in the model's performance. Comparing only top-1 model accuracy between baseline and pruned model amounts to assuming that class accuracies are expected to maintain their relative relationships to the top-1 model accuracy before and after pruning. In this work, we consider whether this is valid assumption. Is relative performance unaltered by pruning or are some classes impacted more than others?
For given model, we compute the class accuracy 8 for class E C and sparsity E [0, We compute overall model accuracy from the set class metrics:
M.M
Σ
3 * No
NM
GEC
where N₂ is the number of examples in class c and NM is the total number of examples in the data set. If the impact of pruning uniform, would expect each class accuracy shift by the same number percentage points as the difference in top-1 accuracy between the sparse and non-sparse model. This forms null hypothesis (Ho) the shift in accuracy for class before and after
3

---

Under review as a conference paper at ICLR 2020
pruming is the same the shift in top-1 accuracy. For each class consider whether to reject Ho and accept the alternate hypothesis (H1) that pruning disparately affected the class's 's accuracy in either positive or negative direction:
Ho : 56 30 B - 1t
H1 : 56 3 * B 3M
Neural net training is most often done in a non-deterministic fashion, and we consider each model k in its population of K 30 models to be sample of some underlying distribution. Given class and a population of K' models trained at a sparsity i, we construct a set of samples Sç of the mean-shifted class accuracy S { -
Evaluating whether the difference between a sample of mean-shifted class accuracy from sparse and non-sparse models is "real" amounts determining whether two data samples are drawn from the same underlying distribution, which is the subject of a large body of goodness of fit literature (D' D'gostnnno Stephens, 1986 Anderson Darling 1954] Huber-Carol al., 2002). In this work, we use two-sample, two-tailed, independent Welch's s 1-test (Welch) 1947) to determine whether the means of the samples Sj and Sj differ significantly. If the two samples were drawn from distributions with different means with 95% or greater probability (p-value <= 0.05), then we reject the null hypothesis and consider the class to be disparately affected by t-sparsity pruning relative to the baseline.
After finding the subset of classes for a given t-sparsity that show statistically significant change relative the baseline, can quantify the degree of deviation, which refer normalized recall difference, by comparing the average t-sparse and baseline class accuracies after normalizing for their respective average model accuracies:
K
Ko
(BE
(66.
K
Kg
2.2 PRUNING IDENTIHEED EXEMPLARS
How does pruning impact model performance on individual images? A natural extension of the hypothesis testing in the prior section is consider whether to reject or retain the null hypothesis that the output probability for given image for a dense and sparse models is equal. However, recent work has highlighted that deep neural networks produce output probabilities that are uncalibrated (Guo iet al.] 2017 Kendall & Gal 2017 Lakshminarayanas et al. (011)) and thus cannot be interpreted as measure certainty. Deep neural networks do not know what they do not know, and often ascribe high probabilities to dist--iiittiitionn data points or are overly sensitive to adversarially perturbed inputs (Hendrycks & Gimpel| 2016 Nguyen et al.) 2014).
We are interested in how model predictive behavior changes through the pruning process. Given limitations uncalibrated probabilities deep neural networks, we focus on the level of disagreement between the predictions of sparse and non-sparse networks on a given image. Let vi,,, be the prediction of the keth t-sparse model of its population for image where = 0 denotes an unpruned model, and let Y [V. 1 1 be the set of predictions for the t-sparse model population on exemplar For set Y, we find the modal label, i.e. the class predicted most frequently by the t-sparse model population for exemplar i, which we denote y Exemplar is classified as PTE, and only if the modal label is different between the t-sparse model and the unpruned (non-sparse) model:
PIE., = {1
if v + v. otherwise
We note that there is no constraint that the non-sparse predictions for PIEs match the true label, thus the detection of PIEs is unsupervised protocol that could in principal be performed test time.
4

---

Under review as a conference paper at ICLR 2020
Sparsity (t)
Model accuracy
Significant
Largest increase
Largest decrease
diff.
incr.
# decr.
class
norm. diff.
class
norm. diff.
0.1
-0.0002
29
22
toaster
0.025
am. chameleon
-0.026
0.3
-0.002
35
34
bathtub
0.036
cleaver
-0.045
0.5
-0.008
91
54
petri dish
0,034
frying pan
-0.047
0.7
-0.017
189
128
player
0.050
tow truck
-0.069
0.9
-0.041
337
245
cd player
0.088
muzzle
-0.128
Table Summary of class-level results for ImageNet. Only classes passing the significance test are included. The model accuracy difference column reports mean B& - 36 the percentage point difference between the pruned and baseline model accuracies; a negative value means the pruned model's average accuracy is lower than the baseline model's.
% point difference
0.05
0.00
-0.05
1ow truck
emersccn chamelzon
stale
pajamo bescon
mareca
man
narvestn lock
crssmnn nn
wn noolin chis clmmbr cogg
you
steee spote dee mee
sezze boscige ed player
myvcrrb hermit
chime
band le eside
con convembeble aopenr
crich
ni Mopplppa crealii awred cord
corc bolere
combinae
youn
Figure 3: Normalized recall difference (bars) and absolute recall difference (points) per class for 70% sparsity (left) and 90% sparsity (right). The class labels are sampled for readability; there are 317 significant classes for 70% sparsity and 582 significant classes for 90%. Note the difference in scale on the y-axis.
RESULTS AND DISCUSSION
3.1 IMPACT OF SPARSITY ON CLASS LEVEL PERFORMANCE
The impact of magnitude pruning on ImageNet classification is disparate classes and amplified as sparsity increases. For example, at 10% sparsity only 51 of 1,000 classes in the ImageNet test exhibit statistically significant change in class accuracy, however at 90% sparsity, accuracy is impacted for 582 classes in a statistically significant way.
The directionality and magnitude of the impact is nuanced and surprising. Our results show that certain classes are relatively robust to the overall degradation experienced by the model whereas others degrade in performance far more than the model itself. This amounts selective "brain damage" with performance on certain classes evidencing far more sensitivity to the removal of model capacity. Table [ shows that more classes show significant relative increase accuracy than decrease at every level, though the overall model accuracy decreases at every pruning level, indicating that the magnitude of class decreases must be larger in order to pull the model accuracy lower. Figure visualizes the magnitude of the normalized recall differences for 70% and 90% pruning and highlights the degree to which the classes spread from model average.
We performed the same analysis the CIFAR-10 models and found that while pruning presents non-uniform impact there are fewer classes that are statistically significant. One class out ten was significantly impacted 10%, 30%, and 50%, and two classes were impacted at 90%. We suspect
5

---

Under review as a conference paper at ICLR 2020
ImageNet Top-1 PIE Non-PIE
CIFAR-10 Top-1 PIE Non-PIE
100
100
Top- test set accuracy (%)
Top-1 test set accuracy [%)
80
80
60
60
40
40
FF ynages
tmaeca
PIE
PIE
PIE
PIEn
PiE.
PIL.
PIE
PI
PIE..
PiE
Figure 4: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both Ima-
geNet and CIFAR-10. This holds for PIE images identified at all levels of spansity considered. Sparse model find generalizing PIE more challenging, and inference on PIE images alone substantially degrades generalization performance. Left: Average top-1 test-set accuracy across 30 non-sparse ResNet-50 models when inference is restricted to PIE images (blue), non-PIE images (dark purple) and random sample of the test set (black line) which is constant independent of PIE sparsity level. Right: Average top-1 test-set accuracy across 30 non-sparse wide ResNets trained on CIFAR-10.
that we found less disparate impact for CIFAR- 10 because, while the model has less capacity, the number of weights is still sufficient to model the limited number of classes and lower dimension dataset.
3.2 IMPACT OF SPARSITY ON INDIVIDUAL EXEMPLARS
At every level of sparsity, for both CIFAR-10 10 and ImageNet we identify a subset of PIE images. At 10% and 90% sparsity, classify 3.34% and 10. 10.27% of all ImageNet test-set images respectively as PIEs. For CIFAR-10, PIEs constitute 0.97% and 2.16% of the test set at 10% and 90% sparsity.
PIES are more difficult for both sparse and non-sparse models to classify. In Fig. we compare the test-set performance of a fully parameterized non-sparse model on a fixed number of randomly selected (1) PIE images, (2) non-PIE images and (3) random sample of the test set. The results are consistent across both CIFAR-10 and ImageNet datasets; removing PIE images from the test- improves top-1 accuracy for both sparse and non-sparse models relative to a random sample. Inference restricted to only PIE images greatly impairs the generalization performance of model. In the appendix, we include additional plots that show that while all models perform far worse on PIE images, the degradation to performance is amplified model sparsity increases.
The most challenging PIEs are identified at low levels of sparsity. Figure accuracy for both sparse and non-sparse models occurs when inference is restricted to PIEs identified 10% sparsity. Test-set accuracy steadily increases for PIEs identified at higher levels of sparsity, This suggests that sparsity first erodes performance the images that the model finds the most challenging.
4 the lowest test-set
Why are PIEs impacted more by pruning? A qualitative inspection of PIEs (Figure suggests that these hard-to-geneene toneerarlllll images tend to be lower image quality, mislabelled, entail abstract representations, require fine-grained classification or depict atypical class examples. We conducted limited human study"too to inspect a random sample of 400 PIE and non-PIE ImageNet images. We broadly group the properties codify as indicative of 1) the exemplar being challenging 2) the task being ill-specified. We introduce these groupings below (after each bucket we report the percentage of PIES and non-PIEs in each category as fraction of total PIEs and non-PIE codified);
1. Poorly specified task
'The authors acted as labelers ahead of the deadline, though the PIE label was shuffled and hidden during laheling. We will reproduce the study using other parties before publication.

---

Under review as a conference paper at ICLR 2020
NON_PIE
PIE
1) challenging exempiar poorly specified task both 1) and
9%
neither
55%
40%
Figure 5: Limited human study the relative distribution PIE and non-PIE properties. Challeng- ing exemplars: images positively codified as showing common image corruptions such blur or overlaid text, or images where the object is in the form an abstract representation or where the exemplar requires fine grained classification. Poorly specified task: images where multiple classes are visible in the same image, or images with incorrect or insufficient ground truth.
incorrect ground truth insufficient information images where there is sufficient information for a human to arrive at the correct ground truth label. For example in Fig. 6 the image of the plate of food with the label restaurant is cropped such that it is impossible to tell whether the food is in a restaurant or in different setting. [2.9% of non-PIEs, 9.5% of PIEs]
multiple labelled classes are clearly visible in the image-imaged depicting multiple objects where a human may consider several labels to be appropriate (e.g.. an image which depicts both paddle and canoe, desktop computer consisting of s screen, mouse and monitor, a barber chair in a barber shop). [22.3% of non-PIE, 50.8% of PIEs]
2. Challenging Exemplars
fine grained classification task involves classifying an object that is semantically close various other classes present the data set (e.g., rock crab and fiddler crab, bassinet or cradle, cuirass and breastplate), [8.3% of non-PlEs, 38.2% of PIEs]
exemplar corruptions images exhibit common corruptions such as motion blur, contrast, pixelation. We also include this category images with super-imposed text, artificial frame and images that black and white rather than the typical RBG color images in ImageNet. [8.7% of non-PIE, 12.1% of PIE]
exemplar abstractions the surfaced exemplar depicts a class object in an abstract form such cartoon, painting, or sculptured incarnation of the object. [1.5% of non-PIE, 3.5% of PIE]
We find that PIES overindex relative to non-PIEs on certain properties, such as having an incorrect ground fruth label or multiple objects. This suggests that the task itself is often incorrectly specified. Both ImageNet and CIFAR-10 are single image classification tasks, however just over half of the PIEs codified by humans were identified as multi-object images where multiple labels could be considered reasonable (vs. 22.33% of non-P The over-indexing of incorrectly structured data in PIE hints that the explosive growth in number of parameters in deep neural networks may be solving problem better addressed in the data cleaning pipeline. However, task mis-specification alone does not fully explain the images where predictive behavior diverges as model capacity is changed. We find that PIE also overindexes on low quality images that present corruptions like blur or overlaid text and exemplars that require fine grained classification This suggests that there may be relationship between number of model parameters and generalization these types of images. We explore the relationship between capacity and robustness to these types of corruptions more thoroughly in Section 3.3
7

---

Under review as a conference paper at ICLR 2020
multi-object image
image corruptions
tennis ball non-sparse tennis ball sparse : racket
true:
projectile
lynx labrador retriever nematode
tricycle tricycle shopping cart
missile
projectile
ground truth label inadequate
fine-grained classification
abstract representation
restaurant non-sparse iie cream sparse: : cauliflower
cup coffee mug
cowbay cowboy hat plastic bag
grocery store crayfish
Figure 6: Visualization of P9o::: images where modal label differs between 30 non-sparse ResNet- 50 models and 30 models trained to 90% sparsity on ImageNet. qualitative inspection of PIEs suggests that there certain shared properties between the images most impacted by pruning; these images tend to be of lower image quality and frequently exhibit image corruptions (overlaid text, contrast, motion blur, zoom), mislabelled or unintuitive label, depict class in abstract form, require fine-grained classification or depict atypical class examples.
3.3
THE ROLE OF ADDITIONAL CAPACITY
The PIE procedure surfaces exemplars that are harder for both sparse and non-sparse models to classify. Given that PIE surfaces data points where there the greatest divergence in behavior between sparse and non-sparse models, it is useful to understand the directionality of some of the properties described in the previous section. For example, we noted that PIE frequently surfaces images with corruptions such as motion blur, contrast and pixelation. Are sparse models more brittle to these type of corruptions? Many PIEs are often atypical or unusual class examples. We have already noted that model degradation when restricted inference on PIEs is amplified sparsity increases. Does this measure of model brittleness mirror other open source robustness benchmarks?
ImageNet-CC ImageNet-C (Hendrycks Dietterich 2019) is open source data that consists of algorithmically generated corruptions (blur, noise) applied to the ImageNet test-set. We compare top-1 accuracy given inputs with corruptions of different severity. As described by the methodology Hendryeks & Dietterich (2019), we compute the corruption error for each type of corruption by measuring model performance rate across five corruption severity levels (in our implementation, we normalize the per-corruption error by the performance of the sparse model on the clean ImageNet dataset). ImageNet-C corruption substantially degrades mean top-1 accuracy of non-sparse models Fig. 7 This sensitivity is amplified at high levels of sparsity, where there is a further steep decline in top-I accuracy. Sensitivity to different corrptiion remarkably varied, with certain corruptions such gaussian, shot an impulse noise consistently causing more degradation.
ImageNet-A ImageNet-A is a curated test set of 705 natural adversarial images designed to produce drastically low test accuracy. We find that the sensitivity of sparse models to ImageNet-A mirrors the patterns of degradation to ImageNet-C and sets PIEs. As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models more brittle to adversarial examples.
8

---

Under review as a conference paper at ICLR 2020
8
Top 1 & Accuracy ImageNet-A uy aocurey 20.5,00curoyy
50
Top-1 Accuracy ImageNet.C
7
4-F 02 schs
45
yl Jutee yey
accuracy %
5
accuracy
40
4
3
35
2
30
0.0
0.2
1.0
25
0.1
02
u.
0.3
0.9
Model Trained to Sparsity
Model Trained to Sparsity
Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity,
models are also more brittle common image corruptions. We measure the relative Top-1 I and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.
4 RELATED WORK
Model compression is diverse and includes research directions such as reducing the precision or bit
size per model weight (quantization) (Courbariaux et al. 2014 Hubara et al. 2016 Gupta et al. 2015).
efforts to start with network that is more compact with fewer parameters, layers computations (architecture design) (Howard et al., 2017 landola et al. 2016 Kumar et al. 2017) and student networks with fewer parameters that learn from larger teacher model (model distillation) (Hinton iet al. 2015) and finally pruming by setting a subset weights or filters to zero (Louizos et al. [2017]
Wen al. 2016, Cun et al. 1990 Hassibi et al.) 1993a] Strm 1997, Hassibi et al.) 1993b Zhu & Gupta 2017 See et al.) 2016 Narang et al. 2017). Articulating the trade-offs of compression has overwhelming centered on change overall accuracy. Our contribution, while limited in scope to model compression techniques that prune deep neural networks, is the first work to our knowledge to consider the limitations of these aggregate performance measures and demonstrate that the impact of pruning in deep neural networks at a class and exemplar level is non-uniform.
We also consider how pruning impacts robustness to natural adversarial examples and image corrup- tions. We note that recent work by (Hendrycks & Dietterich) 2019: Hendrycks et al. 2019) considers complimentary variant of this question by benchmark ImageNet-A and ImageNet-B robustness across limited set of dense non-sparse architectures with different numbers of parameters (for example ResNet-50 vs Ro1).t-101). While our work is focused on understanding the impact of sparsity on an exemplar and class level, one of our key findings is that PIE is far more challenging to classify for both sparse and non-sparse models. Leveraging this subset of data points for interpretability purposes or to cleanup the dataset fits into broader and non-overlapping body of literature that aims to classify input data points as prototypes "most typical" examples of a class - ((Carlini et al. 2018 Stock & Cisse 2017)) or outside of the training distribution (OOD) (Hendrycks & Gimpel| 2016 Lee et al. 2018a Liang et al. 2018 Lee et al. [2018a] Sa Masana et al. [2018 8) and work on calibrating deep neural
network predictions (Lakshminarayanal al. 2017 Guo et al. 2017 Kendall & Gal 2017)
CONCLUSION
We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance. Pruning identified exemplars are a subset of exemplars where there is a high level of disagreement between sparse and non-sparse models. We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

---

Under review as a conference paper at ICLR 2020
REFERENCES
T. W. Anderson and D. A. Darling. A test of goodness of fit. Journal of the American Statistical Association, 49(268);7557665 1954. ISSN 01621459. URL http:/ www. jstor. org/ stable/ 22/1/32111333
Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Mach. Learn. Res., 9:1823-1 1840, June 2008. ISSN 1532-4435. URL http://dl. acm. org/ citation. wwt.:cccmmmmm.mcc..
442792
Nicholas Carlini, Ülfar Erlingsson, and Nicolas Papernot. Prototypical examples in deep learning:
Metries, characteristics, and utility. 2018.
Rich Caruana. Case-based explanation for artificial neural nets. In Helge Malmgren, Magnus Borga, and Lars Niklasson (eds.), Artificial Neural Networks in Medicine and Biology, pp- 303-308, London, 2000, Springer London. ISBN 978-1-4471-0513-88
B.J. Casey, Jay N. Giedd, and Kathleen M. Thomas. Structural and functional brain devel- opment and its relation to cognitive development. Biological Psychology, 54(1):241 - 257. 2000. ISSN 0301-0511. doi: http:///aiiooog/o/../...ss..0010
URL http
www.scieenae
rect com http:////c.eeeeeee.....ee//oooo0
Y. Chen, J. Emer, and V. Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 367-379, June 2016. doi: 10. 00.109//SCA22016.4000
M. D. Collins and P. Kohli. Memory Bounded Deep Convolutional Networks. ArXiv e-prints, December 2014.
Maxwell D. Collins and Pushmeet Kohli. Memory bounded deep convolutional networks. CoRR, abs/1412. 1442, 2014. BBL/:stp.//aaxdv.orr/ab/////////
1442
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. 2016a. URL
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In D. D. Lee,
M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1660-1668. Curran Associates, Inc., 2016b. URL httt:://paper/ pdf
Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Scott Yang. Online Learning with Abstention. arXiv e-prints, art. arXiv:1703.03478 Mar 2017.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv e-prints, art. arXiv: 1ww.1141.7024 Dec 2014.
Yann Le Cun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598-605. Morgan Kaufmann, 1990.
R. B. D' Agostino and M. A. Stephens. Gfodness-offfit techniques. 1986.
Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Andre Esteva, Brett Kuprel, Roberto Novoa, Justin Ko, Susan Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-levee classification of skin cancer with deep neural networks. Nature, 542, 01 2017. doi: 10. 1056
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, abs/1902.09574, 2019, URLhttpp /tttt:://aaxii/ org/ abs//902/ 09574
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. arXiv e-prints, art. arXiv: 706.04599, Jun 2017.
10

---

Under review as a conference paper at ICLR 2020
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. CoRR, (bbs/1608.04493, 2016. URL http:://axxi......m/a//a// http:/ii.org/b////1Sooa aaa/abb//565 04493
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. CoRR, abs/1502.02551, 2015. URL URL http:://axxiv..r3/// abs/
1502. 02551
Karthik S. Gurumoorthy, Amit Dhurandhar, Guillermo Cecchi, and Charu Aggarwal. Efficient Data Representation by Selecting Prototypes with Importance Weights. arXiv e-prints, art. arXiv:1707.01211 Jul 2017.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both Weights and Connections for Efficient Neural Network. NIPS, pp. 1135-1143, 2015.
B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference Neural Networks, pp. 293-299 vol.1, March 1993a. doi: 10.1 109/CNN. WWWICNN.119932000 1993.9987......
Babak Hassibi, David G. Stork, and Stork Cre. Ricoh. Com. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems pp. 171. Morgan Kaufmann, 1993b.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. ArXiv e-prints, December 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations, 2019. URL hRRlhttts:////pneee//
mnet/forumti 6t iCqYm
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv e-prints, art. arXiv:1610.02136. Oct 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. arXiv e-prints, art. arXiv:1907.07174 Jul 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv e-prints, art, urXiv.1503.0253. Mar 2015.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. CoRR, abs/1806. 10758, 2018 URL /ttphht::///xxi..coo http::://aaaii.oor/aa//////////s wttp:rggg/bbb//oooe///////
10758
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Andreetto, and H. Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. ArXiv e-prints, April 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. CoRR, abs/ 10,00669,,7 2016. URL
07061
Huber-Carol, N Balakrishnan, M S. Nikulin, and Mounir Mesbah. Goodness-off of-rssss---FF---ttt-oo Tests and Model Validity. 01 2002. ISBN 978- 18811-4612-6113-6 doi: 00.1077978---4461-00000
F. N. landola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and 0.5MB model size. ArXiv e-prints, February 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03 2015. URL. /ttt:://arxi..... org/
abs abs/1502. 03167
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient Neural Audio Synthesis. In Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmässsnn Stockholm, Sweden, July 10-15, 2018, pp. 2415-2424, 2018.
11

---

Under review as a conference paper at ICLR 2020
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5574-5584. Curran Associates, Inc., 2017.
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. In D. D. Lee, M. Sugiyama, Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2280- Curran Associates, Inc., 2016.
B. Kolb and L.Q. Whishaw. Fundamentals of Human Neuropsychology. A series of books in psychology. Worth Publishers, 2009. ISBN 9780716795865. URL htttt:://bbokk:ee googl Com/bookst 20TT2NN00
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Ashish Kumar, Saurabh Goyal, and Manik Varma. Resource-efficient machine learning in 2 KB RAM for the internet of things. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1935-1944, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
mir wwt.semwww..pp...eeeeeeeeeeeeeee www.s/eessseooo https////o////aral/aaaaaam kumarl a html
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive
uncertainty estimation using deep ensembles. In Proceedings of the 3 International Confer- ence on Neural Information Processing Systems, NIPS' 17, pp. 1405-6416 USA, 2017. Curran Associates Inc ISBN 100-1----1100666-6 URL http: a1. acm. org/ citat ion. cfm?
adb3399522.3225338
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training monfidenne-calibratee classifiers for detecting duttof-distribution samples. In International Conference on Learning Representations, 2018a. URL httptttss////peneeeeeeese
net. ft////ormmm/ de Av2xAZ
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. SNIP: single-shot network pruning based on connection sensitivity. CoRR, abss/1810.02340, 2018b. URL http::
abs 0. 02340
Christian Leibig. Vaneeda Allken, Murat Seckin Ayhan, Philipp Berens, and Siegfried Wahl. Lever- aging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7, 2017. doi: 10. 10003///1118-007-117777
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018 8. URL https:/ / http///ppp///ee/eeee/ee/ee/ee ew net/ ffte//ooumm/mm id=i1vda IxRZ
Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning Efficient Convolutional Networks through Network Slimming. ArXiv e-prints, August 2017.
C. Louizos, M. Welling, and D. P. Kingma. Learning Sparse Neural Networks through L.0 Regular- ization. ArXiv e-prints, December 2017.
Mare Masana, Idoia Ruiz, Joan Serrat, Joost van de Weijer, and Antonio M. Lopez. Metric Learning for Novelty and Anomaly Detection. arXiv e-prints, art. arXiv: NEV.:1808.0442.. Aug 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity Inspired by Network Science. Nature Communications, 2018.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring Sparsity in Recurrent Neural Networks. arXiv e-prints, art. arXiv:1704.0511 Apr 2017.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. arXiv e-prints, art. arXiv:1412.1888 Dec 2014.
12

---

Under review as a conference paper at ICLR 2020
Steven J. Nowlan and Geoffrey E. Hinton. Simplifying neural networks by soft weight-sharing
Neural Computation, 4(4): 493, 1992. doi: 10.1162/mcco.1192444444733 URL https
/ttp...o//...116/// http:/eccc........ 4..
Pasko Rakic, Jean-Pierre Bourgeois, and Patricia S. Goldman-Rakin Synaptic development of the cerebral cortex: implications for learning, memory, and mental illness. In J. Van Pelt, M.A. Corner, H.B.M. Uylings, and F.H. Lopes Da Silva (eds.), The Self-Organizing Brain: From Growth Cones to Functional Networks, volume 102 of Progress in Brain Research, pp. 227 - 243. Elsevier, 1994, doi:
URL TRLpttp:: www sciencedirect
B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hemndez-Lobato, G. Wei, and Brooks. Minerva: Enabling low-power, highly-accurate deep neural network accelerators. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Archttecuree (ISCA), pp- 267-278, June 2016. doi: 10.1109/ISCAA2211....
Abigail See, Minh-Thand Luong, and Christopher D. Manning. Compression of Neural Machine Translation Models via Pruning. arXiv e-prints, art. arXiv:1606.09274 Jun 2016.
Elizabeth R. Sowell, Paul M. Thompson, Christiana M. Leonard, Suzanne E. Welcome, Eric Kan, and Arthur W. Toga. Longitudinal mapping cortical thickness and brain growth in normal children. Journal of Neuroscience, 24(38):8223 8231, 2004. doi: (0.1523/NEUROSCC119900000
URL https: /topps:/wwwww http://www.ooooo..oooooootttttee httpo//ooeoooottee/////////5///
Pierre Stock and Moustapha Cisse. ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases. arXiv e-prints, art. arXiv:1711.11445) Nov 2017.
Nikko Strm. Sparse connection and pruning in large dynamic artificial neural networks, 1997.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and Fisher pruning. CoRR, faos/1801.05780 2018. URL hRR.htttp://axxiv.orr
abs/1801.0577 05787
Karen Ullrich, Edward Meeds, and Max Welling. Soft Weight-Sharing for Neural Network Compres- sion. CoRR, ftss/7702.44008 2017.
Jean-Marc Valin and Jan Skoglund. Lpenet: Improving Neural Speech Synthesis Through Linear Prediction. CoRR, abs/18101 1184,, 2018. URLLattp://axxx..orr//b////
11846
Andreas Weigend, David E. Rumelhart, and Bernardo A. Huberman. Generalization by weight- elimination with application to forecasting. In R. P. Lippmann, J. F. Moody, and D. S. Touretzky (eds.), Advances in Neural Information Processing Systems 3, pp. 875-882. Morgan-Kaufmann, 1991.
B. L. Welch. The generalization of 'Student's' problem when several different population variances are involved. Biometrika, 34:28-35, 1947. ISSN 0006-3444. doi: 10.2307/2332510. URL
http::://doo.oooooo /ttp://...207/23355555
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning Structured Sparsity in Deep Neural Networks. ArXiv e-prints, August 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/ 1605.07011111 146, 2016. URL http:://axxi....t// absp///...111... 07146
M. Zhu and S. Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. ArXiv e-prints, October 2017.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. CoRR, 171001878 2017. htt.htcc///aarii.orr///////////0
13

---

Under review as a conference paper at ICLR 2020
APPENDIX
6.1 MAGNITUDE PRUNING
There are various pruning methodologies that use the absolute value of the weight as way rank importance and remove from the network weights that are below a user specified threshold. This often over the course of training: training is punctuated at certain pruning steps and fraction of weights are set to zero. Many different magnitude pruning methods have been proposed (Collins) Kohli 2014 Guo et al., 2016 Zhu Gupta 2017) that largely differ whether the weights are removed permanently or can "recover" by still receiving subsequent gradient updates. This would allow certain weights to become non-zero again pruned incorrectly. While magnitude pruning is often used as a criteria to remove individual weights, it can be adapted to remove entire neurons filters by extending the ranking criteria to a set of weights and setting the threshold appropriately. Recent work evolutionary strategies has also leveraged an interative version of magnitude pruning (Mocanu et al. 2018).
In this work, use the magnitude pruning methodology proposed by Zhu & Gupta (2017). Pruning is introduced over the course of training and removed weights continue to receive gradient updates after being pruned. For ImageNet, each model trains for total of 32,000 steps. We prune every 500 steps between 1,000 and 9,000 steps. For CIFAR-10, we train the model for 80,000 steps. We prune every 2,000 steps between 1,000 and 20,000 steps. These hyperparameter choices were based upon limited grid search which suggested that these particular settings minimized degradation to test-set accuracy across all sparsity levels. the end of training, the final pruned mask is fixed and during inference only the remaining weights contribute to the model prediction.
6.2
ADDITIONAL CLASS-LEVEL RESULTS
Tables 2 and provide top-line metrics for ImageNet and CIFAR-10, respectively.
The class-level analysis summary for CIFAR-10 is in Table 6.2 Relative to ImageNet, the percentage
of classes significantly impacted at 90% pruning small: 20% for CIFAR- 10 versus 58% for ImageNet. As discussed in the main body, we suspect this is due to CIFAR-10 being simpler task and the network we started from having much more capacity than necessary for the task.
Fraction Pruned
Top
Top 5
# Signif classes
# PIEs
0
76.68
93.25
0.10
76.66
93.25
51
1,694
0.30
76.46
93.17
69
1,819
0.50
75.87
92.86
145
2,193
0.70
75.02
92.43
317
3,073
0.90
72.60
91.10
582
5,136
Table 2: ImageNet top-1 and top-5 accuracy at all levels of sparsity, averaged over all runs. The fourth column the number of classes significantly impacted by pruning.
Fraction Pruned
Top
Signif classes
# PIFs
0
94.53
0.1
94.51
1
97
0.3
94.47
1
114
0.5
94.39
1
144
0.7
94.30
0
137
0.9
94.14
2
216
Table 3: CIFAR-10 top-1 accuracy at all levels of sparsity, averaged over runs. Top-5 accuracy for CIFAR-10 was 99.8% for all levels of sparsity. The fourth column is the number of classes significantly impacted by pruning.
14

---

Under review as a conference paper at ICLR 2020
Sparsity (t)
Model accuracy
Significant
Largest increase
Largest decrease
# incr.
# decr.
class
norm. diff.
class
norm. diff.
diff.
0.1
-0.0002
0
automobile
-0.0016
0.3
-0.0006
1
0
frog truck
0,0022
0,5
-0.0014
1
0
0.0022
0.7
-0.0023
0
0.9
-0.0039
2
0
truck
0.0030
Table Summary of class-level results for CIFAR-10. Only classes passing the significance test are included. The model accuracy difference column reports mean -M -6 the percentage point difference between the pruned and baseline model accuracies; a negative value means the pruned model'ss average accuracy is lower than the baseline model's. The normalized difference is calculated using Equation|
Figure 3 in the main body of the paper visualizes the relative increases and decreases in performance across classes at 70% and 90% pruning on ImageNet. Those figures were shrunk for space; Figure shows the full chart for 70% sparsity for clarity.
6.3 ADDITIONAL PIE RESULTS
In the body of the paper we show the performance of the unpruned model on PIE images found at varying levels of sparsity and showed that performance is worst for PIE1o images which suspect are the most difficuul and performance is still poor but better for PIE, for larger values of t. In Figure T0 we plot the performance of the pruned models on PIEs identified at different levels of sparsity and show that the behavior of the pruned models track the behavior of the unpruned model in this regard.
6.4 ADDITIONAL CORRUPTION AND ADVERSARIAL RESULTS
Sparse models are less robust natural adversarial examples. At high levels of sparsity, models also more brittle to common image corruptions. We include the raw ImageNet-C results in Figure
6.5
HUMAN STUDY
balanced sampled PIE and non-PIE were selected at random and shuffled. The classification PIE non-PIE not known available in the sample. Questions codified for every image considered:
Does label I accurately label an object in the image? (0/1)
Does this image depict single object? (0/1)
Would you consider labels 1,2 and 3 to semantically very close to each other? (does this image require fine grained classification) (0/1)
Do you consider the object in the image to typical exemplar for the class indicated by label I? (0/I)
the image quality corrupted (some common image corruptions- - overlaid text, brightness, contrast, filter, defocus blur, fog, jpeg compression, pixelate, shot noise, zoom blur, black and white rbg)? (0/1)
Is the object in the image an abstract representation of the class indicated by label 1? [[n abstract representation is an object in an abstract form, such as a painting, drawing or rendering using different material.]] (0/1)
15

---

Under review as a conference paper at ICLR 2020
plaver
radio
miniature
probescis
rude
soap
trench
barh
amer con chame
Iato
siberaan
0.06
-0.04
-002
0.00
0.02
0.04
0.06
point difference
Figure 8: Expanded version of the 70% sparsity chart from Figure Normalized recall difference (bars) and absolute recall difference (points) per class. Every third class label is shown.
16

---

Under review as a conference paper at ICLR 2020
Subset of Image Corruptions and Abstract Representations Surfaced by PIE
black and white
defocus
contrast
abstract ruprasentation
Test: Set
true label monito non-sporse prediction: montor spanse prodiction: file
swing plastic butcher shop
Toorball_holnet
barbershop
cleak crash helmet snowmobile
butcher shop grecery store crayfish
gulletine maypole
barbershop
hand blower
Training Set
tricyce
non-sparse prediction: diffle
barrel Mift
triyy
labrador retriever
pertume
maze
sparse predictions book jacket
shopping cart
nomatode
pinvheel
Crossword puzzlo
Figure 9: Images surfaced by PIE evidence common corruptions such as motion blur, defocus or post-processing with overlaid text. Many PIE images depict objects in an abstract form, such as painting, drawing rendering using different material. PIEs displayed were identified by comparing the modal label a set of 90% sparse and non-sparse ResNet-50 models.
ImageNet Top-1 PIE (Sparse Modcls)
10 iodmageNet Top-1 on Non-PIE (Sparse Models)
100
Top-1 Test Set securacy
Top-1 Test Set accuracy
80
60
spantry 33
redel 250k
40
nede: 3pamyy
40
sede: spari.y.2 25
20
20
9-8 model L.w.rrby......
model opprrny.. opwrdyyyyy
PIE Identifeed Spersity Fraction
PIC
Identiied
Fraction
CIFAR-10 Top-1 on PIE (Sparse Models)
FIAR- 10 Top-1 on non-PIE (Sparse Models)
100
100
Top-1 Test Set accuracy
80
3p
Tap-1 Test Set socuracy
80
rede:.pansyy.3
60
redeiopeney_29
60
bav- perfuraance
40
musi qwenl:
mus q5
modd wweittt
modi epentt
0.1
0.2
0.9
0.2
0.8
0.0
1.0
Identified Sparsity Fraction
dentiiied Fraction
Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both
ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.
17

---

pixelate Under review as a conference paper at ICLR 2020
ImageNet Robustness to ImageNet-C Corruptions (By Level of Sparsity)
Corruption Type
Pruning Fraction
Top-1
Top-5
Top-1 Relative
Top-5 Relative
brightness
0.0
0.69
0,89
90.90
95.53
brightness
0.3
0.69
0.89
90.51
95.35
brightness
0.7
0.67
0.88
90.01
95.08
brightness
0.9
0.64
0.86
88.39
94.02
contrast
0.0
0.42
0.62
55.32
66.35
contrast
0.3
0.42
0.62
55.15
66.27
contrast
0.7
0.41
0.62
55.13
66.64
contrast
0.9
0.38
0.58
52.44
64.16
defocus blur
0.0
0.50
0.72
65.10
77.79
defocus blur
0.3
0,49
0.72
64.65
77.52
defocus blur
0.7
0.47
0.71
63.33
76.50
defocus blur
0.9
0.45
0.68
61.60
74.95
elastic
0,0
0.57
0.77
74.68
82.36
elastic
0.3
0.57
0.77
74.33
82.18
elastic
0.7
0.55
0.75
73.46
81.48
elastic
0.9
0.53
0.74
72.80
80.84
fog
0,0
0.56
0.79
73.52
85.08
fog
0.3
0.56
0.79
73.31
85.04
fog
0.7
0.54
0.78
72.62
84.68
fog
0.9
0.50
0.75
69.42
82.46
gaussian noise
0.0
0.45
0.66
59.42
70.50
gaussian noise
0.3
0.45
0.65
58.44
69.63
gaussian noise
0.7
0.42
0.62
56.03
67.52
gaussian noise
0.9
0.33
0.51
45.32
56.53
impulse noise
0.0
0.42
0.63
55.24
67.81
impulse noise
0.3
0.41
0.62
53.97
66.74
impulse noise
0.7
0.38
0.59
50.55
63.65
impulse noise
0.9
0.25
0.43
34.86
47.36
jpeg compression
0.0
0.66
0.86
86.00
92.61
ipeg compression
0.3
0.65
0.86
85.35
92.24
jpeg compression
0.7
0.63
0.85
84.64
91.78
jpeg compression
0.9
0.61
0.83
83.50
90.89
0.0
0.57
0.78
75.00
83.80
pixelate
0.3
0.57
0.78
74.47
83.46
pixelate
0.7
0.55
0.76
73.25
82.43
pixelate
0.9
0.51
0.73
70.73
80.13
shot noise
0.0
0.44
0.64
57.32
68.78
shot noise
0.3
0.43
0.63
56.12
67.73
shot noise
0.7
0.40
0.60
53.19
64.97
shot noise
0.9
0.31
0.49
42.46
53.65
Table 5: Sparse models are more sensitive to image corruptions that are meaningless to a human. We measure the average Top-1 and Top-5 test set accuracy of models trained varying levels of sparsity on the ImageNet-C test-set (the models were trained on uncorrupted ImageNet). For each corruption we consider and the relative measures see appendix (Table. 5) we compute the average accuracy of 50 trained models across all 5 levels of corruption severity.
18


Reviews:
[{"review_id": "rkeVZWq0FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an empirical study on the effect of pruning to the model performance on each class and example, which leads to a novel finding that it has disparate effects to each sample. Specifically, the authors have found out that examples that are affected the most by pruning are more difficult to classify even for the non-pruned network, due to low image quality, mislabeling, or being atypical from the class prototype, and performed a further human study to analyze the source of difficulty. Moreover, the authors performed an additional experiment, which shows that the sparse models are brittle against natural adversarial examples.\n\nPros\n- The paper provides a novel insight on the effect of pruning at the class and the example level, which could lead to a more effective pruning approach that exploit this findings. \n\nCons\n\n- The paper only provides a novel finding but not the solution on how to tackle this problem, and thus the paper looks incomplete. After section 3.3, I was expecting to see some approaches to tackle this problem but the paper abruptly ended. \n\n- The effect of pruning could largely differ from one method to another, but the authors do not experimentally compare the effects of different pruning methods. Also, it is highly likely that the findings discussed in the paper may be only true for input-independent pruning approaches, and may not generalize to input-dependent pruning method. The authors need to perform extensive study of both input-dependent and input-independent pruning approaches to validate their points.\n\nIn sum, the paper provides a novel insight on how pruning affects the performance at the example level, but does not provide a solution, and the current set of experiments is insufficient to validate that the empirical findings that the authors report generalize to other types of pruning approaches, such as input-dependent pruning. Thus I believe that the paper is not ready for publication yet, and vote for rejecting this paper in its current form.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxHnvh4YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims that neural network pruning methods have different impact on accuracy in class-wise and sample-wise. To this end, the paper performs statistical tests to identify such classes (and samples) from a population of neural network models of different levels of pruning. In particular, the identified samples, called PIEs, are shown to have significantly lower test accuracy, i.e., the PIEs are much harder to classify, and a user study is performed to support this claim. The paper also demonstrates that pruned models tend to have lower accuracy against adversarial attacks or common corruptions.  \n\nIn overall, the paper addresses an important problem of investigating the effects of pruning. The experiments performed here seems fairly extensive. One of my primary concerns, however, is that I am still not convinced whether the empirical findings presented here is indeed significant, as some reader might feel those results are not that surprising. I personally feel that it is much more likely that pruning gives heterogeneous effects over the classes, since current pruning schemes do not enforce the \"uniform\" brain damage, i.e., they simply aim to minimize total accuracy drops. Also, the accuracy differences in Table 1 do not appear to be that large in my opinion. The questions in what follows may help to resolve such concerns: \n\n- Section 2.1: For ImageNet dataset, data imbalance in training set might be the reason of such disparate impact? As CIFAR-10 dataset is balanced, I wonder if CIFAR-10 results could be also presented in more details.\n- What if the whole training and pruning is re-performed after excluding the classes which the accuracy is decreased more by pruning? Would it lead to better pruning results, or better overall accuracy?\n- I feel there should be more explanation about the actual statistical implication of the use of Welch's t-test: Is it ok that S^c_t may not be independent to S^c_0? What is the key benefits of not just picking outlier classes among S^c_t with respect to c?\n- What would happens if we exclude the PIEs in training set, and performs training & pruning from scratch? The overall claims may be strenghten if this could improve the pruning efficiency.\n- Figure 7: I think the pure ImageNet top-1 accuracy should be also presented in each plot as a baseline for fair comparison, as this accuracy will be decreased with respect to the sparsity as well.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkxOsIC5FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper study how the pruning impacts the images/classes and observes which images/classes do not generalize in the pruned networks under ResNet-50 with ImageNet dataset. The authors first observe whether there exist classes exhibiting higher/lower accuracy in pruned networks compared to unpruned ones. Then, the authors define images (PIE) that generate inconsistent predictions for unpruned/pruned networks and observe the properties of PIE (e.g., whether corrupted, contains multiple labels, etc.). In particular, PIE images could be considered as `hard examples’ since even the unpruned networks misclassify them w.h.p. The authors also verify the properties of PIE and conclude the paper.\n\nOverall, I think the paper contains some interesting observations but not enough for publication yet, due to the following reasons. First, it is well known that hard (noisy or corrupted) images are harder to be correctly classified in networks of smaller model power (Hendrycks & Dietterich, 2019). That is, results in Figure 4 and Figure 5 are not very surprising. Moreover, there is no further discussion on the hypothesis test after presenting Table 1 and Figure 3. \n\nThe paper is generally well written even it contains few typos and inconsistent presentations (e.g., legends in images are inconsistent. Both underbar and space are mixed.)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates neural network pruning through the lens of its influence over specific exemplars (which are found to often be lower quality or mislabelled images) and how removing them greatly helps metrics.
The insight from the paper is interesting, as recognized by reviewers. However, experiments do not suggest that the findings shown in the paper would generalize to more pruning methods. Nor do the authors give directions for tackling the "hard exemplar" problem. Authors' response did provide justifications and clarifications, however the core of the concern remains.
Therefore, we recommend rejection.
Decision: Reject