Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm
Authors: Aidan Good
Jacky Lin
Xin Yu
Hannah Sieg
Mikey Fergurson
Shandian Zhe
Jerzy Wieczorek
Thiago Serra
Venue: NeurIPS 2022 Accept (2022)

Abstract: Pruning techniques have been successfully used in neural networks to trade accuracy for sparsity. However, 
the impact of network pruning is not uniform: prior work has shown that the recall for underrepresented classes in a dataset may be more negatively affected. In this work, we study such relative distortions in recall by hypothesizing an intensification effect that is inherent to the model. Namely, that pruning makes recall relatively worse for a class with recall below accuracy and, conversely, that it makes recall relatively better for a class with recall above accuracy. In addition, we propose a new pruning algorithm aimed at attenuating such effect. Through statistical analysis,  we have observed that intensification is less severe with our algorithm but nevertheless more pronounced with relatively more difficult tasks, less complex models, and higher pruning ratios. More surprisingly, we conversely observe a de-intensification effect with lower pruning ratios. 

Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm
Aidan Good" Bucknell University
Jiaqi Lin" Bucknell University
Xin Yu" University of Utah
Hannah Sieg Bucknell University
Mikey Ferguson
Shandian Zhe University of Utah
Jerzy Wieczorek Colby College
Thiago Serra Bucknell University
Bucknell University
Abstract
Pruning techniques have been successfully used in neural networks to trade accu- racy for sparsity. However, the impact of network pruning is not uniform: prior work has shown that the recall for underrepresented classes dataset may be negatively affected. In this work, study such relative distortions in recall by hypothesizing an intensification effect that is inherent to the model. Namely, that pruning makes recall relatively worse for a class with recall below accuracy and, conversely, that makes recall relatively better for a class with recall above ac- curacy. In addition, propose effect. Through statistical analysis, we have observed that intensification is less severe with algorithm but nevertheless more pronounced with relatively more difficult tasks, less complex models, and higher pruning ratios. More surprisingly, we conversely observe de-intensification effect with lower pruning ratios, which indicates that moderate pruning may have a corrective effect to such distortions.
pruning algorithm aimed at attenuating such
Introduction
Back in the old days, network pruning important reduce the size of neural networks 23 53 33 41
24
With new advances being increasingly reliant on overwhelmingly large and costly network pruning is back to the game despite the gains in computing power.
models
3 64
10
Network pruning reduces the complexity large models by setting substantial number of parameters to zero and then using gradient descent to fine-tune the sparser model. There is typically trade-off between making the model sparser- i.e., having fewer parameters, since those set to zero can simply keeping the model as accurate as possible ( although moderate gains in sparsity side
be discarded-aad
may be obtained with little impact to accuracy (28). However, recent studies have found
effects of network pruning on model performance are not evenly distributed 30 55
63 69].
Those studies have considered the impact of network pruning on recall-i.c. the number of correct predictions per baass-baed on dataset representability. Namely, they have found that the degradation in performance is influenced by the uneven representation of classes (30 [53 69 and features [3]
as well as class complexity [55] Hence, they corroborate long-standing concerns that unbalanced datasets yield models that are less accurate and potentially harmful to minoritized groups
In this work, we complement those prior studies by posing recall distortion as inherent to network pruning even when the networks are trained and fine-tuned datasets that seemingly balanced; hence investigating when and how such distortions manifest, as well as how to reduce their effect.
"Equal contribution
Corresponding author: thiago..erraaaaee..o.. wwwrasbuckelll.ccommm edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

More specifically, this paper presents the following contributions:
We conduct statistical study of model-level distortion investigate how recall is affected by network pruning due to pruning ratios, dataset and model complexity, and pruning algorithms.
(ii)
We observe an intensification effect, meaning that for classes with recall below accuracy we observe the relative difference between recall and accuracy negatively widening if the network is sufficiently pruned. For classes with recall above accuracy, we conversely observe that gap positively widening. The intensification correlates with excessive pruning ratios as well as more complex data and models, and is more pronounced with some pruning algorithms.
(iii) More surprisingly. we observe that otherwise network pruning has corrective effect recall differences, hence implying a de-intensification effect under moderate pruning.
(iv) We introduce new gradient-based pruning method for networks trained with weight decay, Undecayed Pruning, which attenuates the intensification effect observed with other methods.
2 Related work
The use of large models has been justified and further encouraged by findings that the overparameter- ized regime may avoid the classic dias-variance trade-off [83]]] and lead to better convergence during training [44 (66). However, these models come with steep environmental footprint and hardware
needs 64 Not surprisingly, they represent relevant application of network pruning 20 78).
Network pruning has been motivated by parameter redundancy in models [13] and found to improve
generalization 4 and robustness against adversarial manipulation 774 35 The amount of pruning
that is tolerable by an architecture may depend on the task in which is trained [46]. As observed
on recent survey [9], most approaches to network pruning are guided by either (i) removing the
parameters with smallest absolute value 23 33
22
21
43
17
13
20
68
48 ; or (ii) removing
the parameters with smallest impact on the model
24
25
39
52
14
79
82
5 42 71
72
76
61 80].
We may regard exact pruning as a special case of (ii) in which the model is not affected [39
62 60
[8
Alternative approaches to network pruning include quantization B2 50 2] knowledge
distillation 37 45
70, 67
77.. and the use of regularization during training 81
16 56
In this work, we contribute to the study of recall distortion due to model compression 631, which a particular focus on network pruning. Concurrent to our work, Tran et al. 669 framed
30 55
these recall distortions in unbalanced datasets as a Matthew effect that can be attributed to model (Hessian loss and gradient flows) as well as data characteristics (input norms and distance to decision boundary). Pruning distortion has also been studied under other metrics, such by focusing the samples for which the classification is affected [30 31 36 78 and the relationship between false positive and negative rates [8). The issue is also studied in the context of other compression techniques such as knowledge distillation [27]. in which some approaches to remedy this issue are focused on adjusting the loss function used in the compressed model 33 78
Such disproportional impact on recall across classes relates to fairness in machine learning. The lack of representativeness and the biased context in which data is collected as well as the lack of transparency may lead to models making life-altering decisions that negatively affect minoritized groups, such as in criminal justice [14 (57). These concerns have motivated an extensive discussion on fairness in machine learning [2] and on the proper assessment of datasets [19,, models (51), and the cireumstances in which they are applied [58). In this work, we focus studying the algorithmic bias [49] 22 of network pruning, that is, the bias that is either (i) not present in the input data nor the original model; or (ii) deteriorated relieved by pruning. By subscribing to this line of study, our work aims at reducing the potential negative societal impacts associated with network pruning.
3 Pruning algorithms
We propose Undecayed Pruning (UP) by considering the interplay between the classic representatives of pruning methods described in Section 2 Magnitude Pruning (MP) and Gradient Pruning (GP).

---

Magnitude Pruning MP is a simple but rather (mysteriously) effective technique of selecting among the parameters 0 8 the one with smallest absolute value be pruned next from the network:
i = arg min {10:1}
parameter having a smaller value may not necessarily imply lesser importance to the model 24 (hence the mystery part). However, it is worth noticing that this technique is usually applied to networks that were trained with regularization. Since we cannot add LO regularization to the loss function for directly minimizing the number of nonzero parameters as this would make the loss function not differentiable for gradient descent, we can use or L2 proxy, which induces the values of the parameters to be as small as possible. Although no parameter ends up being zero, some parameters become so small that making them equal to zero implies an almost negligible change.
GP approximates the impact of modifying the parameters of a neural network by estimating the first-order variation of the loss function L on the training set between the trained parameters 8 = 9 to a new set of parameters 0 through the Taylor series around 0 = A:
Gradient Pruning
[(0) (()) + (0- 11-00)00 ()) + O 0
By assuming that 0(10--133) > 0 for small changes, we estimate the change the loss function by pruning a single parameter i-i.e., with 0 such that 0, and 0, 02 for j + i-as as follows:
( C (0) (0 C
(()
Given the approximate nature of estimating the impact on the training set, which may not necessarily reflect on the test set, small absolute perturbations are often preferred negative but larger variations. Hence, the choice of the parameter to be pruned next is effectively framed as follows:
i arg min {I 0, (.C())
Undecayed Pruning We aim to address what we believe is a double-edged approach to reducing the number of parameters. Namely. that it is common to term in the loss function to make the weights as small as possible, such as weight decay, and then that we use the gradient of the loss function for pruning weights while ignoring that the gradient is also affected by the weight decay term. We understand that simultaneously using regularization and GP is conflicting because sparsity is induced in two different ways. However, whereas GP has using some amount regularization tends to be beneficial during training. Hence, we approach this by isolating the effect of regularization from the variation of the loss function. Due to its greater popularity, we consider L2 regularization-ice, weight decay-in what follows. Let us denote by T the loss function of a model after deducting weight decay with hyperparameter e:
direct proxy to model impact,
E 1101
T(0)= = L())
2
With V,T V,C(0) =A,, we estimate the change to the alternative loss function by pruming a single parameter -i..., with 9 such that 0, = and 0, = 0, for j # as follows:
T(t) - T()) ~ (0 ()) = ( = 0. T - -
+ =0f
In other words, discounting weight decay is equivalent to a specific balance of the criteria for MP and GP: the first term is equivalent GP: the second term curiously equivalent to MP; and the latter is prioritized in proportion to the weight decay hyperparameter network training converges to local optimum, in which case VL(0)) = O, then UP is equivalent to MP. Similar to the case of GP, we choose the parameter to prune based on the absolute impact on T:
Moreover, if the neural
arg min {1
0, C( + c0
The way that weights are now ranked reflects a weighted combination of the criteria used for MP and GP. This particularly interesting because UP coincides with MP when the gradient is sufficiently close to zero, hence providing principled argument for the effectiveness of MP under weight decay.
3

---

Model properties
Let A(m) be the accuracy of an unpruned model m as measured on the test data. In other words, A(m) is the number of correct predictions divided by the number of samples. For pruning ratio t, meaning that the number of parameters before pruning divided by the number of parameters after pruning, let Ar (m) denote the accuracy of model m on the test data after pruning. For simplicity, we may assume that
is absent from the notation and omit m always referring to the same model, and thus we may assume that A = A(m) = A1 (m.) The same applies to other metrics.
Similarly, let (m) be the recall for class an unpruned model m on the test data. In other words, R is the number of correct predictions for class divided by the number of samples for class c. For a pruning ratio t, let Ri (m) denote the recall for class of model on the test data after pruning.
We are particularly interested in the how these metrics differ for each class. Namely, let
(m) = (m) A.(m)
denote the recall balance When (m) O, we say that model at pruning ratio overperforms for class c; and when B5(m) < 0 we may say that m underperforms for class c. Finally, let
Bf(m) (5(m) A.(m) (r(m)
(m)
A.(m)
denote the normalized recall balance. The further away this value is from O, the more pronounced is the difference in performance between class and the other classes in model at the pruning ratio t.
Proposition 4.1. If a dataset with set of classes C is balanced, meaning that each class E C has the same number of samples, then for any model m and pruning ratio it holds that
Σ (m) O.
Proof. If the dataset is balanced, then it follows that A, = Enec R2 [C|
and hence Ecec Rf |C|A
Therefore, (R))(R)) - [C] A, = Ee Ad) E.Ec B
Corollary 4.1.1. Ifa a dataset is balanced, then for any model m and pruning ratio t holds that
Σ cEC
(m(m) O.
Proof. Immediate from dividing Eeec (m) by A, (m) due to Proposition
Example Consider model in which A = 80% with two classes X and Y such that R 90% and Rd = 70%, and after pruning we have A, = 60%, R² 70%, and R. 50%. Note that the recall balance remains the same for each class before and after pruning: B. B 10% and BY BY
10%. However, the predictions of the pruned network for class are as good as guess. In turn, the normalized recall balance is the same in absolute value when the two classes are compared, but that value increases due to pruning: B- BB = and B
B t.-
Our choice of normalizing through dividing by (,(m) rather than by 12 (m) aims at avoiding outliers from small recall values as well as undefined results if high pruning ratio leads to zero recall. Moreover, an extension of Proposition 4. such as Corollary 4.1.1 would not be possible otherwise.
We may obtain similar properties by replacing recall R (m) with precision P'm).. In other words, pe is the number of correct predictions for class e divided by the total number of predictions for class By extension may also consider the effect of pruning the F-score, in which case would R. P
consider replacing Ht with the harmonic mean of recall and precision, i.e., R + pe
However, we
have preliminarily found the effect of pruning more expressive on recall than on precision or F-score.
FFor web search on April 22, 2022, there were no hits for either "recall minus accuracy" or "subtract accuracy from recall" and very few references for "difference between recall and accuracy".

---

5 Measuring intensification
For given pruning ratio and class of a model m, we consider the following intensiiicattion ratio:
(m) Normalized recall balance after pruning (m) == (3(m) Normalized recall balance before pruning
This metric can be used to evaluate if pruning widens the performance gap between classes, which happens when E 1. For a class с in which the original model overperforms (B" 0), a ratio greater than 1 after pruning implies that [B > | since 1 > 0 A 15 > - Bi > 1 > 0. Similarly, for class e in which the original model underperforms (B < 0), a ratio greater than after pruning also implies that [ > |Ba] since B < O If > 1 1 Bi < B < 0. In other words, we can use the same metric to determine if pruning comparatively improves and worsens normalized recall balance for classes in which the model respectively overperforms and underperforms.
Example (cont.) For both classes X and Y. we obtain the same intensification ratio 1," = I. = Hence, pruning the model leads to greater disparity in relative performance for those classes.
B3 A and may assume A, A for sufficiently large t, one could argue that A A, alone leads If > 1. However, our experiments show quite the opposite in the case of random pruning, which can be regarded as a less careful selection of weights to prune. Moreover, we consider
Since If B A,
1
B A indicating the relative drop in recall balance (if indeed it drops at all) is less severe than the relative drop in accuracy.
ratio above from another perspective. Namely, 15 > implies
We summarize intensification across classes for model m pruning ratio by estimating the slope ar (m) of a simple linear regression between normalized recall balance before and after pruning:
(mm) (4(m) B (m)
The slope (m) fit using all classes C. We omit the intercept term from this regression as would always be zero in the balanced datasets that we study due to Corollary(4.1.] Consequently, the ordinary least squares estimate of the slope becomes a weighted mean of the intensification ratios If:
Σ B
BB9)T
SEC
REC
a2 =
Σ 3EC
(BO)
(BO
6EC
In this weighted mean, less weight is given to classes in which normalized recall balance before pruning is closer to O. We believe this is appropriate model-level summary, since it de-emphasizes classes for which recall and accuracy are nearly the same before pruning. Hence, we conclude that pruning induces overall intensification on m if a (m) and de-intensification if a am(m) 1.
In addition, within each model there is a statistical dependency between the set of per-class ratios {IE I E C} obtained after pruning. However, the model-level summaries o; (m) are independent across random initializations associated with training each model m (conditional on the dataset), which allows to use simpler statistical inference methods as described in Section 6.2 In our statistical analysis, we evaluate [[au] as the hypothetical expectation across infinitely many m.
Comparatively, the statistical analysis in Hooker et al. 30 would be equivalent to evaluating E[ R, as dependent variable and A as it based on 1s 1. A multiple linear regression with A, A
R
R
as one of the independent variables is fit by Paganni (55) over large number of models and classes.
Experimental setting
We seek to understand when E[as]] > 1, or alternatively B[og] < 1, by evaluating the impact of network pruning algorithms on models trained on seemingly balanced datasets and through varying but well-known architectures. Hence, we consider the dependence of the intensification ratio at the model level by denoting it a,P for dataset D E D, architecture M e M. pruning ratio t T. and pruning algorithm P e IP. In what follows, we may omit the indices if they are constant.
D.M
5

---

6.1 Computational details
We use some combinations of models based on ResNet- (20, 32, 44, 56, 110] 26 and an adaptation of LeNet5 [40] (ii) trained in MNIST [40], Fashion-MNISS [75], CIFAR-10, and CIFAR- 100 [38., and then (iii) pruned using MP, GP, and Random Pruning (RP) from ShrinkBench (9] and our implementation of UP over ShrinkBench, all of which tested with (iv) pruning ratios 2, 4. 10, 20, and 50. For each combination of model and dataset evaluated, we have trained 30 models, which are the same used for pruning at each pruning ratio. The datasets were chosen due to their popularity and equal representation across classes. The architectures and pruning ratios were chosen based on preliminary experiments aiming for good accuracy and also to prune to up ratio with evidence of distortion. MNIST and Fashion-MNIST are only trained in LeNet5. When the dataset or model does not vary, we use CIFAR-I0 and ResNet-56 due to their intermediary complexity. When the pruning algorithm does not vary. we use MP due to its popularity and seemingly better performance than GP [9].
The LeNet5 models are trained with SGD optimizer for 30 epochs, with batch size of 128 and learning rate of 0.01, and then fine-tuned for another 15 epochs after pruning. The ResNet models are trained with SGD optimizer for 60 epochs, with batch size of 128, decreasing learning rate schedule, and weight decay of 0.0005. These values were selected based on preliminary testing. For all models, the weights for the epoch with the lowest validation loss are saved during training and fine-tuning for testing. All experiments are run on GPUs of Nividia GeForce 3060ti and 3090.
6.2 Statistical methods
D.M We are interested in comparing the distributions of otP across different scenarios. Each otP is calculated using the test set recall metrics from just one run of training and pair of models, corresponding to before and after pruning at ratio t. Thus we frame our statistical analyses as inferences about [E(x,, P.M where the expectation is taken over neural network models trained with different random seeds. We have tested varying M, D, or P along with t. Hence, we evaluate the impact of the pruning ratio along every other dimension. For model complexity through a.M, use CIFAR-10 trained on (20,, 32, 44, 56, 110} and pruned with MP. For dataset complexity through aP we all the datasets with their default model and pruned with MP. For pruning algorithm through Or we use CIFAR-10 10 trained on ResNet-56 and pruned with all pruning methods. Due to the encouraging results with UP, have repeated the model complexity with UP for comparison.
D.M
Figures Each scatterplot matrix shows the normalized recall balances before vs after pruning, at several & (rows) and several of either M, D, or P (columns). The scatterplots for varying P are in Figure 5) the remaining scatterplots can be found in Appendix B including Figure 8 for varying M with UP instead of MP as the pruning algorithm. Each point corresponds to one class c for one model m. One regression line and numeric summaries slope using all m together, ,2 is the corresponding coefficient of determination, and A is the average accuracy across all m. Boxplots summarize how the corresponding model-level slopes a, vary across m, and how their distribution changes with t and either M, D, or P. These boxplots are in Figures to in addition to Figure 4 for varying M with UP instead of MP.
overlaid in each subplot: à is an average
D.M (m)
Confidence intervals At each boxplot, we calculate a t-based 99% confidence interval (CT) for its D.M, We chose 99% confidence to achieve 95% family-wide confidence (within each family of pruning ratios at given M, D, or P) after Bonferroni correction for simultaneously reporting dependent Cls. The labels for pruning ratios in the boxplots include <, >, or ? to denote whether the corresponding CI is below has plots for each CL. Figures 2] and 3 indicate that within each M, D, or P. low ratios tend to
did-iitensification) above (intensification), or overlaps 1. Appendix C
have Cls entirely below 1, some moderate ratios have CIs that overlap with 1, and high ratios tend to have Cls entirely above 1. The exception is P=RP, where the CIs are below at all ratios.
Hypothesis tests by I, M, D, or P We carry out t-tests for each set of hypotheses Section 7 Tests comparing pairs of M or of D are independent-sampler tests, but tests comparing pairs of or of are paired-samples tests: for each m there a natural pairing between two as using the same uncompressed m. Each p- value reported in Tables I through (Appendix A has been multiplied

---

CIFAR- 10 @ ResNet-56 - MP
CIFAR-IO 1O @ ResNet-56 GP
CIFAR-IO @ ResNet-56 - UP
CIFAR-10 @ ResNet-56 - RP
Slope
1.5
1.5
1.5
10"
20
50
10 20 50"
10
20
50"
2
10 20 50
Figure I: Boxplots of D.M (m) across m, at each t within each P. Superscripts <, >, or ? denote 1,
where 99% Cls were below I, above or overlapped 1.
CIFAR- @ ResNet-20 - MP CIFAR- @ ResNet-32 - MP CIFAR- 10 @ ResNet-44 -
CIFAR-IO @ ResNet-110 + MP
Slope
2.
2.
2
20
10
Figure 2: Boxplots of D.M a,P (m) across m, each within each M for P MP. Superscripts <, >,
or denote where 99% CIs were below 1, above 1, or overlapped
MNIST LeNet MP
Fashion LeNet
CIFAR-10 @ ResNet-56 MP
CIFAR-100 KesNet-56 MP
Slope di
1.5
1.5
5.5
4
10
20
50
20
10
20
50
2
10
20
50"
Figure 3: Boxplots of (.P (m) across m, at each i within each D. Superscripts >, or ? denote D.M
where 99% Cls were below 1, above 1, or overlapped 1.
by the number of rows in its column of the table, as Bonferroni multiple-testing correction for simultancously evaluating all the rows in that column. When a reported p-value is below the usual significance level 0.05, we have evidence that intensification differs between the pairs in that setting.
Accuracy before pruning In order to assess the impact of pruning on accuracy, the mean accuracy before pruning for each dataset and model reported in Table I0 (Appendix D].
7 Analysis
7.1
The influence of the pruning ratio
We carry out one-sided paired-samples t-tests between the following pairs of hypotheses:
M.M.
E[o E(oM.
H Bla E(aP1
H =[aaf] 2 El
H.M.L. B M Ela...
H. B[a < E|aP_11
H.W. < Bia
In all of those, t, < 1, are consecutive pruning ratios T. We based our a priori on the observation that a higher pruning ratio typically leads to models that have lower performance. As such believe more pruned models would have less capacity to identify as many classes with similar recall when compared with less pruned models, and thus be more susceptible to intensification after pruning.
Figures 2 and suggest that at each M, intensification tends to remain similar or become stronger at higher pruning ratios. Tables and confirm strong evidence of this trend between most consecutive
7

---

CIFAR-10 @ ResNet-20 UP
CIFAR-10 @ ResNet-32 + UP
CIFAR-I0 @ ResNe--4 + UP
CIFAR-I0 @ ResNe-110 + UP
3.
Slope d
2
10" 20 50
10" 20 50
20 50"
10
20
50"
Figure 4: Boxplots a (m) across m, at each t within each M for P=UP. Superscripts <, <,>, or ? denote where 99% CIs were below 1, above 1, or overlapped 1.
pairs of ratios, for both MP and UP, most model sizes, except for ratios vs 4 some larger architectures.
Figure 3 and Figure 7 suggest that at each D, intensification tends to remain similar or become stronger higher pruning ratios. Table 3 confirms that we have strong evidence of increasing intensification for MNIST when going from pruning ratio 20 to 50; for Fashion, when going from 10 to 20 and from to 50; and for both CIFAR-I0 and CIFAR-100, for every consecutive pair of ratios.
Figure D and Figure 5 suggest that at each P except RP, intensification tends to remain similar or become stronger higher pruning ratios. Table 4 confirms strong evidence of this trend between most consecutive pairs of ratios at most P. However, for RP, the average slope actually appears to decrease instead during several ratio increases.
7.2
The influence of model complexity
For each ratio E T, we carry out a one-sided indeenddnnt-sapple t-test of
Ho M.t.
BBaa, M4 < El(a M++1
H.M.L. B(a Mi > E|a; M
for M, and Me+1 as consecutive models in ResNet- (20, 32, 44, 56, 110) on CIFAR-10, using MP and UP. We based our priori the observation that larger (deeper) networks able to learn more complex nonlinear functions than smaller (shallower) networks. As such we believe smaller pruned models would have less capacity identify many classes with similar recall when compared to larger pruned models, and thus be more susceptible to intensification after pruning.
Figures and suggest that most ratios, smaller model sizes tend to have more intensification than larger model sizes. Tables 5 and confirm strong evidence this trend between most consecutive pairs of model sizes (except 56 110 with MP) at high t, but rarely at low t.
7.3
The influence of dataset complexity
For each ratio E T. we carry out a one-sided independent-saneee samples t-test of
11 D.dd
E[a P:] > EE(a D
H. Bla < Ela,") Do1
for the following (D,, D2) pairs: (MNIST, CIFAR-10); 10); (Fashion, CIFAR-10); and (CIFAR-IK 10. CIFAR-100, using MP. We based our priori on how "more complex" datasets often require larger (deeper) models to achieve acceptable accuracy. As such we believe by increasing dataset complexity the pruned model would not be able to identify as many classes with similar recall when compared to less complex datasets, and thus be more susceptible to intensification. We judged that MNIST and Fashion have comparable complexity (10 black-and- -hck-and-white classes), but both are less complex than CIFAR-10 10 (10 RGB classes), which is less complex than CIFAR- 100 (100 RGB classes).
Figure 3 and Figure 7 suggest that at most ratios, MNIST and Fashion tend to have shallower slopes (less intensification) than CIFAR-10, 10, which has shallower slopes than CIFAR-100 Table confirms strong evidence of this trend between MNIST and CIFAR-10 10 at the smaller ratios; between Fashion

---

and CIFAR- 10 only at moderate ratios; and between CIFAR-10 and CIFAR-100 only at the higher ratios.
7.4
The influence of the pruning algorithm
For each ratio & E T, we carry out a two-sided paired-samples t-test of
Ho :
Ela 1 Elc: "1
Ha:
Ela:") + =
for pruning algorithms Pt, P) {MP, GP, UP, RP) on CIFAR-10 with ResNet-56.. Without clear priori expectations for which algorithms would experience more intensification than others, we compare all pairs and use two-sided tests. We also compared just MP and UP all models in ResNet-(20,, 32, 44, 56, 110).
Figure 1 and Figure Sjsugeet a trend in which, at most ratios, UP tends to have shallower but still positive slopes (less intensification) than MP or GP, while RP always has negative slopes. Table g confirms we have evidence that the average slopes for MP, GP, and UP differ from each other higher pruning ratios, but not necessarily at the lowest ratios. We also have strong evidence that all three methods differ from RP, each ratio. Moreover, comparing Figure to Figure O suggests that UP tends to have less intensification than MP at each f and M. Table 9 confirms this for the larger ratios and models.
We can also observe by comparing Figure 5 and Figure 8 that is always smaller for UP, whereas A is only slightly greater for MP in 3 out of 20 cases: ResNet-20 with t 4, ResNet-110 with and ResNet-110 with & =4 4.
8 Supplemental discussion and experiments
Following the advice from the anonymous reviewers, we have augmented our discussion and the experiments carried out in the paper. First, we consider the tradeoff' between accuracy and intensifica- tion lower pruning ratios to discuss the operationalization of our results in Appendix EE] Second, we report the variance of recall balance in Appendix[F] Third, we discuss at greater length the use of our metric of interest in Appendix G Finally, we included additional results comparing accuracy and intensification for two recent approaches, I.TH 17 and CHIP 65 in Appendix H
9 Conclusion
In this work, we have found evidence that network pruning may cause recall distortion even in models trained scemingly balanced datasets, thereby complementing prior studies with underrepresentedd classes and features [30, 555 31
I 63 669. In our experiments, we have observed a statistically significant effect of higher pruning ratios, increased task difficulty, and greater model complexity intensifying the effect normalized recall balance - the difference between recall and accuracy divided by accuracy. This happens for both underperforming classes (recall below accuracy) and overperforming classes (recall above accuracy) for any pruning method other than Random Pruning (RP), hence showing that accuracy reduction alone does not lead to intensification. More surprisingly. however, is that we have observed the opposite phenomenon at lower pruning ratios. Namely, we have observed de-intensification effect, by which model accuracy improves the same time that the differences between class recall and model accuracy goes down. To the best of our knowledge, no other work has identified this corrective effect as positive externality of moderate pruning.
We have also introduced Undecayed Pruning (UP) as a variant of Gradient Pruning (GP) that discounts the effect of weight decay from the loss function, leading to an algorithm that is more closely aligned with the better acclaimed Magnitude Pruning (MP). Since UP consists of a fine adjustment between the MP and GP approaches, Nevertheless, it shows that a better alignment of the pruning method with the loss function alleviates recall balance. We may explore in future how deduct other forms of regularization before pruning.
not surprised that this leads only
minor improvement.
One possible limitation of our study that it does not include a vast list of pruning algorithms, which would nevertheless be prohibitive for a comprehensive statistical evaluation. However, it emphasizes

---

CIFAR-10 @ Re -1 MP
CIFAR-10 @ ResNet-56 GP
CIFAR-10 @ ResNer-56 UP
CIFAR-IE 1O @ ResNet-56 RP
0.928
1,36
y² 0.078
0.976
0.979
04557
0.897
0.9
B.635
Rate:
-0.16
-011
-0.5
0.15
0,15
0.15
0.15
0.16
0,16
④ 0.903
0.983
0.931
0.756
0.975
0.15
1
0.976
0.15
0.974
0.15
0.B14
0.9
A 0.593
A 0.895
4
0.897
6631
Rate: 4
-0.15
-0.15
-0.15_
0.99
0.15
0,15
0.15
0,15
00.15
0,15
0.9
1.057
1.128
1.005
0.968
0.967
0.15
2 0.968
0.15
0.971
0.15
0.9
Á 0.881
Á 0.879
Á 0.888
D5686
Rate: 10
0.15
0.15
0.15
0.9
-0.15
0,15
-0.15
0,15
-0 0.11
0,15
-0.9
0.9
1,239
1.368
1.16
1.19
0.953
0.958
= 0.96
029
Rate: 20
-0.15_
0.9
0,15
0,15
0,15
0.9
6 1.791
à 1.912
à 1.553
0.985
0.2
2
0.2
0.2
0.014
0.9L
Á 0.787
À 0.785
A 0.818
D
Rate: 30
0.2
-02
-02
-099
Figure 5: Scatterplot matrix of (m) (x-axis) vs B (m) (y-axis), at several values of t (rows) and P (columns). Each scatterplot point corresponds to one for one m. See Section 6.2
the classic methods conventionally used and sheds light on how they can be combined through our new algorithm. Nevertheless, we believe that the insight how to develop better variations of simple algorithms such as MP and GP helps us understand how to design more complex network pruning algorithms as well. In fact, additional experiments carried out by suggestion of the anonymous reviewers show that the intensification effect can also be observed in modern pruning methods.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for the constructive discussion, including their suggestions to expand the scope of our work as well as to consider future directions of work.
Aidan Good, Jiaqi Lin, Hannah Sieg, and Thiago Serra were supported by the National Science Foundation (NSF) grant 2104583. Hannah Sieg was supported by the H. Royer Undergraduate Research Fund. Mikey Ferguson was supported by Bucknell University's Presidential Fellowship. Xin Yu was partially supported by the National Science Foundation (NSF) grant IIS 1764071. Shandian Zhe was supported by the National Science Foundation (NSF) grant IIS 1910983.
10

---

References
O. Ahia, J. Kreutzer, and S. Hooker. The low-resource double bind: An empirical study of pruning for low-resource machine translation. Findings of EMNLP, 2021.
[2] R. Alvarez, R. Prabhavalkar, and A. Bakhtin. On the efficient representation and execution of deep acoustic models, arXiv,
2016.
[3] D. Amodei, D. Hernandez, G. Sastry, J. Clark, G. Brockman, and I. Sutskever. Al and compute. https /openaaicco//b// ai and compute/ 2018, Accessed: 2022-05-16.
[4] B. Bartoldson, A. Morcos. A. Barbu, and G. Erlebacher. The generalizatioo-stabllyy tradeoff in neural network pruning. In NeurfPS, 2020.
C. Baykal, L. Liebenwein, Gilitschenski, D. Feldman, and D. Rus. Data-dependent coresets for compressing neural networks with applications to generalization bounds. ICLR, 2019.
[6] M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. PNAS, 11663))15849-1555 2019.
E.M. Bender, T. Gebru, A. McMillan Majoor, and S. Shmitchell. On the dangers of stochastic parrots: Can language models too big? FAceT, 2021.
[8] C. Blakeney, N. Huish, Y. Yan, and Z. Zong. Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation. arXiv preprint arXiv:2106.0748 2021.
[9] D. Blalock, J. Ortiz, Frankle, and Guttag. What the of neural network pruning? In MI. 2020.
[10] R. Bommasani, D.A. Hudson, E. Adeli, R. Altman, S. Arora, von Arx, M.S. Bernstein, Bohg,
A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji A. Chen, K. Creel, J.Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh,
L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D.E. Ho. J. Hong. K. Hsu, J. Huang. T. Ieard, S. Jain, D. Jurafsky, P. Kalluri, Keeling, F. Khani,
S. Karamcheti,
Khattab, P.W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, L. Levent, X.L. Li, X. Li. T. Ma, A. Malik, C.D. Manning. S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nic, J.C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J.S. Park, C. Piech, E. Portelance, C. Potts, A. Raghumathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, Sadigh, Shih, K. Srinivasan, A. Tamkin, R. Taori. A. W. Thomas. Tramèr, R.E.
S. Sagawa, K. Santhanam, Wang, W. Wang. B. Wu, J. Wu, Y. Wu, S.M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models. arXiv, 2108 2108.07258, 2021.
[11] J. Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In FAT 2018.
[12] S. Corbeut-Dvies and S. Goel. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv, 1808.00023, 2018.
[13] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. Freitas. Predicting parameters in deep learning. In NeurlPS, 2013.
[14] X. Dong, S. Chen, and S. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In NeurlPS, 2017.
[15] B. Elesedy, V. Kanade, and Y.W. Teh. Lottery tickets in linear models: An analysis of iterative magnitude pruning, 2020.
[16] A. Fan, F. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. arXiv, 1909.1 1556,, 2019.
[17] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICL.R, 2019.
[18] I. Ganev and R. Walters. Model compression via symmetries of the parameter space. 2022.
[19] T. Gebru, J. Morgenstern, B. Vecchione, J.W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford. Datashects for datasets. Communications of the ACM, 64(12):86 92, 2021.
11

---

[20] M. Gordon, K. Duh, and N. Andrews. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Rep4NLP Workshop, 2020.
[21] S. Han, H. Mao, and W. Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and Huffman coding. In ICLR, 2016.
[22] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
[23] S. Hanson and I.. Pratt. Comparing biases for minimal network construction with back-propagation. In NeurIPS, 1988.
[24] B. Hassibi and D. Stork. Second order derivatives for network pruning: Optimal Brain Surgeon. In NeurIPS, 1992.
[25] B. Hassibi, D. Stork, and G. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, 1993.
[26] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual leamning for image recognition. In CVPR, 2016.
[27] G. Hinton, O. Vinyals, and Dean. Distilling the knowledge in neural network. arXiv, 1503.02531, 2015.
[28] T. Hoefler. D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv, 2102.00554, 2021.
[29] S. Hooker. Moving beyond "algorithmic bias is data problem". Patterns, 2(4))100241, 2021.
[30] S. Hooker, A. Courville, G. Clark, Y. Dauphin, and A. Frome. What do compressed deep neural networks forget? arXiv, 1911.05245, 2019.
[31] S. Hooker, N. Moorosi, G. Clark, S. Bengio, and E. Denton, Characterising bias in compressed models. arXivv 2010.03058, 2020.
[32] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-ariihmeticonng inference. In CVPR, 2018.
[33] S. Janowsky. Pruming versus clipping in neural networks. Physical Review A, 1989.
[34] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, Tinybert: Distilling bert for natural language understanding. arXiv, 1909,10351,200 2019.
[35] A. Jordão and H. Pedrini. On effect pruning on adversarial robustness. ICCVW, 2021.
[36] V. Joseph, S.A. Siddiqui, A. Bhaskara, Gopalakrishnan. S. Muratidharan, M. Garland, Ahmed, and
A. Dengel. Going beyond classification accuracy metrics in model compression. arXiv, 2012.01604, 2020.
[37] Kannan, A. Kurakin, and I. Goodfellow. Adversarial logit pairing. arXiv, 1803.06373, 2018.
[38] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[39] V. Lebedev and V. Lempitsky. Fast ConvNets using group-wise brain damage. In CVPR, 2016.
[40] Y. LeCun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEF, 1998.
[41] Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. In NewrlPS, 1989.
[42] N. Lee, T. Ajanthan, and Torr. SNIP: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.
[43] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Graf. Pruning filters for efficient convnets. In ICLR, 2017.
[44] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. In NewrIPS, 2018.
[45] Y. Li, S. Gu, C. Mayer, L.V. Gool, and R. Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In CVPR, 2020.
12

---

[46] L. Liebenwein, C. Baykal, B. Carter, D. Gifford, and D. Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. In MLSys, 2021.
[47] L. Liehenwein, C. Baykal, H. Lang, D. Feldman, and D. Rus. Provable filter pruning for efficient neural networks. In ICI.R 2020.
[48] S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D.C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In NeurlPS, 2021.
[49] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
[50] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. arXiv, 1710.03740, 2017.
[51] Mitchell, S. Wu, A. Zaldivar, P. Baress, L. Vasserman, B. Hutchinson, Spitzer, I.D. Raji. and T. Gebru. Model cards for model reporting. In FAT*, 2019.
[52] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
[53] M. Mozer and P. Smolensky. Using relevance to reduce network size automatically. Connection Science, 1989.
[54] C. O'Neil. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, 2016.
[55] M. Paganini. Prune responsibly. arXiv,
2020.
[56] C. Riera, C. Rey, T. Serra, E. Puertas, and O. Pujol. Training thinner and deeper neural networks: Jumpstart regularization. In CPAIOR, 2022.
[57] C. Rudin, C. Wang, and B. Coker. The age of secrecy and unfairness in recidivism prediction. Harvard Data Science Review, 2(1), 2020.
[58] A.D. Selbst, D. Boyd, S.A. Friedler, S. Venkatasubrammanin, and J. Vertesi. Fairness and abstraction in sociotechnical systems. In FAT*, 2019.
[59] T. Serra, A. Kumar, and S. Ramalingam. Lossless compression of deep neural networks. In CPAIOR, 2020.
[60] T. Serra, X. Yu, A. Kumar, and S. Ramalingam. Scaling up exact neural network compression by ReLU stability. In NeurIPS, 2021.
[61] S.P. Singh and D. Alistarh. WoodFisher: Efficient second-order approximation for neural network compression. In NeurlPS, 2020.
[62] G. Sourek and F. Zelezny. Lossless compression of structured convolutional models via lifting, In ICLR, 2021.
[63] S. Stoychev and H. Gunes. The effect of model compression fairness in facial expression recognition. In AMAR, 2022.
[64] E. Strubell, A. Ganesh, and McCallum. Energy and policy considerations for deep learning in NLP. In ACI, 2019.
[65] Y. Sui, M. Yin, Y. Xie, H. Phan, S. Zonouz, and Yuan. CHIP: Channel independence-bbase pruning for compact neural networks. In NeurlPS, 2021.
[66] R. Sun, D. Li, S. Liang, T. Ding, and R. Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37(5)955-088, 2020.
[67] S. Sun, Y. Cheng. Z. Gan, and J. Liu. Patient knowledge distillation for bert model compression. arXiv. 1908 00035,955,1111 2019.
[68] H. Tanaka, D. Kunin, D. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In NewrIPS, 2020.
[69] C. Tran, F. Fioretto, J.-E, Kim, and R. Naidu. Pruning has a disparate impact on model accuracy. In NewrIPS, 2022.
13

---

[70] I. Ture, M.-W. Chang. K. Lee, and K. Toutanova. Well-read students learn better: The impact of student initialization knowledge distillation. arXiv, 1908.08962, 2019.
[71] C. Wang, R. Grosse, S. Fidler, and G. Zhang. BigenDamage: Structured pruning in the tronecker-factored cigenbasis. In JCML, 2019.
[72] C. Wang, Zhang, and Grosse. Picking winning tickets before training preserving gradient flow. In ICLR, 2020.
[73] Wang. F. Wei. Dong, Bao. Yang. and Zhou, MiniLM: Deep self atteeatttntonn distillation for task-agnostic compression pre-trained transformers. NeurlPS, 2020.
[74] D. and Wang. Adversarial neuron pruning purifies backdoored deep models. NeurlPS, 2021.
[75] Xiao, K. Rasul, R. Vollgraf. Fashion-MNIST. a novel image dataset benchmarking machine learning algorithms. arXiv, 1708.07747, 2017.
[76] X. Xing, L. Sha, Hong. Z. Shang, and J. Liu. Probahilistic connection importance inference and lossless compression of deep neural networks. In ICLR, 2020.
[77] C. Xu, W. Zhou, T. Ge, F. Wei, and M. Zhou. Bert-of-theseus: Compressing bert by progressive module replacing, arXiv, 2002.02925, 2020.
[78] C. Xu, W. Zhou, T. Ge, K. Xu, J. McAuley, and F. Wei. Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression. arXiv, 2109.03228, 2021.
[79] R. Yu, A. Li, C. Chen, J. Lai, V. Morariu, X. Han, M. Gao, C. Lin, and L. Davis. NISP: Pruning networks using neuron importance score propagation. In CVPR, 2018.
[80] X. Yu, T. Serra, S. Ramalingam, and S. Zhe. The combinatorial brain surgeon: Pruning weights that cancel one another neural networks, ICML, 2022.
[81] X. Yu, Z. Yu, and S. Ramalingam. Learning strict identity mappings deep residual networks. In CVPR, 2018.
[82] Zeng and Urtasun. MLPrune: Multi-layer pruning for automated neural network compression. 2018.
[83] C. Zhang, S. Bengio, M. Hardt, Recht, and O. Vinyals. Understanding deep Icarning requires rethinking generalization. ICLR, 2017.
[84] Zhou, Xu, and McAuley. Meta learning for knowledge distillation. arXiv, 2106.04570. 2021.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Instruction
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL.)? [Yes] See the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6.1.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 6.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section6.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
Code
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "rcUp4_MIe2O", "reviewer": "Reviewer_hKwV", "summary": "Paper studies the problem of recall distortion in neural network pruning. Particularly, authors observe that pruning makes recall relatively worse for a class with recall below accuracy  and makes recall relatively better for a class with recall above accuracy. Authors proposed a new pruning algorithm namely undecayed pruning which can attenuate such effect. Observations made in the paper will be helpful to the future work on network pruning.", "questions": "Please see 1-3 in Weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. Paper studies an important problem of recall distortion in neural network pruning. \n2. Selected number of criteria is quite representative and contains magnitude based criteria, gradient based criteria and random pruning. \n3. Proposed properties of the pruning are quite interesting and novel. For example, intensification is less severe with the proposed undecayed pruning algorithm but nevertheless more pronounced with relatively more difficult tasks, less complex models, and higher pruning ratios\n4. Many pruning papers consider only one setting, but authors use the same framework (datasets, models, pruning ratios) and provide insights by rigorous statistical tests.\n\n**Weaknesses:**\n1. Paper lacks some conclusions about how to interpret intensification. In other words, how intensification helps design better pruning algorithms. \n2. The proposed undecayed pruning is affected by the regularization techniques applied during training. For example, weight decay is often used in normal training. Batch Normalization and Dropout have implicit regularization. Could authors comment on how those settings will affect the undecayed pruning?\n3. Other pruning methods: Significant progresses have been made recently in developing advanced network pruning techniques, such as [1, 2]. Do these methods have the same effect of recall distortion? Or are these methods already a stepping stone in the right direction towards reducing intensification? \n[1] Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In ICML, 2020.\n[2] Yang Sui et al. CHIP: CHannel independence-based pruning for compact neural networks. In NeurIPS 2021.\n\n**Post rebuttal**\n\nThank you for your detailed rebuttal. All my concerns are addressed, it includes a comparison with other pruning methods. I decide to increase my rating from 5 to 6. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e8fcohud-4D", "reviewer": "Reviewer_X7NS", "summary": "The authors study how pruning network parameters affects relative recall across classes. In particular, they argue that high levels of pruning increases the asymmetry in recall already present in the dense network. If recall for a class is below accuracy before pruning, it tends to decrease relative to accuracy after pruning, and if it is larger than accuracy before pruning, it tends to improve relative to accuracy after pruning. \n\nThey evaluate this hypothesis by constructing a model-level metric for “intensification” that aggregates information across classes and conducting statistical tests to verify this intensification effect. Using similar statistical tests, they observe that intensification is higher at high pruning ratios, and for high model and data complexity, but lower for low pruning ratios. They also study how the intensification effect varies across 3 existing pruning methods and introduce a new pruning method which they claim decreases this intensification effect.\n\n**EDIT:** The authors have sufficiently addressed my main concerns and the additional experiments are quite promising and make this a stronger paper. The methodology in this paper is quite thorough and I think that the findings will be interesting to the community. I am raising my score from a 6 to 7.", "questions": "**First, minor points on clarity:**\n\nThe contributions section is not well worded and rather unclear. The paper would benefit from more proofreading of that section and a clearer, more explicit statement of the contributions. In particular:\n1. Contribution (i) has grammatical errors that make it a little unclear. I also am not sure what exactly the contribution is here: the method of statistical analysis is not novel.\n2. Contribution (ii) should be more explicit: what exactly is meant by \"less severely affected\" and \"more severely affected\"?\n\n**Questions about $\\alpha$:**\n\nI think that there are some issues about $\\alpha$ that need to be addressed since this is the basis for most of the experiments. I am not sure $\\alpha$ is the best way to aggregate information across classes. As mentioned in the paper, it down-weights classes with small $\\|R^c - A\\|$ _before_ pruning. This seems problematic as it would down-weight the examples where there is small negative recall balance before pruning but large negative recall balance after pruning, which is a problematic case. I think that it is also important to include an ablation study of just using the simplest obvious metric: $I^c_t$ averaged across classes without weightings to make sure that the the results still hold.\n\n**Significance:**\n\nI would encourage the authors to be more explicit about what additional information this paper provides compared to previous work and how that is useful to the community. \n\nRight now, the benefit of the newly introduced pruning method is unclear. The method does not seem very well motivated. Why is doing GP with this additional L2 term a useful thing to do? Why does it reduce intensification of the recall distortion? Also though UP does seem to do better than MP, the gains seem to be marginal. A more explicit and clearly presented head to head comparison and statistical significance of the gains would be helpful here. \n\nAnother route to impact would be to further explore the reduction of recall balance from small amounts of pruning. How significant is this finding and how sensitive is it to the fraction pruned? How does the overall accuracy change? Would the authors recommend this as a good practice to improve all models post training, and if so, why?\n\nAnother direction to explore would be: how can this finding of the connection between recall balance pre and post pruning be operationalized? Are there ways we can mitigate this effect? Can this be used for model selection between models that perform with similar accuracy, i.e. if two models have similar accuracy before pruning but one has less recall distortion, will both models have similar accuracy after pruning? If so, this could be used as a metric for selecting models with lower post pruning recall distortion. Is there a tradeoff between lower recall distortion and overall accuracy after pruning? Or perhaps, this new metric should be another axes along which pruning methods should be evaluated. In that case, a well documented and easy to use code base that would enable people to very easily and quickly adopt this metric into their analysis would be helpful. \n\nI think that improving this work in 1 or more of these directions to really operationalize this would take it beyond an interesting observation and turn it into a strong and impactful work for the community.", "limitations": "The authors have sufficiently addressed the limitations of the work. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThough previous work has demonstrated that the impact of pruning is uneven, as far as I know, the specific hypothesis that pruning intensifies a pre-existing recall distortion is novel. This is an interesting hypothesis and has potential impact as it enables one to predict, pre-pruning i.e. based on the dense model, which classes are likely to be more affected by pruning. The authors perform a thorough and systematic statistical analysis to verify their hypotheses. The authors also study under what conditions (variations in model/dataset complexity and pruning ratio) the hypothesis holds. The methodology is sound and commendable. Additionally, the finding that recall distortion at low pruning ratios decreases intensification is potentially very interesting and impactful. The paper is overall well written and clear, though a few minor typos and grammatical errors are a little distracting—I would recommend a careful proofread.\n\n\n**Weaknesses:**\n\nDespite the thorough experimentation, this work has a few weaknesses that I think needs to be further addressed. I will outline them here but please see the Questions section for a more thorough discussion.\n1. The primary metric $\\alpha$ used in this paper has some weaknesses.\n2. The direct utility of this work is a little unclear. A more explicit discussion about how this work is different from previous work and what the authors think the marginal benefit of this work is would also be helpful.\n   (a) The motivation for and benefit of UP is unclear. \n   (b) Reduction in recall distortion from small amounts of pruning is under-explored.\n   (c) Lacks a demonstration of how this can be operationalized.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UKfz6ixIfN1", "reviewer": "Reviewer_imMk", "summary": "This paper claims that network pruning inherently makes the model's class-wise accuracy (i.e., recall) imbalanced across classes, which the authors call intensification effect. In order to prove the hypothesis, some ratios are introduced, which models relative performance of each class with respect to the overall accuracy, and thereby intensification ratio is finally defined and experimentally measured. In the experiments, intensification effect is examined in terms of model complexity and dataset complexity, and the corresponding results show that the more complex the dataset, the simpler the model, the stronger the intensification effect.", "questions": "1. Please refer to Weakness 3 above.\n2. How is the variance of class-wise recalls? Does it become larger with a higher pruning ratio?\n", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(Strengths)\n1. The paper is well written and motivated, providing novel insights on network pruning.\n2. The hypotheses are adequately designed and analyzed in a statistical manner.\n3. Undecayed pruning is devised via a novel finding about the relationship between magnitude pruning and gradient pruning.\n\n(Weaknesses)\n1. As the authors also mentioned, the tackled pruning schemes are somewhat naive and conventional, compared to the SOTA pruning methods. This weakness makes this work less interesting in practice.\n2. Some new metrics should be more rigorously justified. For example, recall balance and intensification ratio can be defined in many other ways rather than computing the relative figure with respect to the overall accuracy.\n3. In Sections 7.2 and 7.3, the expected results on intensification effect are not well explained. Thus, why the smaller models are prone to intensification effect? and why the more complex datasets can make the effect stronger?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept