Pruning’s Effect on Generalization Through the Lens of Training and Regularization
Authors: Tian Jin
Michael Carbin
Daniel M. Roy
Jonathan Frankle
Gintare Karolina Dziugaite
Venue: NeurIPS 2022 Accept (2022)
Keywords: pruning
generalization
empirical deep learning

Abstract: Practitioners frequently observe that pruning improves model generalization. A long-standing hypothesis based on bias-variance trade-off attributes this generalization improvement to model size reduction. However, recent studies on over-parameterization characterize a new model size regime, in which larger models achieve better generalization. Pruning models in this over-parameterized regime leads to a contradiction -- while theory predicts that reducing model size harms generalization, pruning to a range of sparsities nonetheless improves it. Motivated by this contradiction, we re-examine pruning’s effect on generalization empirically.

We show that size reduction cannot fully account for the generalization-improving effect of standard pruning algorithms. Instead, we find that pruning leads to better training at specific sparsities, improving the training loss over the dense model. We find that pruning also leads to additional regularization at other sparsities, reducing the accuracy degradation due to noisy examples over the dense model. Pruning extends model training time and reduces model size. These two factors improve training and add regularization respectively. We empirically demonstrate that both factors are essential to fully explaining pruning's impact on generalization.


Pruning's Effect on Generalization Through the Lens of Training and Regularization
Tian Jin Michael Carbin' Daniel M. Roy² Jonathan Frankle Gintare Karolina Dziugaite" +
MIT PUniversity of Toronto, Vector Institute MMoaicML
*Google Research, Brain Team
Abstract
Practitioners frequently observe that pruning improves model generalization. A long-standing hypothesis based bias-variance trade-off attributes this general- ization improvement to model size reduction. However, recent studies on over- parameterization characterize achieve better generalization. Pruning models in this over-parameterized regime leads to a contradiction while theory predicts that reducing model size harms generalization, pruning to range of sparsities nonetheless improves Motivated by this contradiction,
model size regime, in which larger models
re-examine pruning's effect generalization empirically. We show that size reduction cannot fully account for the ceneralization-imrrvvin effect of standard pruning algorithms. Instead, we find that pruning leads to better training at specific sparsities, improving the training loss over dense model. We find that pruming also leads additional regularization at other sparsities, reducing the accuracy degradation due to noisy examples over the dense model. Pruning extends model training time and reduces model size. These two factors improve training and add regularization respectively. We empirically demonstrate that both factors essential to fully explaining pruning's 's impact generalization.
1
Introduction
Neural network pruning techniques remove unnecessary weights to reduce the memory and computa- tional requirements of model. Practitioners remove a large fraction (often 80-90%) of weights without harming generalization, measured by test error 33
[3] While recent pruning research focuses on reducing model footprint, improving generalization has been a core design objective for earlier work on pruning recent pruning literature also frequently notes that improves generalization
19
19 [3
53
30 44
[1 67 34
65
54
35 37
57 55
68
60
31
22
19 13]
How does pruning affect generalization? An enduring hypothesis claims that pruning may benefit generalization by reducing model size, defined as the number of weights in a model We refer to this as the size-reduction hypothesis. However, algorithms for pruning neural networks and our understanding of the impact of model size on generalization have changed.
First, pruning algorithms have grown increasingly complex. Learning rate rewinding (53], a state- of-the-art algorithm prunes weights iteratively. An iteration consists of weight removal, where the
CCorrespondence to: somjinncsail...m mit.muu,,.,u mcarbin@csail. mit. edu, gädzūgoogle. com
'This hypothesis traces back to seminal work in pruning, such as Optimal Brain Surgeon [22] "without such weight elimination, overfitting problems and thus poor generalization will result." And again in pioneering work (19) applying pruning deep neural network models: "we believe this accuracy improvement due to pruning finding the right capacity of the network and hence reducing overfitting." Because the latter hypothesis leaves the definition of network capacity unspecified, we examine an instantiation of it, measuring network capacity with the number of weights in neural network model.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Re Re-trinnnn Epochs
2000
4000
VGG16 93.13% Sparsity
VGG16 99.08% Sparsit
Pruning
Augmented
0.0000
Dense
0.015
-010005
Ang A
Texl
CTF
-0.0110
0.010
-010015
0.005
-010020
4
10a
10
10
% Weights Remaining
Influence Generalization (EL.2N Score Percentile, higher more influence)
(a)
(b)
Figure 1: (a) Retraining the VGG-16 model on CIFAR- 10 exactly as done in pruning, but without removing any weights produces models with similar generalization as pruning the shaded range of weights remaining. We show the x-axis for pruning/augmented training below/above the plot respectively. (b) We show the loss difference between the sparse/dense model produced by prun- ing/standard training respectively for training examples grouped by EL2N score percentiles. Negative value indicates that the sparse model attains better loss. Left: We find that pruning to relatively low sparsity may improve generalization by decreasing training loss, with a particular decrease on influential examples. Right: We find that pruning to relatively high sparsity may also improve generalization by increasing training loss, also with a particular increase on influential examples.
algorithm removes subset of remaining weights, followed by retraining, where the algorithm contin- ues to train the model, replaying the original learning rate schedule. Across iterations, this retraining scheme effectively adopts cyclic learning rate schedule (58), which may benefit generalization.
Second, emerging empirical and theoretical findings challenge our understanding of generalization for deep neural network models with ever-increasing number of weights 48 12
46
69). In
particular, the size-reduction hypothesis builds on the classical generalization theory on bias-variance trade-off, which predicts that reducing model size improves the generalization of an overfitted model [23]. However, recent work 5 47) reveals that beyond this classical birs-variance trade-off regime lies werrparameterized regime where models are large enough to achieve near near-zero training error. In this regime, bigger models often achieve better generalization [5] 47 29 (7 [59]. Practitioners routinely apply modern pruning algorithms to models in this werrparammeterized regime. While our renewed understanding of generalization predicts that reducing the size of an over- wer-parameterized model should harm generalization, pruning to range of sparsities nevertheless improves it. The size-reduction hypothesis is therefore inconsistent with our empirical observation about pruning.
A new direction. We first confirm that the size-reduction hypothesis does not fully explain pruning's effect on generalization. Figure Ia compares the generalization of a family of sparse models of different sizes generated by state-of-the-we pruning algorithm [53], and the generalization of family of models generated by the same algorithm, except modified to no longer remove weights. This algorithm, which we refer to as the augmented truining algorithm, differs from standard training two ways: (1) the number of training epochs is larger; (2) the learning rate schedule cyclic.
The results show that for any sparse VGG-16 model generated by the pruning algorithm with more than 1% of its original weights, its generalization indistinguishable from that of the model generated by the augmented training algorithm that does not remove weights but trains the model for the same number of epochs and with the same learning rate schedule as the pruning algorithm sparsities, pruning's 's effect on generalization remains unchanged in the absence of weight removal, illustrating that the reductionn hypothesis cannot fully explain pruning's effect on generalization.
P For range of
Our approach. We instead develop an explanation for pruning's impact on generalization through an analysis of its effect on each training example's loss the difference in the example's training loss between the sparse model produced by pruning and the dense model produced by standard training.
In Appendix C we show this phenomenon for 4 additional benchmarks, and other pruning algorithms.

---

We then interpret pruning's effect on each example in relation to the example's 's influence on general- ization. In particular, an example's influence on generalization is the absolute change generalization due to leaving this example out, which Paul et al. [51] propose to approximate with the 1.2 distance between the predicted probabilities and one-hot labels early in training, called the E1.2N (Error I.2 Norm) score. Paul et al. [51] show that an example with high E1.2N score may cause large weight update, thereby substantially changing the predictions and loss of other examples. We thus refer to high EL2N examples as influential examples. Our analysis shows the following results.
Better training. We prune a VGG-16 model on the CIFAR-10 dataset to the optimal sparsity, the sparsity with the best validation error. At this sparsity, we find that the pruned model achieves better training loss than the dense model. In the left plot in Figure Tb] we present the change in training loss due to pruning for the training examples grouped by their EL2N score percentiles. We observe that influential examples show the greatest training loss improvement. Pruning may therefore lead to better training, improving the training loss of the sparse model over the dense model, particularly on examples with a large influence on generalization.
Additional regularization. We prune the same VGG-16 model on CIFAR-I0 dataset with the pruning algorithm configured to produce the sparsest model that still improves generalization over the dense model. At this sparsity. we find that the pruned model achieves worse training loss than the dense model. In the right plot in Figure [b] we present the change in training loss due to pruning for the training examples grouped by their EL2N score percentiles. The results show that pruning may also improve generalization while increasing the training loss of the sparse model over the dense model, particularly on examples with a large influence on generalization.
While it is perhaps counter-intuitiin that increasing training loss, particularly on the most influential examples may improve generalization, examining these pruning-affectte examples leads us to hypothesize that it is due to an observation that Paul et al. [51] made: in image classification datasets, small fraction of influential training examples have ambiguous or erroneous labels, training on which impairs generalization. Therefore, it is instead better for the model to not fit them.
We examine this hypothesis by introducing random label noise into the training dataset. We find that on such datasets, pruning may improve generalization by increasing the training loss on the noisy examples, while still preserving the training loss of examples from the original training dataset. The result is an overall generalization improvement for the sparse model. Notably, reducing model width has similar effect training loss and generalization, suggesting a reduced number of weights in the model as the underlying cause for generalization improvement in the presence of noisy examples. We refer the effect of removing weights as additional regularization, which reduces the accuracy degradation due to noisy examples over the original dense model.
Implications. It has been a long-standing hypothesis that pruning improves generalization by reducing the number of weights (via the Minimum Description Length principle [18] or Vap- nik Chervonenkis theory (63)). Our empirical observations suggest that the theory and practice of pruning should instead focus on the two effects of modern pruning algorithms: better training and additional regularization, both of which are indispensable to explaining pruning's 's benefits to generalization fully. Our results also suggest that quantifying pruning's heterogenous effect across training examples is key to understanding pruning's influence on generalization.
Contributions.
We present the following contributions:
We find that pruning may lead betler training, improving the training loss, as well as the generalization over the dense model at optimal sparsity.
2. We find that pruning may also lead to additional regularization, reducing the accuracy degradation due to noisy examples, and improving the generalization over the dense model at optimal sparsity.
3. We demonstrate that our deconstruction of pruning's 's effect into training and regularization cannot be further simplified- - alternative explanations such as extended training time and reduced model size only partially explain the effects of pruning generalization.
3

---

2 Preliminaries
We study Iterative Magnitude Pruning (IMP) with learning-rate rewinding 153 consists of four steps: (1) Train neural network to completion. (2) Remove a specified fraction of the smallest-mannituee remaining weights. (3) Reset the learning rate to its value in an earlier epoch t and (4) Repeat steps 1-3 iteratively until the model reaches the desired overall sparsity.
This algorithm
Terminology. Dense models refer to models without any removed weights. Sparse models refer to models with removed weights as result of pruning. The optimally sparse model is the model pruned to the sparsity at which achieves the best validation error. The optimal sparsity is the sparsity of the optimally sparse model. Generalization refers to the classification error on the test set.
Experimental methodology. We use standard architectures: LeNet 32). VGG-16 [56], ResNet- 20, ResNet-32 and ResNet-50 [24], and train on benchmarks (MNIST, CIFAR-10, CIFAR-100, ImageNet) using standard hyperarraeeter settings and standard cross-entropy loss function 03
14 66). Following Frankle and Carbin [3]. Frankle et al. [4]. we set the t in IMP to t = 0 for
MNIST-LeNet benchmark and 10 for the others. Appendix B shows further details.
We report all results by running the same experiment 3 times with distinct random seeds. We conclude that any real-valued results of two experiments differ if and only if the mean difference between three independent runs of respective experiments is at least one standard deviation away from zero, otherwise we say that the results match. We use PyTorch [50] on TPUs with OpenLTH library [I3]
Limitations. Our work is empirical in nature. Though we validate each of our claims with extensive experimental results on benchmarks with different architectures and datasets, we recognize that our claims may still not generalize due to the empirical nature of our study.
3 Generalization Improvement from Better Training
Not all training examples have the same effect on training and generalization 66 51
25 In
this section, we measure pruning's effect on each training example by examining the difference in its training loss between the dense model produced by standard training and the sparse model produced by pruning. Our results show that pruning most significantly affects training examples that are most influential to model generalization.
Method. Paul et al. [51 propose to estimate an example's 's influence on model generalization with the EL2N (Error L.2 Norm) score the L2 distance between the predicted probability and the one- hot label of the example early in training. Details for computing the EL2N score is available in Appendix B.3 They show that an example with a high EL2N score may cause large weight update, thereby substantially changing the predictions and loss of other examples. Paul et al. [51] rely on this relationship between EL.2N score and generalization to remove examples with little influence on generalization from the dataset to accelerate training without affecting model generalization. We similarly use the EL2N score to assess the impact of pruning on generalization.
To measure the difference in training loss on examples due to pruning, we partition the training set into M subgroups, each with a different range of EL2N score percentiles. We then compute the average training loss difference on examples in each subgroup. We compute this difference by subtracting the training loss of the dense model produced by standard training from that of the sparse model produced by pruning. A negative value indicates that the sparse model has better training loss than the dense model. We pick M = 20 because is the largest value of M that enables us to clearly present the per-subgroup training loss difference.
We measure training loss at two sparsities of interest: the sparsity that achieves the best generalization. and the highest sparsity that still attains better generalization than the dense model. In Appendix L we present pruning's effect on subgroup training loss wider range of sparsities.
Results. Figure 2a presents pruning's 's effect on subgroup training loss. Our results show that at the sparsity that achieves the best generalization, for all subgroups, the average training loss of the pruned model either matches or improves over the dense model, indicating better training.
Figure 2a also shows that pruning's 's effect on training example subgroups is nonuniform. Pruning most significantly affects the most influential examples - the ones with the highest range of EL.2N scores. Specifically, the average magnitude of pruning's effeet examples with the top 20% EL2N

---

MNNST-LeNus 48.8%
CIFARIO-R=-Ne00 48.8
CIFAR 10. VOD. 83.22
CIFAR Ro0 59 59.06%
InageeNs-Rooaee ReaeeeeNee- 15.0%
0092
Ang. CTC
Ang. Cittas
-003
10.01
-0.10
.A15
-0.933
20
100
20
100
20
100
100
300
E1EN Pe
1E.N% Pecnatis
11.20 Parverit
10.20 Perverlle
(a) At sparsity achieving the best generalization
MINIST-LeNe 63.228
CIFARIO-RcoNec00 83.226
CIFAR.O-VGGI6
CIFARL00-ResNed222 73.79%
800
80
0.0
Jnn ACLLan
0.0-
6/6
0.02
0006
00000
Desertle
Fostmal
Pencrtle
Pemeealle
(b) At highest sparsity with generalization exceeding that of the dense counterpart
Figure 2: Difference between the average training loss of the dense model produced by standard training and the sparse model produced by pruning on example subgroups with distinct EL2N percentile range. Negative values indicate that the sparse model has better loss than the dense model.
scores 3.5-7Ix that on examples with the bottom 80% EL2N scores. Pruning the said sparsity thus improves generalization while improving the training loss of example subgroups nonuniformly. with an emphasis influential examples. In Appendix [ validate the effect of pruning-improved examples on generalization: excluding 20% of than excluding a random subset of the same size.
examples hurts generalization more
Conclusion. On standard datasets, pruning to the sparsity that achieves the best generalization leads to better training- the training loss of most example subgroups improves the dense model. The subgroup with the most influence on generalization sees the largest training loss improvement.
4 Generalization Improvement from Additional Regularization
Pruning to the highest sparsity that improves generalization (Figure 2b displays similar effect as pruning to the sparsity that attains the best generalization (Figure 2a generalization improves with an overall improved training loss over the dense model, except for CIFAR-10, where pruning improves generalization while worsening training loss on most subgroups of examples. Specifically, for ResNet20 CIFAR-10 83, 3,22%% sparsity (Figure 2b) second from left) and VGG-16 CIFAR- 10 at 99.08% sparsity (Figure 2b third from left), the training loss of the sparse model worse than the dense model. While pruming always increases training loss when it removes large enough fraction of weights, the accompanied generalization improvement does not always occur in general.
The second and third plots of Figure 2b show that the loss increase is especially pronounced on particular subgroups of influential examples. Examining these pruning-affectte training examples in Appendix K. leads us to hypothesize that the generalization improvement is due to an observation Paul et 51 made: although influential examples are predominately beneficial for generalization, there exist a small fraction of noisy examples, such as ones with ambiguous or erroneous labels, that exert a significant influence on generalization, albeit in harmful way
To precisely characterize pruning's 's effect on noisy examples, we inject random label noise into the training dataset. Doing so enables comparison of pruning's 's effect on noisy data and original data.
Method. We inject p% random label noise by selecting p% examples uniformly at random and changing the label of each example to one of the other labels in the dataset sampled uniformly at random. We refer to the data that is not affected as original data. We sample and fix the random labels before each independent run of experiment.
We present leave-subgroup-ou retraining experiments for these examples in Appendix K
5

---

MNNIS-LLNNe
CIFARIO-VGGIS
CIPARIO-Ro
ImngeNettBeaNear ReaNa
115
FP5
30%
0.5
31%
65
3F5
D5
195
124
60%
MMS
10
100
32'
Weights Bocckiinn
Weight Rencaking
Wwighes Recatiing
WWigges Reasisiin
Figure 3: Pruning improves generalization over the dense model in the presence of random label noise. We show the generalization of the sparse models function of the % weights remaining. Horizontal dashed lines indicate the baseline generalization of the dense models. We mark the optimal sparsity with star. Legends show the fraction of the training examples with random label noise.
We report the following results for models trained on datasets with and without random label noise: (1) pruning's impact on generalization versus sparsity (FigureBy B) (2) pruning's 's impact on training loss for the original data and random label data versus sparsity (Fig (Figure 4 4 (3) pruning's 's impact on
training loss for subgroups of examples, each with a different range of HL.2N scores (Figure(5).
Generalization results. Fiowrs that, in the presence of random label noise, the generalization of the optimally sparse models better than that of the dense models on 11 out of 13 benchmarks and matches that of the dense models on the rest of the benchmarks. Our results provide evidence that pruning to the optimal sparsity improves generalization in the presence of random label noise.
The extent of pruning's ' generalization improvement at the optimal sparsity grows as the portion of label noise increases. For example, for VGG-16 on CIFAR- 10 benchmark, pruning to the optimal sparsity improves generalization by 5.2%, 16.1 1..1% 1% and 34.6% on 15%, 30% and 60% label noise.
MNNST-LLNNa
CIFAR No01
CIFARIIO-OOG
CIFAR N0
385
180-
32.0
135
7.5
3.-
60
10'
Figure 4: Pruning to the optimal sparsity increases training loss over the dense model in the presence of random label noise. Noisy examples see particular large loss increase. We show the training loss on the original and noisy data with dashed and solid lines respectively, as functions of % weights remaining. We mark the optimal sparsity with star. Legends show the random label noise level.
Original versus noisy data. F shows that pruning initially reduces the average training loss on noisy examples: for instance, the training loss on noisy examples is slightly lower for the sparse ResNet20 models on CIFAR-10 10 with many weights remaining (i.e., more than 10% ) than for the dense model. Since noisy examples are influential examples [51], these results indicate better training. which improves model training loss with particular emphasis on influential examples.
As the fraction of weights remaining drops further, the loss of the sparse model on noisy data grows. For instance, the training loss on noisy examples increases for the sparse ResNer20 models on CIFAR-10 with few weights remaining (i.e., less than 10%). Moreover, the gap between the average training loss of the sparse model on original versus noisy examples widens beyond the dense model, indicating that pruning leads additional regularization. In particular, for range of sparsities where generalization improves over the dense model, the sparse model fits original examples significantly better than those with randomized labels. Taking the dense VGG-16 model trained on the CIFAR-10 dataset with 15% 60% random label noise as an example, we observe that this model incurs 0.03 0.13 higher training loss on the partition with injected random label noise than on the partition without. Pruning to the optimal sparsity widens this training loss gap to 1.38- 2.97. Furthermore, the gap between the average training loss of the sparse models on original versus noisy examples eventually diminishes as the sparsity increases even further.
'Numerical results available in Table 11 in Appendix M

---

MINIST-LeNet
CIPARIO ResNet20
CIPARIO VGG16
CIPAR TOO Rt5
EmugeNett Ree
1.5
01
10%
19%
Axg. CE Colos
1.0
15%
30%
AVV CE
199
Avg. CE
400
CD
15%
0.5
-0.1
0.0
50
100
50
100
100
50
100
50
100
ELAN Perccotle
BL2N Percentlle
BL2N Percentlee
EL.N Percento
EL,2N Parcnrile
Figure 5: the presence of random label noise, pruning to the optimal sparsity often increases training loss (except for ResNet50) over the dense model, especially to highly influential examples. For example subgroups with distinct EL.2N percentile ranges, we show the difference in average training loss between dense and optimally sparse models. Legend indicates label noise level. We do not show LeNet with 5% noise because the dense model achieves the best generalization. For ResNet20 and ResNet32 with 60% noise, the training loss for high EL2N examples are already very high for dense models to begin with, leaving little room for their loss to increase further by pruning.
Difference in loss by subgroups. Figure 5 shows the difference between the training loss of the dense and optimally sparse model on subgroups of examples with different EL2N scores, but this time the presence of random label noise. The results show that pruning to the optimal sparsity often increases the loss of influential examples, contrary to what Figure shows when training on original data only where pruning decreases the training loss of influential examples.
As random label examples are influential and harm generalization [51, increasing the training loss on influential examples may therefore improve generalization. In Appendix E we validate that excluding examples misclassified by the optimally sparse models improves generalization similar to pruning.
Conclusion. We study pruning's s effects training and regularization by connecting them noisy examples. By introducing random label noise, we show the two effects of pruning more clearly. At relatively low sparsities, lower than the dense model, especially on influential examples highly influential to generalization. As sparsity increases, regularization effects dominate, because the sparse models see larger gap between training loss on the noisy versus original data than the dense model. We note that across all sparsities, pruning has a highly nonuniform effect across example subgroups with different influences on generalization (EL2N scores) on both standard datasets and datasets with random label noise.
observe better training because the training loss of sparse models is
Crucially, the effect of pruming at the optimal sparsity be it better training or additional regularization depends on the noise levels in the dataset. On datasets with few noisy examples, Figure shows that at the optimal sparsity, better training is the key to good generalization, because training loss reduces. On datasets with injected random label noise, Figure shows that the optimal sparsity is the one at which pruning-induced regularization effects increase noisy data loss.
Isolating the Effects of Extending Training Time and Reducing Model Size
We identified two effects of pruning on generalization: better training and additional regularization. They intersect with two components of the pruning algorithm: extended training time and model size reduction, respectively. In this section, we study these two effects in isolation by extending dense model training time and reducing the dense model size by scaling down its width We confirm that both effects of pruning are necessary to fully explain pruning's 's impact on generalization. Thus our deconstruction of pruning's effect in terms of training and regularization cannot be further simplified.
[]
5.1 Extended Training Time
What happens to generalization if the weight removal part is removed from pruning, and only the retraining part remains? To answer, we train dense models exactly as done in the pruning algorithm, for the same number of gradient steps and learning rate schedule, but without removing weights.
'To isolate other effects of pruning, we also evaluate the contribution of design choices of pruning algorithms, such as the heuristic selecting which weights to prune, to generalization via ablation studies in Appendix !
7

---

Method. We study the effect of training dense models for extra gradient steps on generalization, without removing any weights. When training the dense models, we use the same learning rate schedule as pruning, replaying the original learning rate schedule designed for the dense model multiple times, effectively adopting eyclic learning rate schedule. We refer to this training algorithm as extended dense training. We report and compare the generalization of models that pruning and extended dense training algorithm produce.
MNNST-LLNat
CIFARIO-ReNNe20
CIFARIO VGG16
CIFARIO0 RoiNe132
ImageNett ResNed0
Toa Zone
60-
925
32.
25.0
150
2000
1000
2000
230
Thiekiy Squchs
Trinkin Rpeck
Toxtiing Spechs
Figure 6: Extended training time does not explain the full extent of pruning's benefits to generalization. We show the test errors of the dense and sparse models with circle and square markers respectively as a function of the training epochs. Legends show random label noise level.
Results. Figure 6 shows test errors of models that extended dense training and pruning produce as function of training epochs The two algorithms produce models with similar generalization on standard datasets without random label noise, where pruning improves training at optimal sparsity, as shown in Section 3 Indeed, in Appendix L we show that longer training time similarly leads better training. However, on benchmarks with random label noise, extended dense training underperforms pruning on out of the 13 benchmarks. On LeNet, ResNet20, VGG-16, ResNet32 and ResNet50 benchmarks with noise, extended dense training achieves test errors that are worse than pruning by 0 (matching) 2.6%, 0.4 to 14.5%, 3.6 to 34.2%, 0 (matching) 4.8% and -0.3%, respectively.
Conclusion. On standard datasets without random label noise, the extended dense training algorithm produces models with generalization that matches or exceeds the optimally sparse model. However, with random label noise, the extended dense training algorithm can no longer produce models that match the generalization of the optimally sparse model.
5.2 Size Reduction
What happens to generalization if the retraining part is removed from pruning, and only the weight removal part remains? We compare pruning with width down-scaling applying standard training to dense models with reduced widths. The latter removes weights without the need for retraining.
At optimal sparsity, we show that size reduction is not necessary to replicate pruning's ' generalization improvements with dense models in Figure Ta] from Section [ in this section, we show that size reduction is also not sufficient to replicate pruning's generalization improvement with dense models.
Method. We reduce the model size by down-scaling the width of a dense model: we train a sequence of dense models using the standard number of training epochs, where the next model in the sequence has 80% of the width of the model preceding it. This width scaling ratio mimics our sparsity schedule, where the pruned model always has 80% of the remaining weights in the previous iteration. We then compare the generalization of models that pruning and width wwwnl-saaing produce.
Results. Figure 7 presents the test errors of models that pruning and width down-scaling produce, using square and circle markers respectively. Width down-scaling under-performn pruning on out of 5 benchmarks datasets without random label noise. When training with random label noise, width down-scaling has notable regularization effects and significantly improves generalization, similar to pruning. In Appendix[] [ we show that the regularization effects of both algorithms manifest similarly across example subgroups, suggesting that pruning's 's regularization effect is consequence of model size reduction in general. However, despite this improvement, gap remains between the minimum test error of models that pruning and width down-scaling produce achieve: pruning does better on LeNet, ResNet20, 16, ResNet32 and ResNet50 benchmarks by -4 (pruning does worse) to 0.4%, 0% (matching), 1.5 to 1.8%, 0 (matching) to 2.1 %, and 0.36% in test error, respectively.
'Numerical results available in Table in Appendix M
8

---

MINIST-LeNAA 15R
C.FARIO-ReNNNDO SOT
CIPARIO-VOOIG 60%
CIFARIOO RRSNNE 60%
ReugsNNocReaNet50 158.
30
315
300
05
35%
100
305
0%
195
Tea Bove
15G
1PN
60%
30
33%
Teatimee
10
10"
10"
10
LV
322
101
10
10*
Weigis Rezeiiong
Wiights Romsiniin
4 Weights Rotaiiin
Wiaga Recokning
1 #Weghht Rocating
Figure 7: Size reduction does not explain the full extent of pruning's benefits to generalization. We show the test emrors of the dense/sparse models with round/square markers respectively as function of the weights remaining. Legends show label noise level. Appendix F includees detailed views.
Conclusion. On standard datasets with random label noise, width down-scaling produces models with similar generalization (no worse by more than 2.1%) as the optimally sparse models. However, without random label noise, width down-scaling under-performs pruning.
5.3 Conclusion
By removing the weight removal part from pruning in Section 5.1 we show the importance of pruning- induced additional regularization for achieving good generalization in the presence of label noise more clearly than Section 4 Similarly, by removing the retraining part from pruning in Section we show the importance of gouning-induced better training for achieving good generalization using the original dataset without random label noise more clearly than Section 3
We confirm that both effects of pruning required to fully account for its impact generalization.
Our deconstruction of pruning's ss effect in terms of training and regularization is therefore minimal.
Related Work
Pruning has an extended history: early work starting from the 1980s found that pruning enhances
model interpretability and generalization [5] 31,
22). More recently, the advent of deep neural
network models motivated the adoption of pruning reduce the storage and computational demand
37 [7 [55] 9 68 60). In this section, we describe several branches of pruning research pertinent to our work.
of deep models [19, 20 30 40 7 44 67
4 34 65, 54 35
Pruning algorithm design. Recent pruning research often focuses on improving pruning algorithm design. An extended line of research studied the heuristics that determine which weights to remove [44 33 the simplest heuristic is to remove weights with the smallest magnitude. For example, Molchanov al. [43] Louizos et al. [42] proposed to learn which weights prune as part of the optimization process. Another line of research concerned the structure of the weights to remove to achieve computational speedup without significantly hurting model generalization. For example, Li et al. [36], Liu et al. 41 proposed to prune weights in groups or to prune neurons, convolutional filters and channels. Our work does not produce new pruning algorithm design. Rather, we study the existing pruning algorithms to understand their effect on model generalization.
Pruning's effect on generalization. There are many reported instances in which pruning benefits
model generalization [9 [3]. This effect has sparked interest in examining pruning as a technique
gain additional generalization benefits beyond common regularization techniques. Weight decay is a typical regularization technique that encourages model weights to have a small Lg norm as part of the training objective. Giles and Omlin [16] demonstrated that pruning can improve the generalization of recurrent neural network models better than weight decay. Thimm and Fiesler [6]]. Augasta and Kathirvalavakumar [I] compared the effect on the generalization of several contemporary pruning techniques. Bartoldson et al. [3] discovered positive correlation between pruming instability, defined as the drop test accuracy immediately following weights removal, and model generalization. The authors found that higher pruning-induced instability leads to increased flatness of minima, which uurn improves generalization. Our work differs from prior work as we study pruning's 's effects on generalization through a novel perspective, examining pruning's impact on training examples.

---

Pruning's effect beyond generalization. Our work contributes to growing line of work 22 38 investigating pruning's effect on examples beyond test error. Hooker et al. [25] discovered that model compression can have a disproportionately large impact on predicting the under-representee long-tail of the data distribution. Liehenwein al. [88] studied the performance of pruning using metrics such as out-of-distributt preserve these alternative performance metrics even when it preserves test accuracy. To the best of our knowledge, our work is the first to examine pruning's impact on examples in the training set.
generalization and resilience to noise, and found that pruning may not
7 Closing Discussion
Importance of training improvement. Our work sheds light on an overlooked and underestimated effect of pruning that boosts generalization training improvement. Seminal work on pruning focuses on its regularization effect and attributes pruning's s beneficial influence on general- ization to regularization that comes with model size reduction. Instead, our work discovers that, on standard image classification benchmarks, the state-of-the-as pruning algorithm attains the optimal generalization consistently by improving model training.
[2 19
How does model size reduction impact generalization? The relationship between model complexity
47.. Informally, model complexity refers to the model's 's ability to fit wide variety of functions 17 For example, model size [5] and the norm of model weights [49 both provide meaningful measures of model complexity for the study of generalization. Classical theories (e.g., on VC dimension [64] and Rademacher complexity 2) show that reducing model complexity decreases the likelihood for the learning algorithm to produce trained models with a large gap between training and test loss, and may therefore improve generalization. We instead focus on learning dynamics of training example subgroups and show that reducing model size improves generalization by mitigating the accuracy degradation due to noisy examples. Therefore, our work contributes a complementary perspective to the understanding of the relationship between model complexity and generalization.
and generalization has long been subject of immense interest [52] 23 5
Societal impact. Our work examines pruning's effect on generalization. While we measure general- ization using test eπrоr, Hooker et al. [25] show that test error may be an imperfect characterization of model prediction quality due lack of consideration of fairness and equity. Concretely, our work does not address the potential disproportionate impact of model pruning on label categories. Our work must therefore be interpreted within the context of the community's 's current and future understanding of pruming's 's potential contribution to systemic bias, especially against minority groups that may be underrepresented existing training datasets.
Implication for fairness. A growing line of work concerns the adverse impact of pruning on model fairness 25 26
they show that pruning disproportionatel worsens the prediction accuracy on a small subgroup of examples. However, our results show that their work does not completely characterize pruning's 's effects pertinent to fairness: (I) We show that only at relatively high sparsities can pruning harm the prediction accuracy of any example subgroup, where pruning's regularization effects dominate.
(2) We show that size reduction general, rather than pruning in particular, underpins the disproportionately worse accuracy of the pruned model on certain subgroups of examples. Our results call for further assessment and discussion of pruning's s risk to model fairness.
Conclusion. We show that the long-standing size-reduction hypothesis attributing pruning's 's beneficial effect on generalization to its reduction of model size does not fully explain pruning's impact on generalization. Inspired by studies [22] (51) that show nonuniform effects of training examples on generalization, we develop an analysis to study pruning's impact on generalization by interpreting its effect on an example's 's training loss in relation to the example's 's influence on generalization.
With this novel analysis, we find that, at the optimal sparsity, pruning leads to either better training additional regularization, which improves training loss over the dense model, and reduces the accuracy degradation due to noisy examples over the dense model, respectively. Both effects contribute to improving model generalization.
Our novel analysis adds to our empirical toolkit for studying the effect of learning algorithm on generalization through its effect on the loss of training examples. Using our novel analysis, we derive findings that advance our empirical understanding of pruning a learning algorithm.
10

---

References
[1] M. Augasta and Kathirvalavakumar. Pruning algorithms of neural networks - comparative study, Central European Journal of Computer Science, 3, 2013.
[2] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. The Journal of Machine Learning Research, 3, 2003.
[3] B. Bartoldson, A. Morcos, A. Barbu, and G. Erlebacher. The ieneeaaizatiionstabiiiie tradeoff in neural network pruning. In Advances Neural Information Processing Systems, 2020.
[4] C. Baykal, L. Liebenwein, I. Gilitschenski, D. Feldman, and Rus. Data-dependent coresets for compressing neural networks with applications to generalization bounds. In International Conference on Learning Representations, 2019.
[5] M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-lcarning practice and the classical (32), 2019.
trade-off. Proceedings of the National Academy of Sciences, 116
[6] C. Blakeney, N. Huish, Y. Yan, and Z. Zong. Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation. arXiv preprint arXiv:2106. 07844,, 2021.
[7] D. Blalock, J. Gonzalez Ortiz, J. Frankle, and J. Guttag. What is the state of neural network pruning? In Conference on Machine Learning and Systems, 2020.
[8] T. Calders, F. Kamiran, and M. Pechenizkiy. Building classifiers with independency constraints. In IEEE International Conference on Data Mining Workshops, 2009.
[9] T. Chen, B. Ji, T. Ding, B. Fang, G. Wang, Z. Zhu, L. Liang, Y. Shi, S. Yi, and X. Tu, Only train once: A one-shot neural network training and pruning framework. In Advances in Neural Information Processing Systems, 2021.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE conference on computer vision and pattern recognition, 2009.
[11] X. Dong. Chen, and S. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, 2017.
[12] G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Conference on Uncertainty in Artificial Intelligence, 2016.
[13] J. Frankle and M. Carbin. The lottery ticket hypothesis: networks. International Conference on Learning Representations, 2019.
Finding sparse, trainable neural
[14] J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[15] J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021.
[16] C. Giles and C, Omlin. Pruning recurrent neural networks for improved generalization perfor- mance. IEEE Transactions Neural Networks, 5(5), 1994.
[17] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.
[18] D. Grünwald. The minimum description length principle. MIT press, 2007.
[19] S. Han, J. Pool, Tran, and W. J. Dally. Learning both weights and connections for efficient neural networks. In Conference on Neural Information Processing Systems, 2015.
[20] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
11

---

[21] M. Hardt, E. Price, E. Price, and N. Srebro. Equality of opportunity in supervised learning, In Conference on Neural Information Processing Systems, 2016.
[22] B. Hassibi, D. Stork, and G. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, 1993.
[23] T. Hastie, J. Friedman, and R. Tisbshirani. 2.9 Model Selection and Bias- Stas-Variance Tradeoff. page 37-38. 2017.
[24] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference Computer Vision and Pattern Recognition, 2016.
[25] S. Hooker, A. C. Courville, N. Dauphin, and A. Frome. What do compressed deep neural 2019.
networks forget? arXiv preprint
[26] Hooker, N. Moorosi, G. Clark, S. Bengio, and E. Denton. Characterising bias in compressed models. arXiv preprint arXiv:2010.03055 2020.
[27] Y. Huang, Y. Cheng, A. Bapna, Firat, D. Chen, M. Chen, H. Lee, Ngiam, Q. V. Le, Wu, and z. Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Conference on Neural Information Processing Systems, 2019.
[28] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Conference on Neural Information Processing Systems, 2012.
[30] V. Lebedev and V. Lempitsky. Fast convnets using group-wise brain damage. In IEEE Confer- ence on Computer Vision and Pattern Recognition, 2016.
[31] Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. In Conference on Neural Information Processing Systems, 1990.
[32] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
[33] Lee, S. Park, S. Mo, S. Ahn, and J. Shin. Layer-adaptive sparsity for the magnitude- pruning. In International Conference on Learning Representations, 2021.
[34] N. Lee, T. Ajanthan, and P. Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
[35] N. Lee, T. Ajanthan, S. Gould, and P. H. S. Torr. A signal propagation perspective for pruning neural networks at initialization. In International Conference on Learning Representations, 2020.
[36] H. Li, A. Kadav, I. Durdanovie, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
[37] I.. Liebenwein, C. Baykal, H. Lang, D. Feldman, and D. Rus. Provable filter pruning for efficient neural networks. In International Conference Learning Representations, 2020.
[38] L. Liebenwein, C. Baykal, B. Carter, D. Gifford, and D. Rus. Lost in pruming: The effects of pruming neural networks beyond test accuracy. In Conference Machine Learning and Systems, 2021.
[39] X. Lin, S. Kim, and J. Joo. Fairgrape: Fairnes--wware gradient pruning method for face attribute classification. European Conference on Computer Vision, 2022.
[40] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. In IEEE International Conference on Computer Vision 2017.
12

---

[41] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
[42] C. Louizos, M. Welling, and D. P. Kingma. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018.
[43] D. Molchanov, A. Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks.
In International Conference Machine Learning, 2017. 2017.
[44] Molchanov, S. Tyree, T. Karras, Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference Learning Representations, 2017.
[45] M. C. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Conference on Neural Information Processing Systems, 1 1988.
[46] V. Nagarajan and J. Z. Kolter. Uniform convergence may be unable to explain generalization in deep learning. In Conference on Neural Information Processing Systems, 2019.
[47] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever. Deep double de- scent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020.
[48] B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 2014.
[49] B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, 2015.
[50] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, Kopf, Yang, Z. De Vito, M. Raison, Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Processing Systems, 2019.
[51] M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on data diet: Finding important examples early in training. In Conference on Neural Information Processing Systems, 2021.
[52] C. Rasmussen and Z. Ghahramani. Occam's razor. In Conference on Neural Information Processing Systems, 2000.
[53] A. Renda, J. Frankle, and M. Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
[54] T. Serra, A. Kumar, and S. Ramalingam. Lossless compression of deep neural networks. In International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research, 2020.
[55] Serra, Yu, A. Kumar, and S. Ramalingam. Scaling up exact neural network compression by relu stability. In Conference on Neural Information Processing Systems, 2021.
[56] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[57] S. P. Singh and D. Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Conference on Neural Information Processing Systems, 2020.
2020.
[58] L. N. Smith. Cyclical learning rates for training neural networks. In IEEE Winter Conference on Applications of Computer Vision, 2017.
[59] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions, In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
13

---

[60] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Conference on Neural Information Processing Systems, 2020.
[61] G. Thimm and E. Fiesler. Evaluating pruning methods. In International Symposium on Artificial Neural Networks, 1995.
[62] M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
[63] V. Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[64] V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events their probabilities. Theory of Probability and its Applications, 16, 1971.
[65] C. Wang, R. Grosse, S. Fidler, and G. Zhang. EigenDamage: Structured pruning in the tronecker-factored eigenbasis. In International Conference on Machine Learning, 2019.
[66] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. International Conference on Learning Representations, 2020.
[67] R. Yu, A. Li, C. Chen, J. Lai, V. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis, NISP: pruning networks using neuron importance score propagation. arXiv preprint arXiv:1711.05908 2017.
[68] X. Yu, T. Serra, S. Ramalingam, and S. Zhe. The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks. In International Conference on Machine Learning, 2022.
[69] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3), 2021.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] Section 2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Appendix[B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Section 2
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "n0OzC4lFlI", "reviewer": "Reviewer_TyWA", "summary": "This paper investigates what makes network pruning improve accuracy despite the removal of parameters from the model. The authors contend that the folkloric claim about smaller models implying better generalization is not sufficient given the recent studies about the overparameterized regime, in which increasing model size may also improve generalization. As an alternative, the authors investigate the effect of pruning directly on the samples used for training by using a metric from the literature to identify how relevant each sample is for training the neural network. Consequently, they find out that pruning leads to a greater reduction in the loss function for samples that are more challenging to classify correctly, whereas pruning may also lead to an increase in the loss function for noisy samples that actually hurt generalization.", "questions": "0) Can you please give more consideration to the other modern lines of work on network pruning? \n\n1) Your work heavily implies that Renda et al. (2020) is the state-of-the-art on network pruning. Can the authors cite other papers acknowledging that status? What about the extensive recent work on pruning at initialization, which spans way more than Frankle et al. (2021) (see Wang et al. (2020) + [7,10,13])?\n\n2) The fact that a single algorithm is used for pruning is a noticeable limitation of the work. Have you tried other algorithms, even if something as simple as one-shot magnitude-based / gradient-based pruning, or - better yet - random pruning just to make a contrast?\n\n3) I am not entirely convinced of the results in Section 5.2: the plot seems to indicate that there is a largely negative distinction for higher p when compared to dense models, and that the results are otherwise very similar. In line 269, how can you do better by -4 to 0.4%? Please explain.\n\n4) The expression \"better optimization\" is very imprecise and in fact confusing (in line with people saying \"more optimal\"). The same about wording such as \"the change in optimization due to pruning\". Can the authors please consider a less ambiguous and more specific expression, such as \"better training\"?\n\n5) Figure 1 (b): Have you measure these changes on the test set as well? \n\n6) Your paper implies that negative changes due to the loss function are due to relevant but challenging examples, whereas positive changes are due to noisy examples. Whereas you have done some ablation of the noisy part with artificially inserted noise, have you looked into whether the samples for which the loss function change the most are indeed not the same as the pruned percentage goes higher and the changes become predominantly positive?\n\n7) I am a bit curious about your thoughts on the rightmost plot of Figure 8 (a): Does that imply that MNIST is such an uniform dataset that the noisy discussion does not apply as much there?\n\n8) In Section 4 and Figures 3-5, how did you choose the percentage p of random label noise? Why is it different across models/datasets? Can you add p=0% as another curve in each plot of Figures 3-5?\n\n9) In the conclusion of Section 4, can you please describe in more detail how your computational results imply that low pruning levels are dominated by the optimization and that high pruning levels are more influenced by regularization?\n\n10) What do you mean by pruning \"stability\" in line 297?\n\n11) What \"novel analytical technique\" are you claiming in line 331?\n\n12) How do you think your results relate to what happens in the case exact compression methods [14,15]?", "limitations": "The authors did a good job describing the limitations, except for using a single pruning algorithm to derive their conclusions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents a well organized and very compelling analysis on the effects of pruning on how the neural network learns from the training set. However, I was disappointed at how this paper, like many others in the LTH bandwagon, simply ignored the rest of the work currently being done in network pruning. Whereas the conclusions are very interesting, it makes me wonder what would happen if the same analysis was done with other pruning algorithms as well, or at least with random pruning for contrast. \n\nI find the first two contributions very interesting, although I have questions below about how to distinguish more clearly between them. On the other hand, the third contribution is a bit of a stretch in my opinion: your work shows two factors that may partly explain what makes network pruning so useful. In that sense, they remain in equal footing with model reduction in terms of explaining something but not everything. I believe the wording is lines 92-94 is unnecessarily drastic. For example, the results in Section 5.2 make it clear to me that we actually cannot ignore the effect of model reduction in itself.\n\n### Ignored work\n\nThe first paragraph cites modern papers that are exclusively based on magnitude-based pruning, but cites older work based on gradient-based pruning. The latter literature has grown into something that is often denoted as impact-based pruning and has remarkably good results as well. For example, the survey by Blalock et al. (2020) [1] outline these two kinds of approaches as predominant in the literature. With exception of  Wang et al. (2020), I see none of this second line of work acknowledged in the paper. I believe the authors should not ignore it, since it is a very active branch of the network pruning literature [2-14].\n\n[1] https://arxiv.org/abs/2003.03033\n\n[2] https://arxiv.org/abs/1506.02515\n\n[3] https://arxiv.org/abs/1611.06440\n\n[4] https://papers.nips.cc/paper/2017/hash/c5dc3e08849bec07e33ca353de62ea04-Abstract.html\n\n[5] https://arxiv.org/abs/1711.05908\n\n[6] https://openreview.net/forum?id=HJfwJ2A5KX\n\n[7] https://openreview.net/forum?id=B1VZqjAcYX\n\n[8] https://arxiv.org/abs/1905.05934\n\n[9] https://arxiv.org/abs/2001.00218\n\n[10] https://arxiv.org/abs/1906.06307\n\n[11] https://arxiv.org/abs/1911.07412\n\n[12] https://arxiv.org/abs/2004.14340\n\n[13] https://arxiv.org/abs/2006.05467\n\n[14] https://arxiv.org/abs/2102.07804\n\n[15] https://arxiv.org/abs/2107.07467\n\n[16] https://arxiv.org/abs/2203.04466\n\n### Other comments about the writing of the paper\n\nAbstract:\n- I could not understand the last sentence of the abstract before reading the paper. I would recommend rewriting it, possibly breaking it down into multiple sentences.\n\nIntroduction:\n- Line 38: Add space between word \"generalization\" and following citations\n- Line 67: Remove first \"but\"\n- Line 82: Remove \"count reduction\" \n\nSection 2: \n- Line 116: Remove second period\n\nSection 3:\n- Line 122: \"are affected\": add \"more\" between those words?\n- Line 129: \"he\" -> \"they\" (that work has multiple authors)\n\nSection 5:\n- The title of this section is confusing, and I honestly was expecting something else when I read it: that you would try to isolate the effects of optimization and regularization from each other. Can you please change it to something more direct, such as \"Isolating training time and model size effects\"?\n\nReferences:\n- Han et al. (2015) is cited twice: remove one of those\n\nAppendix B:\n- Line 514: \"interests\" -> \"interest\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T6KwRxvJ9_", "reviewer": "Reviewer_wS1n", "summary": "This paper provides emperical study on the effect of pruning on the generalization of neural network model. The discussion focus on state of the art iterative pruning pipeline, and discuss the effect of pruning on achieving better optimization and better regualrization for the model training process.", "questions": "1. Does longer training mainly contributes to better optimization, and smaller model mainly for stronger regularization?\n2. What would be the effect in Figure 7 be if the weight is pruned in a elementwise fashion with the same zero positions as the fully trained sparse model from iterative pruning?", "limitations": "The limitation and social impact is adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Understanding the effect of pruning on model generalization is an interesting and important topic, this paper makes insightful contribution along the line.\n2. The paper is overall well written and easy to follow.\n3. The paper make novel contribution by subgrouping the data with the difficulty of learning, and make seperated observation for data of different EL2N percentile. This leads to new insight on pruned model's optimization outcome and can inspire future investigation on other neural network optimization techniques. \n\nWeaknesses:\n1. The paper introduces two effects: better optimization and stronger regularization, and two possible reasons: longer training and reduced model size, but doesn't make clear discussion on what's the relationship between the reasons and effects. It would be better to discuss if (following common belief) longer training mainly leads to better optimization, and smaller model size mainly leads to stronger regularization; or the effect of the two potential reasons cannot be distinguished\n2. The experiment setting on model size reduction may be improved: Since the work is mainly based on the pruning method used in lottery ticket hypothesis I believe the discussion should be focusing on element-wise pruning, yet the model size reduction experiment is set for an channel-wise pruning scenario. A better experiment may be reinitialize the nonzero weight element of a sparse model and retrain the model for the same epochs, this will give a fairer comparison with previous iterative pruning observations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4n8rNe_yTEj", "reviewer": "Reviewer_iZ5z", "summary": "The paper finds that the size-reduction hypothesis can not fully explain the pruning's effects on generalization increase. It proposes the decomposition of such increase to better optimization with lower training loss and stronger regularization by avoiding hard training samples. Furthermore, the paper shows that neither extended training time nor size-reduction alone can not fully explain the generalization improvement. Thorough experiments are conducted to support the claim with the help of the EL2N score.", "questions": "I think it is important to show the training accuracy of width down-scaling methods in Figure 7. If the training accuracy of the two methods is far apart, the validity of the conclusion may be undermined. Since if there is a great training accuracy drop after using width down-scaling, there is no point talking about generalization. \n\nAlso see my concerns in weaknesses.\n\n[Minor issues:]\n1. It would be better to also include the optimal sparsity level for Figure 3.\n2. Maybe it is better to give a further explanation of the augmented training method for Figure 1. If I understand correctly, % weights remaining is always 100% for the augmented training method. However, augmented training is also plotted using changing % weights remaining as the x-axis.", "limitations": "Yes, the authors have addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Most of the claims in the papers are backed by extensive experiments.\n2. The paper is easy to follow.\n3. It is a very interesting phenomenon to me that pruning actually improves optimization at a low sparsity ratio. Since most pruning papers mainly focus on test accuracy alone.\n4. The decomposition of pruning's effect into better optimization and stronger regularization through the EL2N score is original and important.\n\nWeaknesses:\n\n1. For section 5.2, the comparison actually seems a bit unfair to me. First, the down-scaling width method, though having the same number of parameters compared to the weight pruning method, may have much less representation power (model capacity) due to lower feature map size. Second, the granularity of down-scaling width methods is not fine enough. For example, only 5 sparsity ratios are tested for the C100-ResNet32 combination while there are 10 for weight pruning methods. We may see a matching/better performance on datasets with random label noise.\n\n2. For Figure 2, I think it is also helpful to plot average CE loss before and after pruning. It may give a better visualization if the neural networks avoid fitting difficult examples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept