What Makes a Good Prune? Maximal Unstructured Pruning for Maximal Cosine Similarity
Authors: Gabryel Mason-Williams
Fredrik Dahlqvist
Venue: ICLR 2024 poster (2024)
Keywords: Pruning
Deep Learning
Neural Networks
Interpretability
Loss landscapes
Optimization
Kurtosis
Primary area: general machine learning (i.e., none of the above)
TL;DR: This paper explores the mechanisms that underpin pruning, ultimately showing that the higher the kurtosis of a model's parameter distribution, the more it can be pruned while maintaining performance.

Abstract: Pruning is an effective method to reduce the size of deep neural network models, maintain accuracy, and, in some cases, improve the network's overall performance. However, the mechanisms underpinning pruning remain unclear. Why can different methods prune by different percentages yet achieve similar performance? Why can we not prune at the start of training? Why are some models more amenable to being pruned than others? Given a model, what is the maximum amount it can be pruned before significantly affecting the performance? This paper explores and answers these questions from the global unstructured magnitude pruning perspective with one epoch of fine-tuning. We develop the idea that cosine similarity is an effective proxy measure for functional similarity between the parent and the pruned network. We prove that the L1 pruning method is optimal when pruning by cosine similarity. We show that the higher the kurtosis of a model's parameter distribution, the more it can be pruned while maintaining performance. Finally, we present a simple method to determine the optimal amount by which a network can be L1-pruned based on its parameter distribution. The code demonstrating the method is available at https://github.com/gmw99/what_makes_a_good_prune

WHAT MAKES A GOOD PRUNE? MAXIMAL UNSTRUCTURED PRUNING FOR MAXIMAL COSINE SIMILARITY
Gabryel Dason-Williams, Fredrik Dahlqyist
Department of Computer Science Queen Mary University
London, UK
g. t. www.nmasonwil. 11 ialiiams@@mm .ac. uk, f. d.dahlqvist@@@mull ac.
ABSTRACT
Pruning is effective method to reduce the size of deep neural network mod- els, maintain accuracy, and, in some cases, improve the network's overall perfor- mance. However, the mechanisms underpinning pruning remain unclear. Why can different methods prune by different percentages yet achieve similar per- formance? Why can we not prune at the start of training? Why some models more amenable to being pruned than others? Given a model, what is the maximum amount can be pruned before significantly affecting the perfor- mance? This paper explores and answers these questions from the global un- structured magnitude pruning perspective with one epoch of fine-tuning. We develop the idea that cosine similarity is an effective proxy measure for fune- tional similarity between the parent and the pruned network. We prove that the LI pruning method is optimal when pruning by cosine similarity. We show that the higher the kurtosis of a model's parameter distribution, the more it can be pruned while maintaining performance. Finally, present simple method determine the optimal amount by which a network can be LI-pruned based on its parameter distribution. The code demonstrating the method is available https: http::://gittuu.com github.com// gmw 99/ whattthatteemmm
INTRODUCTION
Deep neural networks have grown in size achieve state-of-the-art performance in various fields, e.g. object detection (Redmon et 2016) natural language processing tasks (Brown et al.) 2020). going from few million parameters (He et al. 2016) hundreds of billions of parameters (Brown al. 2020). This increase has required substantial computational and memory resources for training and inference, making deployment difficult in resource-comstraince environments.
Various approaches have been developed compress neural networks while maintaining their per- formance (Blalock et al. 2020). (Gholami et al. 2021). One such approach, dating back to the 80s (LeCun al.] 1989), neural network pruning. the process of removing parameters from the par- ent network while maintaining accuracy. Pruming methods are evaluated against each other by the percentage of parameters pruned in the network (compression ratio), along with the change in the accuracy. Pruning has been shown to be an effective method, applicable to various architectures, but reaching varying degrees of sparsity. the number of parameters removed from the network (Blalock) ct al. 2020). This leads to the following questions:
1. Why can different methods prune by different amounts yet achieve similar performance?
2. Why can we not prune at the start of training?
3. Why are some models more amenable being pruned than others?
4. What is the maximum amount model can be pruned before affecting the performance?
This paper aims to answer these questions and provide a deeper understanding of global unstructured magnitude pruning and neural network learning dynamics. We show that:

---

Cosine similarity allows to compare pruning methods as it is good proxy measure for functional similarity.
The LI magnitude pruning method is optimal for maintaining maximal cosine similarity.
Maintaining high cosine similarity with the parent network ensures improved accuracy or limited drop-off in performance.
Neural networks are brittle, i.e. highly sensitive to changes in high- magnituee parameters.
The higher the kurtosis of model's parameter distribution, the more it can be pruned while maintaining performance.
The optimal amount by which network can be L1-pruned can be computed explicitly from its parameter distribution. We illustrate this procedure by computing the optimal level of Ll-pruning for the LeNet Small, ResNet18 8 and VGG11/ BN networks.
2 BACKGROUND AND RELATED WORK
Neural networks are trained a set of features, {x;}. and corresponding labels, {vi J. by minimiz- ing a loss function, I(0)= = n: Σi= {x,yy; 0), where n is the number of samples, 0 represents the parameters of the network, the function E(T.y; measures how effective the network with parame- ters 0 is at predicting the label {yi} 3 for the features {æi} (Murphy 2022). The number of parameters |0) is typically very large, and the loss function defines a |0]-dimensional subspace of RI0++1 referred as the loss/objective landscape.
2.1 PRUNING
Pruning is defined as taking model with parameters 9 and creating new model by applying binary mask M f0, 1 the parameters of ll resulting in the model A (0 M). where @ is the Hadamard product (Blalock et al., (020). Thus, the pruned network operates in a parameter subspace of its parent. The two main pruning methods are structured and unstructured pruning (Blalock et al., 2020). Structured pruning removes groups of parameters, i.e. neurons and channels, whereas unstructured removes individual parameters. These methods
be applied across the whole neural network, globally, or to specific sections of the network, i.c. particular layers. The parameters are selected for pruning based on score. The score can take many factors into account, the simplest being the magnitude of the parameters. More sophisticated factors include information from the network, such the gradients or contributions the network's activations (Blalock al.) 2020). After pruning, the models are fine-tuned, which commonly refers to continuing train the network using the trained parameters the pruned neural network. There are many pruning regimes, including one-shoi pruning, i.e. pruning network once and then fine-tuning, and iterative pruming, ie. repeating one-shot pruning until a desired pruning percentage. heavily pruned network will contain many aroo-pareettrrs and so be very sparse, thus sparsity measures the amount of pruning.
2.2 LOsS LANDSCAPES
Recent advances. Li et al. (2018)) have shown that many problems solved by neural networks can be solved in a smaller dimension with comparable results to the full-dimension solution and provided upper bound for the minimum dimensions required to solve problem. Draxler et al. (2088) and Garipov et al. (2018) have empirically shown that a path of low loss can connect two local minima. Draxler et al. (2018) proposes that these paths exist there is low-loss manifold that the points exist on. [Garipvv et al. (2018) suggests that these points are not isolated but belong low loss valley in the landscape. Fort et al. (2020) showed that models achieving the same loss value can represent very different functions.
Visualisation. Due to the high dimensionality of the loss landscape, is intrinsically hard to visualise and reason about the dynamics of neural network learning. We will use several methods developed to reduce the dimensionality and create faithful visualisations.
First, Fort al. (2020) showed effective method for visualising the functional space, where the functional space is the space of all functions solving the given problem which an architec- ture/topology can represent. The method works by taking the model softmax outputs for dataset,
2

---

which are flattened into vector to represent the model predictions; the t-SNE algorithm (Van der Maaten Hinton 2008) is then used to reduce the dimensionality of the predictions to two- dimensional plot, which IS used to visualise the models' similarity in function space.
Second, Fort et al. (020) also presents an effective method for visualising the loss landscape from the perspective of the origin (where all parameters set to zero) and the two local minima points of interest. It works by perturbing the parameters radially along the weight space two local minima points of interest ó, 7J. which are then multiplied by the radial coordinates Rx. Ry. The loss of the network L(0 + Roó + Ryr) is calculated at this point which is then plotted against 2, y to create dineesiinnni representation (using colours for the third dimension) of the loss landscape. A radial slice is used as the softmax cross- cross-entropy loss means they travel radially through the landscape during training (Fort et al. 2020).
Finally, the Filter-Wiss Normalization method (Li et al., 2018b) focuses on visualising the loss landscape from the perspective of the local minima. It works by perturbing the parameters along two random approximately orthogonal filter-wise normalised directions in weight space 6, 1.. The loss L(O + x9 yл) is then plotted against the perturbation magnitudes æ, y to create an effective three- dimensional representation of the loss landscape around the local minimum. Random directions are used instead of principle component analysis (Pearson I901) as they allow for visualising a more expansive space instead of just the optimised parts of the landscape.
COSINE SIMILARITY PRUNING
Cosine similarity ( is similarity measure that measures the cosine of the angle (a,b between two non-zero vectors a, b in an inner product space.
A similarity value of zero means the vectors orthogonal, value of one means the vectors have the same direction, minus one means the vectors have opposite directions.
Sc b) == cos(sPa,b) = (a. b)
(1)
||a|| |b]l
Neural network parameters, 0, are composed of several tensors of various ranks, which can be flattened by concatenation form a vector representation of the neural network parameters. Global unstructured pruning performs a transformation network parameters and its vectorization) by zeroing some elements, thus creating new vector 0', We will use the cosine similarity (0, 0') to measure how much the parent network has changed under pruning. Note that the order in which tensors are concatenated when vectorizing the model does not change the cosine similarity.
this vector 0 (we use the same symbol for the
The cosine similarity between the pruned sub-network and its parent will monotonically decrease as pruning increases. As consequence, a binary search algorithm (Appendix B Algorithm I can be used to find the amount of pruning required to reach a target cosine similarity (within a specified E).
In order to maintain the maximal cosine similarity when pruning single parameter, one must prune the parameter with the smallest absolute magnitude.
Theorem (Maximal Cosine Similarity via Pruning).
(a, a
(n)
max
is reached for n. arg min ||as]]
aa
(1)
Where (n)) is the vector whose components those of apart from the th which is zero.
The pruning method based on this theorem is known the LI pruning method since it minimises the distance. The proof of the theorem can be found in the Appendix A
4 EXPERIMENTAL SETUP
To explore the questions in the introduction, we trained three distinct convolutional neural net- works with varying sizes and architecture designs (Appendix C Table T). We use the LetNet Small
3

---

Whitaker & Whitley 2022) in order to have model that is small and achieves low performance. The ResNet18 (He et al. 2016) and VGGI1.BM (Simonyan & Zisserman 2014) networks are used as they are complex, achieve high performance, and have been shown have different loss land- scapes, with the VGG11 BN having a more complex loss landscape Li et al. (2018b).
The VGG1 BN, ResNet18 and LeNet Small trained the CIFAR1O dataset (Krizhevsky) 2009). which is a dataset consisting of 10 classes with 50,000 training images and 10,000 test- ing images, using the Adam (Kingma Ba 2014) optimizer with a learning rate of 0.001 and batch size of 256. The VGG11 BN and ResNet18 models were modified to handle the CIFAR-10 dataset as specified by Phan (202I). The VGGI1_BA and ResNet18 models were trained for 100 epochs, and the LeNet Small models were trained for 25 epochs, where an epoch is a complete pass through the training dataset.
The models were then independently globally one-shot pruned, including all the parameters, weights and biases, using Random and L1 unstructured pruning from 0 to 99% pruned with a step size of 1% and then fine-tuned for one epoch. 100% is not used as it would result in a model with zero parameters. 0% pruning is used to create a baseline of what the model would have achieved it had been trained for the additional epoch instead of being pruned.
The ImageNet dataset (Russakovsky et al. 2015) with trained VGGII, ResNet18 and MobileNetV2 (Sandler et al. (2018) models is used to demonstrate that the proposed method works at scale and highlight an edge case, see [5.4] The models were independently globally one-shot pruned and fine-tuned as described above.
(Paszke et al. 2019) pre-
5 RESULTS AND DISCUSSION
5.1 USING COSINE SIMILARITY TO COMPARE PRUNING METHODS AND OPTIMAL PRUNING
T. Touthhopab L2ze
Wnd pag.aing
R VGGII BBN and Lmall change in test negative log loss and accuracy on CIFAR1O after pruning only (Figures 100
Figure
Pruned
(a) Pruned
(b) Pruned and Fine-tuned
Ia and pruning plus one epoch of fine-tuning (Figures 1b ld plotted against per- centage pruned using Random and L11 unstructured pruming. The mean (line) and standard deviation (hue) from 10 runs are displayed.
Prarod
(c) Pruncd
(d) Pruned and Fine-tuned
F Iigustaatltsstttsss illustrates the point made by the first question in the introduction that there is no relationship between, on the one hand, pruning method and pruning percentage, and on the other, changes in loss and accuracy. This can be seen very clearly in Figures [i with marked difference in accuracy between the Random and LI methods at 80% pruned. We see in Figure Ib that Random and Ll pruning follow similar loss trajectories: first dip in the loss function, followed by monotone increase as more of the network gets pruned. There is however significant lag between the two pruning regimes, with the increase in loss occurring much earlier (at around 30% pruned) with Random pruning than with I.1 pruning (at about 90% pruned). This lag is also visible in if one ignores the strange behaviour of VGG11_BE under random pruning (which we cannot explain). Thus, which parameters are pruned is more important than how much of the network is pruned.
To investigate the importance of this choice we examine the effect of pruning in terms of cosine similarity with the parent network, i.e. pruning to target cosine similarity. Figure 2 shows much clearer relationship between the cosine similarity and the change in loss and accuracy of the network across all pruning methods. Comparing Figures 2c and 2d to Figures Ic and ld see that the choice of pruning method has a much smaller impact on the change in accuracy when targets a given cosine similarity. Similarly, we that the lag between the loss trajectories of Random and L.1
4

---

pruning in Figures la
lb almost vanishes in Figures 2a 2b again illustrating that cosine similarity is much more strongly associated with loss than either pruming method or sparsity/pruning percentage.
mandlacs
88 VGGII_BN and Lmall change in test negative log loss and accuracy on CIFARIO after pruning only (Figures 22 2c and pruning plus one epoch of fine-tuning (Figures 2b 24
Figure 2:
Coine Smitarty
Cosina Sintentty
(a) Pruned
(b) Pruned and Fine-tuned
plotted against cosine similarity with the parent network using Random and LI unstructured pruning. The mean (line) and standard deviation (hue) from 10 runs are displayed.
190 2013
60
Coolie Simiaartty
Cosine Strila by
(c) Pruned
(d) Pruned and Finc-tuned
These observations suggest that cosine similarity is vital for understanding pruning. In particular, they suggest that an optimal pruning mechanism should maintain the highest possible cosine sim- ilarity with the parent model. Theorem | re- in-terppreted as stating that in order to prune a vector whilst maintaining maximal cosine similarity with the original vector the parent parameters one must use the LI pruning method.
guarantees precisely this condition and can therefore be
5.2 How PRUNING TRANSFORMS THE FUNCTION SPACE
We hypothesise that cosine similarity can often be used as an efficient proxy measure for the func- tional similarity between the parent and pruned network. This cannot be universal rule. Imagine a situation where a layer L is connected to a network by weights of very small magnitude; this net- work will have high cosine similarity to the functionally very different network where these weights are pruned away, disconnecting C from the network. We will encounter a likely example of this phenomenon in 5.4 However, these are corner cases for which a diagnostic method will be pro- vided. In general, we will see that cosine similarity is very good proxy for functional similarity. If this holds, logically follows that a network with high cosine singularity will produce network with similar performance to the parent, as the functional representations are close. In particular, the optimal pruning method for cosine similarity (I.1 pruning) should have the closest representation to the parent's 's function, as it removes the least impactful weights.
To explore this idea, we examine the effects of pruning on the VGGI1_B BN function space through the visualisation method of Fort et al. (2020). The VGG1 1 1_BN network has been shown to have complex loss landscape (Li et al. 2018b), therefore, any effect that pruning has on the function space should be more apparent. In Figure and end of training, along with predictions of the pruned models, and convert them to a function space. Figure 3 provides functional perspective to the phenomenon illustrated through Figures [ Figures 3a and (3b] show that the L. LI-pruned models stay functionally close to the parent (*) even when highly pruned, whereas the randomly pruned networks functionally diverges from the parent (very quickly in the case of Figure 3a and, interestingly, return to a state that is functionally close the untrained initial network
take the VGG11 BGGL1_BNY _BNN model's predictions at initialisation
When pruning by cosine similarity the functional dif- ference between LI and Random pruning fades away. Figures 33 and 3d show that networks with similar cosine similarity with the parent are functionally close, irrespective of the pruning method. Observe from Figure 3c that for high cosine similarity values, the models remain functionally very similar to the parent, whereas models with low cosine similarity with the parent explore a region of the function space which is close to the untrained parent. After fine-tuning, Figure 3d shows that models with high cosine similarity with the parent return to a state that is functionally close to the parent but, curiously, networks with low cosine similarity with the parent again tend to return to state functionally similar to the untrained parent model.
We hypothesise that the reason why models at high cosine similarity return after fine-tuning for one epoch is that they stay connected to a low loss 'valley' or 'tunnel' (Draxler et al. 2018). (Garipov et al. 2018) that leads to the local optima and, thus, can return to similar function space.
5

---

Fruned
Rall dom Prumed
Figure 3: t-SNE projections of the training dataset predic- tions for the VGG11 _BN model, including initialisation
8 L1 Prumed
and end of training (*)) Figure 3a tresp. 3b) shows the func- tion space of the models when pruned from 0 to 99% using Random (Red) and LI (Blue) pruning (resp. pruning followed by one epoch fine-tuning). Figure [c (resp. [3d] shows the function space of the models when pruned from 1 to 0.1 co- sine similarity using Random (Red) and LI (Blue) pruning (resp. pruning followed by one epoch of fine-tuning).
11
(a) Pruned
(h) Pruned and Fine-tuned
Bandem
Similartty Pruning
Randon End
-32
Randam
-10
3.0
3.0
50
10
(c) Pruned
Pruned and Fine-tuned
We suspect that this is because the high cosine similarity forces the networks operate in subspace of the parent function.
Test Loss
Test Accuracy
Text Loss
Test Accuracy
Origin
Dright
Origin
Purent Optimum
Parent Ootimum
Purent Optimum
Parent
sunco Outimunt
Optimun
suried Outimum
tuned Optimur
Woco Srectto
Weggtt
Weight
Weight direction
(a) LI pruning to cosine similarity of 0.95
LI pruning cosine similarity of 0.30
Figure 4: VGGI l_BN Parent to someed-nnd-ine-tume Radial Landscape slices with respect the test data. The left-hand plots of Figures[4al 4b) show the loss landscape along the paths of the parent and the pruned-and fine-tuned model, the right-hand hand plots show the same landscape for accuracy.
To explore this, we follow Fort al. (2020) and take two-dimensional radial slice of the loss left-hand plots in Figure 4 and test right-hand plots of Figure X landscapes along the weight directions of the parent and the conned-and-fine-tunee
network. In Figure 4a| the network stays
within the local optima malley'/'tunnell for high cosine similarity but operates in functionally different area of the loss landscape; this supports the idea of low loss manifold exists put forward by Draxler et al. (2018). For low cosine similarity, Figure 4b the fine-tuned network becomes functionally very different from the parent. The fact that it cannot return within one epoch suggests that the network has become 'disconnected" from the low loss manifold. To see if these models can return to the low-loss region after more fine-tuning we repeated our experiments with 10 epochs of fine-tuning Appendix D The conclusion is that more fine-tuning amplifies what observe with I epoch of fine-tuning. After small prune, 10 epochs of fine tuning result in network which is more similar to the parent than epoch of fine-tuning. Conversely, after a big prune, 10 epochs of fine-tuning leads to bigger functional change w.r.t. the parent than I epoch of fine-tuning.
5.3 SPARSITY, LARGE PARAMETERS AND THE LOSS LANDSCAPE
We hypothesise that neural networks tend to be (a) stable under increases sparsity (e.g. through L1 Ll-pruning) but also (b) inherently brittle, that say they highly impacted by changes to high- magnitude parameters. These two aspects can be quantified using the kurtosis of a network's 's weight distribution: the higher the kurtosis, the more can be pruned (see and the more high-magnitude

---

parameters it contains. It is interesting to consider the layer-wise kurtosis of network, and in particular its kurtosis of kurtoses which measures the extent to which some layers have significantly particularly brittle: not only
more kurtosis than others. Networks with high kurtosis of kurtoses will they have more layers containing many large parameters, but the layer-wise kurtosis distribution will contain peak of very low-kurtosis layers composed only of now-magnitude parameters which can get entirely pruned away, thereby disconnecting the network altogether 5.4).
To examine the effect of sparsity on the loss landscape, the VGG1 1GG11_DD __BN loss landscape visualised using the Wise Normalization method (Li et al. 2018b). Figure 55 represents the loss land- scape of the parent VGGI I_BN network after 100 epochs of training. Figure [5a] shows the loss landscape of the network after LI Li-pruning to 0.99 cosine similarity. Finally Figure [5c shows the loss landscape of the network after poudomm-pruinng to 0.99 cosine similarity. Again, we choose the VGG11 _BN model because of its complex loss landscape (Lii et al. 2018b).
Figure 2c showed that the pruning method has little impact on the pruned model's accuracy, the main driver of accuracy being cosine similarity with the parent. Similarly, Figure 3c showed that pruned networks with high cosine similarity stayed functionally close to their parent, irrespective of the pruning method. By examining the loss landscapes more globally in []
we see that the pruning method does have an impact, although it is not noticeable in a small neighbourhood around the local parent network, which is why it is not apparent in Figures 2c
Indeed, comparing the loss landscapes of the two pruning methods (Figures 5a and 5c with the loss landscape of the parent (Figure 5b it becomes clear that:
3c
(a) The large increase in the sparsity of the network going from parent Ll Ll-pruned (where 70.06% of the network has been set to zero) has left the loss landscape largely unchanged nf distance between the surfaces of 65.28).
(b) The small changes in the network caused by Random pandoompruuing,.. which zeroes large param- eters with the same probability as small parameters, has let to a very small increase in sparsity (only 1.99% of the parameters have been pruned) but large change in the loss landscape dramatically (! distance between the surfaces of 484.03).
These observations indicate that the loss landscape is heavily determined by the high-value magni- tude parameters of the network. We call this sensitivity large parameters brittleness.
121
424
130
-42
-12
-05
-50
-10
-00
-00
100
213
8.0
65
120
150
010
228
52
000
(a) L1 (70.06%) Pruned
(b) Parent
(c) Random (1.99%) Pruned
Figure 5: VGGI EGGIL_BNT Test Loss Landscape: (a) LI pruning to a cosine similarity of 0.99 resulting in a 70.06% pruned network, (b) Parent network trained 100 epochs and Random pruning to cosine similarity 0.99 resulting in a 1.99% pruned network.
The brittleness of the models we have examined explains why high functional similarity to the parent is required for pruning, as without this, the model does not have enough freedom to reduce the impact of the high-magnitude parameters and thus cannot easily explore other regions of the loss landscape. This idea is supported in part by Fort et al. (2020), which showed that networks with dif- ferent parameter initialisations end up with different functional representations. This phenomenon is well explained by brittleness: since models are disproportionately affected by high-value param- eters, and since different initialisations are very likely to assign high-values to different parameters, it is not surprising that they create such drastically different functions.
Given this observation, we suspect that high parametrisation is required early in training unless given particularly good initialisation, it allows for more degrees of freedom, which in turn makes it easier to transverse the loss landscape as the network is less affected by the initial initialisation of the network and parameters that would otherwise lead to poor performing model.

---

5.4 OPTIMAL COSINE SIMILARITY FOR MAXIMUM MAGNITUDE PRUNING
When pruning is viewed through the lens of cosine similarity and, by proxy, functional similarity to their parent model, it can be reframed into multi-objective optimisation problem, where the goal is to maximise both the cosine similarity and the percentage of the network pruned. In this framing, the LI pruning method is the Pareto frontier, the of all efficient solutions, and the utopia would be a cosine similarity of I to the parent with 100% of the network pruned. The closest point to the Utopia is the optima point for pruning while maintaining the highest accuracy as will have the maximal pruning percentage to cosine similarity ratio, i.e. either side of this point will benefit the other metric more. The exact value of the optimal point will vary depending on the distribution of the network's parameters. In particular, Figure 6 shows that the distance between the optimal pruning and utopia decreases as the kurtosis of a network's 's parameter distribution increases.
1.0
Figure 6: Ll-pruned Pareto Front for 100,000 parameters distributed according to probability laws with decreasing kur- tosis: Log normal(1), Laplace, Logistic, Normal, Cosine and Uniform (0, 1] distributions, when optimising for Cosine Sim- ilarity the parent and percentage Pruned. Displaying the closest point on each front to the Utopia (* at (100,1)).
0.
sem Po foor
opean fopeto-frur
Coare Pusto frunt
1. Jeba "iniz Part
Mapii
Phunce
This observation makes intuitive sense: a distribution with a higher kurtosis will have more high- magnitude parameters which will dominate both the numerator and the denominator of (I) in sim- ilar proportion, allowing for more low-magnitude weights to be Ll-pruned whilst maintaining a high cosine similarity. also explains why the VGG11 BNN network can be pruned more than the LeNll network, which can itself be pruned more than the ResNet18 network: the distribu- tions of 10 trained networks yield average kurtoses of 8.53 for VGGI BN, 4.82 for LeNet. Small and 3.79 for ResNet18.
The Pareto front illustrated in Figure 6 trained network, the method consists in the following steps (1 and 2 are computationally cheap):
determines an optimal L.Ipprunin methodology. Given
Step 1: Compute the Pareto front of the parameters (i.c. the cosine similarity from 0 to 99% prun- ing) and find the closest point to utopia. This gives the optimal pruning amount.
Step 2: Prune by this amount (unless the kurtosis of kurtoses is very high, which case it might be necessary to prune by less than this amount, see MobileNetV2 example below).
Step 3: Fine-tune (by epoch in our case).
CIFARIO models. This pruning procedure is presented graphically in Figure 7 for the LeNet Small, ResNet18 and VGG11 1_N networks. Note that this procedure prunes the LeNet_Small network more aggressively than might be expected from Figure 7b. yet performance remains high after one epoch of fine-tuning (Figure
AcEAUcu %
TORAACUUO 9
-22
GpiLIBAL
Cos: 70PE1a Urap
(a) Step 1
(b) Step 2
(c) Step 3
Figure 7: Optimal magnitude pruning for cosine similarity in Small, and ResNet18 and VGGIL_BN ImageNet models. We demonstrate that the method described above holds at scale and with an 'out- of- of-sample' dataset by applying it the VGG11, ResNet18 and MobileNetV2 models pre-trained on ImageNet and provided by PyTorch. The results are presented in Figure g and show something in- teresting. Whilst our optimal LI Ll-pruning works well for VGG11 and ResNet18, it performs very
By definition, the Utopia is the point whose coordinates are the optimal values of each objective. In general this point is self- celer-cntrai and lays beyond the Pareto frontier.
8

---

poorly for MobileNetV2.. The reason described in 5.3
MobileNetV2 is example of network containing seyyllo--manittuee layers which get pruned away by LI Li-prnning, dramatically modify- ing the behaviour of the network. This phenomenon can be detected by computing the parameters' kurtosis of (layer-wise) kurtoses, reporting values of 1.42 for VGGI 1, 5.81 81 for ResNet18 8 and 64.40 for MobileNet_V V2.
The conclusion of this experiment is that for global unstructured L1 Ll-pruning. the optimal amount of pruning given by the Pareto front of Figure 6 might have be adjusted by a parameter which decreases the network's ' kurtosis of kurtoses (²2) increases. From our limited set of experiments, if the pruning method described above produces poor accuracy, then multiplying the optimal pruning percentage by !/Inx(?)) seems like effective, conservative choice (see Figure 11] Appendix Alternatively, local (layer-wise) cosine similarity pruning might be a better strategy.
Tux Accursay
1p22243
L3gt
1351
35 Pruned
Praned
(a) Step 1
(b) Step 2
(c) Step
Figure 8: Optimal magnitude pruning for cosine similarity in VGG11, and ResNet18 and MobileNetV2
CONCLUSIONS
This paper set out to quantify and explore what makes good prune and answer the questions set out in the introduction through the exploration of global magnitude pruning.
Why can different methods prune by difffernnt amounts yet achieve similar performance? We showed in Section 5.1 cosine similarity with the parent network relates pruning to performance and can be used to compare pruning methods. Two different methods, pruning by different percent- ages, can lead to similar cosine similarities, and thus similar performances. L1-pruning was shown to the optimal method for maintaining cosine similarity. Moreover, we showed in Section 5.2 that cosine similarity is a good proxy for functional similarity. In particular, two different methods, pruning by different percentages, can also lead to functionally similar networks.
not prune at the start of training? Section 5.3 showed that neural networks brittle, i.c. their loss landscapes can change dramatically when high- -aanitude parameters are per- turbed. L1-pruning model before training would result in model with proportionally more high- magnitude parameters (since the ooweitmag ones will have been pruned) and therefore highly susceptible to bad initialisation, making very difficult train.
Why
Why can some models be pruned more than others? This answered in Section 5.4 which shows that the optimal amount of pruning typically increases with the kurtosis of the model's parameter distribution. The exception to this rule is provided by models like MobileNetV2 on which pruning can disconnect entire layers. These situations can be detected by computing the kurtosis of layer- wise kurtoses. Models with high kurtosis of kurtoses can be pruned less that models with low kurtosis of kurtoses.
What is the maximum amount a model can be pruned before affecting the performance? This question is also answered in Section [5.4] which shows that the maximum amount of pruning before affecting performance is the closet point to the utopia (100% pruned with a cosine similarity of 1) on the I.1 Pareto front corresponding to the parameter distribution. This point, the equal trade-off between functional similarity with the parent and sparsity of the network, can be computed ex- actly from the parameter distribution of the trained network, as was illustrated for the VGGI1/LBN, and
LeNet. Small and ResNet18 8 networks in Figure and X
In answering and exploring these questions, we have shown that the optimal global unstructured pruning for maximal cosine similarity is the L1 pruning method and that the optimal point for max- imal pruning without dramatically affecting performance is the closet point to the utopia (100% pruned with cosine similarity of 1) the L.1 Pareto front which can be computed efficiently.

---

REFERENCES
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of Machine Learning and Systems, volume 2, pp. 129- 146, 2020. URL
File/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, et al. Language mod- els are seot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020. URL /ttpo///oo///pppppeeeeaeaaaaaaaa
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1309-1318, 2018.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv: 2020. doi [004855500axxvw1111222777
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, volume 31, 2018. URL
Paper.pdf
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao. Michael W. Mahoney, et al. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 13630, 2021. 400110048550aaXX22113....00
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi 100.0.19///PRR2006..00
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 40.8555055/// w1t.://..aii.111........ 1412.69888
Alex Krizhevsky, Learning multiple layers of features from tiny images. 2009. URL http:
/ww.cce.toorontoccc////e///
features 1509-TR.ppE.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information
processing systems, 2, 1989. URL
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the in-
trinsic dimension of objective landscapes. arXiv preprint arXiv: 180.00880, 2018a.
1001100.455///xXi11100...8
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, vol- ume 31, 2018b. URL http:://pppeeed....eerr..e..aaas
hatpa//aapepppppe....eeee//eep 201 |01/111444aaaa300000000022711133
Kevin P. Murphy, Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, al. Py-
Torch: An Imperative Style, High-Performance Deep Learning Library. In Ad-
vances in Neural Information Processing Systems 32, pp. 8024-8035. 2019. URL
10

---

Karl Pearson. LIII. On lines and planes of closest fit systems of points in space. The London, Edinburgh. and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901. 10. 000000000147864410944777777
Huy Phan. huyunphanngyyorrch..ffffl cifa11. Zenodo, 2021. doi 00011005211/7eeodo.4443333
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016. doi N0.1109//VPR220166
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3): 211-252, 2015. doi
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018. doi 10.1109/CVPR2201.00044
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rxiv:1409.1556, 2014. doi 1409.15500011
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of ma- chine learning research, 9(86):257--200) 2008. URL http: http::///i....rg/ppaeps///
wardermaatendaa..unn
Tim Whitaker and Darrell Whitley. Prune and tune ensembles: low-cost ensemble learning with sparse independent subnetworks. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 36, pp.
2022. URL httpp::/oojs.aaai.crr//exxxxxx/p
A.A/arriicle/viee/226222
A MAXIMAL COSINE SIMILARITY VIA PRUNING PROOF
The proof for Theorem is as follows:
Proof. Assuming that we' re working N for fixed number of dimensions N. Let's write (a, b) for the usual scalar product in N, namely
(a, b) = N. a.b.
Let's write (*) for the vector whose components those of a, apart from the n which is set to zero. Formally
a!)) = =aj 1n anti,...,aN) aN))
We want to show that
(a, (m)))
max n
lall2 |a(m]]] reached for n = arg min ||ai]]-
(3)
To see that this the case, first note that
(a, (m))
Eizr aaa")) (1)
|a
a000
Iaall v E, (n()))
Eign (as)²
|aall v Eisen (as)2
||0 ||a(1)112 ||2
11

---

Therefore,
(a, (n
(m)
mnxx
|a12
max
| 2
Since |all2 is fixed, the maximum is reached for the maximizing expansion, we get that
| 112
By using a Taylor
Σ
(a))2
=
inf
|a,12
infn
a,,12 2 |0,l
lan"" 2
amn
Σ |a
2
8( E. aa (2)
lan
16 Σ, la.12)
Thus ||a(")|2 is maximized by subtracting the series in the smallest possible term |a,, which is precisely the smallest possible term la,, (since squaring is monotone), in other words, the condition of equation
B ALGORITHMS
Algorithm Binary Cosine Similarity Search
Require: model, similarity find, epsilon
low <- 0
high <- 100
while low high do
mid + low high-low
pruned
<- pnunemmodel, mid)
similarity < cosine stmilarity(modeel pruned model) similarity = similarity to find then return mid else if similarity 2 similarity stotttto_orrr fondind then
low < mid epsilon
else
high + mid epsilon
end if
end while
return mid
C MODEL. DETAILS
Model
No. Parameters
Batch Norm
Skip Connections
LetNet Small
343,402
X
ResNet18
11,173,962
VGGTT BN
28.149.514
X
Table I: Model Details
12

---

D THE EFFECT OF ADDITIONAL FINE TUNING
Figure D shows the effect the functional similarity when pruning by cosine similarity, and finetun- ing for additional 10 epochs instead of the one epoch regieme explored in the body of the paper. It shows that after a relatively small prune, 10 epochs of fine tuning results in a network which is more similar to the original than epoch of fine-tuning. Conversely, after big prune, 10 epochs of fine-tuning will lead to bigger functional change w.r.t. to the parent than 1 epoch of fine-tuning
Start Erd
Start
Cossie Similantty Pruning
Breeuring
Fieebadd
-34
6.1
20
-30
Random Cosnn
-20
-20
-30
(a) L1 Prumed and Fine-tuned for 10 epochs
(b) Random Pruned and Fine-tuned for 10 epochs
Figure 9: I-SNE projections of the training dataset predictions for the VGG1 I_BN model, including initiali- sation and end of training (+). Figure 9a (resp. 9b) shows the function space of the models when pruned from 0.1 cosine similarity using L1 (Blue) Random (Red) pruning followed by 10 epochs of fine-tuning.
Figure shows the effect of additional epochs of training with respect to the loss landscape and the test accuracy over a range cosine similarity values, which highlights that ten epochs of fine-tuning can yield some minor improvements, but the network essentially remains in the same loss region as one epoch of fine-tuning.
13

---

Accurecy
Lona
Actaaccy
Orain
Orishn
Porert Optimum Pinciumcz Oulimum
Optimum
Pirent Sptimum
Fanwet Optimum
Beigit dtrction 2
Prieuuel Opcimum
Pincouraa Ostimum
hireturol Opiimum
Werggt direction
Weight Srectoo
Wes
Weight directo
Weggit drectin
Weight disciion
Weight drection
(a) LI pruning to cosine similarity of 0.95
(b) L1 pruning to a cosine similarity of 0.85
Orgin
Grijin
Orgin
Orijit
Pirert Optinum
Farent Ostimum
Porert Optimum
Farent Ostimum
Fin-sune Oztimum
mentueed Optimum
Fineduria Oslimm
tundd Optimum
Weight Waightdiweciin divection
Woggttan direrrtree
Woggt
Weeght
%
Neiggtt direction
Weight diection
Weight direction
Weight diection
(c) LI pruning to cosine similarity of 0.75
(d) L1 pruning to a cosine similarity of 0.65
Test
Aecurayy
Test&ccurcy
Or gin
Grigi
133
Dr
Grigin
Pirert Optinum
Foron: Ostinum
Forent Optimum Finesurned Oztimur
Farent Optinum
Finesunes Optimum
Fretured Optimum
faned Optimum
Widppe to drocction
Wegght
Weegit
drection
drectim
(e) pruning to cosine similarity of 0.55
(I) LI pruning a cosine similarity 0.45
Test Lecs
TestAccurayy
Test Locs
TestAccuaay
Oriin
Crigin
Orgin
Grigin
Optimum
100
Weegght drection
Waegght
Neight direction
Weight dewction
Weight diection
Deight drection
(g) LI pruning to cosine similarity of 0.35
(b) Ll pruning cosine similarity of 0.25
Figure 10: VGG11 BN Parent to Pruned-and-fin fon-dd----tuuedd after 10 epochs Radial Landscape slices with respect test data. The left-hand plots of Figures 100 10h show loss landscape along the paths parent and the pruned-and -nn------n------------- fine-ture tnnettned model, the right-hand plots show the same landscape for accuracy.
14

---

E KURTOSIS OF (LAYER-WISE) KURTOSES ADJUSTED PRUNING
In the scenario where a large kurtosis of (layer-wise) kurtoses observed, it may be beneficial to adjust the optimal amount of pruning given by the Pareto front of Figure 6 by a parameter which decreases as the network's 's kurtosis of kurtoses x²2) increases (from limited set of experiments a/tnx( looks like a sensible, conservative choice). This is shown in Figure 11 with the conservative point shown in purple, with ResNet and MobileNetV2 having the conservative estimate of 51 1% and % respectively, given by (optimal-prenn prrnne * 1/mx(11).
Accurac
Corervativ Edtmer
Usila
Donervttive Divede
(a) Step 1
(b) Step 2
(c) Step 3
Figure 11: Optimal conservative magnitude pruning cosine similarity VGGII, and ResNet18 and
MobileNetV2
5.-
25
(a) VGGII
(b) ResNet18
(c) MobileNetV2
Figure 12: Layerwise Kurtosis
15


Reviews:
[{"review_id": "kCX8gzbxt8", "reviewer": "Reviewer_U3zy", "summary": "The authors explore the use of cosine similarity for quantifying the sparseness-accuracy tradeoff when performing unstructured global pruning of a pretrained neural network. The authors hypothesize that larger values of cosine similarity of the trained weights and the trained weights after unstructured sparsification indicate that the sparsified weights are more amenable to fine-tuning to regain accuracy lost due to pruning. To study this hypothesis, they perform an empirical study using 3 architectures of varying complexity on a single dataset and analyze the cosine similarity of two pruning strategies (random and L1). Research into this problem is motivated by the desire to better understand the complexities of model pruning (e.g., why some pruning strategies and architectures can produce sparse models with higher accuracy).", "questions": "1. At the top of p. 7 you state “It is still unclear whether, given more fine-tuning steps, these models can return to the low-loss region from their current position.” Did you consider exploring this more? I think it would be an interesting and worthwhile to empirically explore this direction by increasing the number of fine-tuning steps to see if the pruned models with lower cosine similarity can regain accuracy lost due to pruning.\n\n2. While I find the premise and findings to be interesting, I think the evaluation is limited in that it is only performed on a single dataset. I think the addition of empirical results at least a larger scale dataset, like ImageNet, and additional unstructured pruning strategies would better support the generalizability of the takeaways.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Findings of cosine similarity on fine-tunability of pruned models:** Figures 3, 4, and 5 are interesting and summative of findings. Particularly, high cosine similarity of pruned weights to original weights enables 1 fine-tuning step to converge to point in loss landscape close to original optimum (i.e., regaining accuracy lost due to unstructured pruning).", "weaknesses": "**Limited evaluation:** Experiments only utilize 3 architectures, 2 pruning strategies, and 1 dataset (CIFAR10). I would expect an empirical paper at ICLR to consider at least one additional dataset (ImageNet) and some additional unstructured pruning strategies (e.g., lottery ticket rewinding) would increase impact of findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9eiOXPSqBL", "reviewer": "Reviewer_cZNh", "summary": "The paper investigates the underlying mechanisms of neural network pruning. It aims to answer questions about why different methods yield similar performance, why pruning cannot be done at the start of training, and how much a model can be pruned without affecting performance. The paper introduces the concept of cosine similarity as an effective measure for functional similarity between the parent and pruned networks. It proves that L1 magnitude pruning is optimal for maintaining maximal cosine similarity and shows that higher kurtosis in a model's parameter distribution allows for more pruning without performance loss. The paper also presents a method to determine the optimal amount of L1-pruning based on a network's parameter distribution.", "questions": "See questions in weaknesses above. Additionally,\n\nIt looks to me you're computing cosine similarity of a vectorized weight vector and its pruned version, the former containing ALL weights in the network and its size would be humongous, how do you deal with that? Also, that weight vector contains weights of different DNN layers which're segregated by nonlinear activations in the network, why grouping them into one huge vector would work at all? More insights or analytical explanations are needed here.\n\nMoreover, only LeNet, ResNet18 and VGG11 are experimented. I would be interested in seeing edge networks like MobileNet to see how \"brittle\" they are, and whether these edge nets are already tight for further pruning.\n\nAs mentioned, can the findings be extended to other types of neural networks, such as recurrent neural networks or Transformers?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper delves into the intricate mechanisms of neural network pruning, providing a understanding of why and how pruning works. This adds a layer of conceptual depth to the existing literature.\n\n2. The paper employs rigorous mathematical proofs to substantiate its claims for the optimality of L1 pruning for maximal cosine similarity. This lends credibility to the research.\n\n3. The paper conducts experiments on multiple architectures like LeNet Small, ResNet18, and VGG11, providing a relatively broad empirical basis for its findings.", "weaknesses": "1.The observation mainly made from the results on the Cifar-10 dataset, whether the observation and conclusion is extendable to other large-scale datasets remain unclear.\n\n2. The paper focuses on specific architectures (LeNet_Small, ResNet18, and VGG11) and does not provide insights into how the findings might generalize to other types of neural networks like Transformers, or other tasks like text understanding. This contradicts the third question, which targets different models.\n\n3. The analysis section, which comprises a significant portion of the paper, lacks logical structure and clarity.\n\n4. Certain observations, such as the point at 80% pruned (Sect. 5.1) in Figures 1c-1d, are confusing, why 80%?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "732Fp0fixB", "reviewer": "Reviewer_AdU1", "summary": "This paper proves that the L1 pruning method is optimal when pruning by cosine similarity. Also it presents a simple method to determine the optimal amount by which a network can be L1-pruned based on its parameter distribution.", "questions": "* Basically, a pruning is supposed to be retrained a lot, repeatedly. It’s unclear how valuable to show that maximizing cosine similarity is the same as removing the least magnitudes (L1 pruning), because the parameters will be retrained (fine-tuned) – then the maintained similarity will be disturbed as well. The theorem holds only when there is no retraining/fine-tuning on L1 pruning.\n\n* The approach does 1-epoch fine-tuning. Is it just for the pruned network? Then what’s the similarity after a 1-epoch fine-tuning? Or if it does not care about cosine similarity after fine-tuning, why does it fine-tune only for 1 epoch, but not multiple times as the SOTA pruning approaches do?\n\n* Can this work be compared with other SOTA pruning approaches?", "limitations": "", "rating": 1, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "* It described the research questions that they would answer clearly early on, and then summarized them again in the conclusion.", "weaknesses": "*\tIt lacks a justification on why the consine similarity needs to be maximized. The (sub-)structure of the pruned network is different from the original network, which means their (combination of) parameters are not necessarily similar. It would have been nicer if the authors described why it should be consine-similar.\n\n*\tThe paper showed that maximizing cosine similarity is to L1-prune. However, that does not tell if the L1-pruned one is anyway the optimally/best pruned network. That is because, again, similar (combination of) parameters of parent and pruned networks do not necessarily mean that the pruned network is the optimally pruned network.\n\n*\tBasically, a pruning is supposed to be retrained a lot, repeatedly. It’s unclear how valuable to show that maximizing cosine similarity is the same as removing the least magnitudes (L1 pruning), because the parameters will be retrained (fine-tuned) – then the maintained similarity will be disturbed as well. The theorem holds only when there is no retraining/fine-tuning on L1 pruning.\n\n*\tAlso, the approach does 1-epoch fine-tuning. Is it just for the pruned network? Then what’s the similarity after a 1-epoch fine-tuning? Or if it does not care about cosine similarity after fine-tuning, why does it fine-tune only for 1 epoch, but not multiple times as the SOTA pruning approaches do?\n\n*\tThis work lacks necessary comparisons with SOTA pruning approaches, such as Weight rewinding [1], Learning rate rewinding [2], and Gradual magnitude pruning [3][4][5]. Please consider comparing it with them. Comparing with Random pruning does not provide extremely interesting information.\n\n*\tIt could be overlooked as a minor issue, but because all the results were shown with only one dataset (CIFAR10), they are not convincing. The work is encouraged to be shown with at least 3 benchmark datasets.\n\n*\tMinor typo: in page 8: in “VGG11 network can be pruned more that the LeNet Small network,”, “that” needs to be “than”.\n\n\n[1] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019\n\n[2] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. ICLR 2020\n\n[3] Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Workshop Track.\n\n[4] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, abs/1902.09574, 2019. \n\n[5] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cPe0Qlj6i", "reviewer": "Reviewer_vmPe", "summary": "This paper presents empirical and some theoretical arguments for making maximal cosine similarity between the parent network and pruning network a more reliable target metric to focus on for optimal pruning and argues for that in a one-shot pruning regime. \n\nThe paper also presents that the longer the tail of the parameter weight distribution it is easier to prune more.\n\nThe brevity of the review doesn't stand for the quality of the review or of the paper. The paper was easy to follow and had a precise goal with only a few comments and questions from my side.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Motivating problem and setup\n2) Precise investigation of what is important.\n3) The proposal of cosine similarity as a proxy is simple, intuitive, and just works\n4) The experiments help us understand that changing proxy metrics for pruning results in a more reliable way to determine better accuracy of pruned networks. \n5) The empirical investigation is on CIFAR across 3 networks. \n6) Furthermore investigation into loss landscapes and transformation of function space provide interesting insights into a very well-studied problem. \n7) The experimentation and analysis to find the optimal cosine similarity are very interesting and further using it for pruning of neural nets to have minimal loss in accuracy.", "weaknesses": "1) The cosine similarity argument while intuitive and powerful is obvious from the magnitude pruning perspective -- however, what makes it interesting is the generality of it over the course of multiple 1% pruning steps.\n2) I understand for every dataset network pair one can find the closest point to utopia, however, this is not sustainable, how to make this scale up across various dataset network pairs at scale? \n3) My major concern is that pruning results on CIFAR-10 often are too easy and need more investigation at Tiny ImageNet and ImageNet scale to verify if the empirical insights translate. I would be very happy to increase my score and advocate for acceptance with the presence of ImageNet results on one or two networks (see Blalock et al., 2020 for best practices)\n\nOn similar lines, the networks used for CIFAR-10 are often way too overparameterized and that would be handled by experiments on ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)