What to Prune and What Not to Prune at Initialization
Authors: Maham Haroon
Venue:  (2021)
Keywords: Network Sparsity
Machine Learning
Initialization Pruning

Abstract: Post-training dropout based approaches achieve high sparsity and are well established means of deciphering problems relating to computational cost and overfitting in Neural Network architectures. Contrastingly, pruning at initialization is still far behind. Initialization pruning is more efficacious when it comes to scaling computation cost of the network. Furthermore, it handles overfitting just as well as post training dropout. It is also averse to retraining losses.

In approbation of the above reasons, the paper presents two approaches to prune at initialization. The goal is to achieve higher sparsity while preserving performance. 1) K-starts, begins with k random p-sparse matrices at initialization. In the first couple of epochs the network then determines the "fittest" of these p-sparse matrices in an attempt to find the "lottery ticket" p-sparse network. The approach is adopted from how evolutionary algorithms find the best individual. Depending on the Neural Network architecture, fitness criteria can be based on magnitude of network weights, magnitude of gradient accumulation over an epoch or a combination of both. 2) Dissipating gradients approach, aims at eliminating weights that remain within a fraction of their initial value during the first couple of epochs. Removing weights in this manner despite their magnitude best preserves performance of the network. Contrarily, the approach also takes the most epochs to achieve higher sparsity. 3) Combination of dissipating gradients and kstarts outperforms either methods and random dropout consistently.

The benefits of using the provided pertaining approaches are: 1) They do not require specific knowledge of the classification task, fixing of dropout threshold or regularization parameters 2) Retraining of the model is neither necessary nor affects the performance of the p-sparse network.

We evaluate the efficacy of the said methods on Autoencoders and Fully Connected Multilayered Perceptrons. The datasets used are MNIST and Fashion MNIST.

Under review as a conference paper at ICLR 2021
WHAT TO PRUNE AND WHAT NOT TO PRUNE AT INI- TIALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Post-training dropout based approaches achieve high sparsity and are well estab- lished means of deciphering problems relating to computational cost and overfit- ting in Neural Network architectures citesrivastava201 14rorr (Pan et 2016). Zhu & Gupta (2017). LeCun et al. (1990). Contrastingly, pruning at initialization is still far behind Frankle et al. (2020). Initialization pruning is more efficacious when it comes to scaling computation cost of the network. Furthermore, it han- dles overfitting just as well as post training dropout. is also averse to retraining losses.
In approbation of the above reasons, the paper presents two approaches prune initialization. The goal is to achieve higher sparsity while preserving performance. K-starts, begins with k random p-sparse matrices initialization. In the first couple of epochs the network then determines the "fittest" of these p-sparse ma- trices in an attempt to find the "lottery ticket" Frankle Carbin (2018) p-sparse network. The approach is adopted from how evolutionary algorithms find the best individual. Depending on the Neural Network architecture, fitness criteria can be based on magnitude of network weights, magnitude of gradient accumulation over an epoch combination of both. 2) Dissipating gradients approach, aims at eliminating weights that remain within a fraction of their initial value during the first couple of epochs. Removing weights in this manner despite their magnitude best preserves performance of the network. Contrarily, the approach also takes the most epochs to achieve higher sparsity. 3) Combination of dissipating gradients and kstarts outperforms either methods and random dropout consistently.
The benefits of using the provided pertaining approaches are: 1) They do not require specific knowledge of the classification task, fixing of dropout threshold or regularization parameters 2) Retraining of the model is neither necessary nor affects the performance of the p-sparse network.
We evaluate the efficacy of the said methods on Autoencoders and Fully Con- nected Multilayered Perceptrons. The datasets used are MNIST and Fashion MNIST.
INTRODUCTION
Computational complexity and overfitting in neural networks is a well established problem Frankle Carbin (2018). Han et al. (2015). LeCun et al. (1990), [Deil et al. (2013). We utilize pruning approaches for the following two reasons: 1) To reduce the computational cost of a fully connected neural network. 2) To reduce overfitting in the network.
Given a large number of post-training pruning approaches (Sriaastava et al. (014)) Geman et al. 19922. ,, Pan al. (2016), the paper attempts to propose two pre-training pruning approaches: kstarts and dissipating gradients. Moreover, it appears to be the case that when isolated from other factors sparse networks outperform fully connected networks. When not isolated they perform at least as well up to a percentage of sparsity depending on the number of parameters in the said network. kstarts and dissipating gradients provide are simple nevertheless effective methods to quickly look for best sparse networks to.
The approaches exploit the knowledge that a network has multiple underlying p-sparse networks that perform just well and in some cases better when contrasted with their fully connected

---

Under review as a conference paper at ICLR 2021
counterparts Frankle & Carbin (2018). What percentage of sparsity is realized, depends largely the number of parameters originally present in the network. Such sparse networks are potent in preventing over-fitting and reducing computational cost.
The poset-training pruning has several approaches in place such as adding various regularization schemes prune the network Louizos al. (2017), Pan al. (2016) using second derivative or hessian of the weights for dropout LerpouttLeCun LeCun et al. (1990), Hassibi & Stork (1993). Han et al.] (2055) an efficient iterative pruning method to iteratively
Alford al. 22019, Zhu & Gupta (2017)
increase sparsity. Srivastava et al. (2014) dropout random hidden units with probability instead of weights avoid overiitiig in general. Each of these approaches effective and achieves good sparsity post-training.
We use simple intuitive models that achieve good results and exploits the fact that a number of sub networks in Neural Network has the potential to individually learn the input Srivastava et al. (2014). We decide on a sparse network early on based on the dropout method and use only that for training. This provides an edge for faster computation, quicker elimination of excess weights and reduced generalization error. The sparsity achieved is superior to random dropout.
Section II gives general introduction to all the methods, section III defines p-sparsity, section IV provides the algorithm for both approaches, section describes experimental setup and results, section VI discusses various design choices, section VII gives general discussion of results, section VIII discusses limitations the approach and section IX provides conclusions and final remarks.
PRUNING METHODS
2.1 KSTARTS
2.1.1 KSTARTS AND EVOLUTIONARY ALGORITHMS
We take the concept of k random starts from Evolutionary Algorithms (Vikhar 2016) that fitness function or heuristic to perform "natural selection" in optimization and search based problems Goldberg & Holland| 1988). It is relatively simple fit genetic algorithms the problem hand. Other method that would be equally effective with a little bit of modification are Hunting Search Oitadeh et al. 2010), Natural Adaptation Strategies (Wierstra al. 2008), firefly algorithm (Yang) 2010) etc.
The basic components the algorithm are: (1) Population: A product network weights and sparse matrices. (2) Individual: An instance of the population. (3) Fitness Function: The heuristic chosen for evaluation of the population.
2.1.2 POPULATION
We first initialize sparse matrices, single instance of these K sparse matrices can be seen in equation ??. In every iteration we multiply model weights W of the Network layer in question with every instance of the sparse matrices. The resulting set matrices is our population for that iteration. Each iteration is referred to as new generation.
population W K SptrseMatrices
2.1.3 INDIVIDUAT.
Each individual, in population of K instances, a sparse matrix of size equal to the size of network weights. W. The number of 0's and 1's in the sparsity matrix determined by the connectivity factor which is further described in section An sparse matrix of p ze 0.3 will have z 50% 0's and Rs 50% 1s.
2

---

Under review as a conference paper at ICLR 2021
2.1.4 EVALUATION/FITNESS FUNCTION
The fitness of an individual ranked by determining the sum of each individual in population as given in [ such that the fittest individual in a generation is given by equation
i4c ind of population)
fittest = arg max and
(2)
where i & c is the size of each individual and ind refers to the individual in population.
2.1.5 NEXT GENERATION SELECTION
Assuming each iteration the next generation. In each generation the fit individual favoured so:
The fittest individual is passed of as weight to the next generation.
Every 5 generations as per the decided elimination frequency, the individual with lowest
fitmess is discarded from the population.
2.2 DISSIPATING GRADIENTS
Magnitude and gradient based pruning approaches popular in post-training pruning SSiiaataaal et al. (2014), LeCun et al. (999)). It does not make much sense to employ them pre-training be- causeof the randomly generated weights. But in order reduce error, any network aims to target updating weights that influence the results most. Based upon that hypothesis, in each epoch we sum the
gradients over and eliminate weights whose weights are not getting updated. equation total number of iterations for an epoch. In equation Depsilon is 1e-6 for all experiments.
N
Accumuulated du
dW
(3)
W dwu
(4)
One consideration in this approach is to not do this for too many epochs which can be only 2 if the image is very monochrome and more than if the gradients are dissipating more slowly. Moreover, specific weights have reached their optimal learning, their gradients will dissipate and we don't t want to eliminate them.
2.3 COMBINATION DROPOUT
Combination dropout is merely combining Kstarts with dissipating gradients. The weights elimi- nated use both approaches. We fix for Kstarts to a certain value of minimum sparsity and further eliminate weights that dissipating gradients method will eliminate well. The approach achieves better performance than either methods.
DEFINING P-SPARSITY
In a p-sparse network layer approximately p percent of connection between two layers are elimi- nated. Figure showss fully connected conventional neural network (figure Tab and three sparsely connected networks with different values (figures 1c Id].
3.1 CONNECTIVITY FACTOR
Connectivity factor, determines the percentage of connections to be removed between any two layers the network. For instance if = 0.0 than the network be fully connected as shown in figure Ia if on the opposite extreme, p=1.0, then there will be connections between two layers of neurons. 0=5.5, figure lb only approximately 50% of the connection in the network remain. If p=0.3, figure Ic approximately 70% of the connection still exist and if p=0.7, figure Id mere 30% the connections are active between two network layers. In short determines percentage of 0's in individual sparse matrix shown in equation ??.

---

Under review as a conference paper at ICLR 2021
fully connected or 0.0-Sparse network layer
(b) sample 0.5- Sparse network layer
sample Sparse network layer
(c)
(d) A sample 0.7-
Sparse network layer
Figure 1: Sample P-Sparse Network Layers
ALGORITHM
The autoencoder, two and three layered neural networks are trained standard manner with adam optimizer and batch update.
4.1 KSTARTS ALGORITHM
Algorithm
[ gives the kstarts method:
We have K number of sparse matrices as shown in equation ?? generated and we call them KI in algorithm |
2. Every time the weight W needs to be updated instead of the usual gradient update we add one step further and using the fitness function, pass the fittest individual as the new W which biases the network towards that sparse matrix.
3. Every 5 iterations, the individual with lowest fitness is dropped.
Algorithm 1: k Random starts
input : Data, params,K,p
output: W.b
initialize W.b:
KT K individuals with approximately 1-p percent active connections;
for maxiterations do
Run Neural Network:
Update weights;
One individual KI is left then
W individual;
else
W individual with maximum sum (of weights);
for every 5 itertiion do
pop individual with minimum sum (of weights) from KI;
end
end
end
4.2 DISSIPATING GRADIENTS ALGORITHM
Algorithm 2 is more simple and just eliminates weights with sum of gradients equal to zero in the first 1 1-4 epochs depending the desired sparsity.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Dissipating Gradients
input : Data, params
output: W.b
initialize W.b;
for maxepochs do
for Maxiterations do
Run Neural Network;
accumulated.dd -- accumulated,ddW+ dW;;
Update weights;
end
accumulated_dc _dW < 0.0001 then
accumulated.dWW. else
accumulated,de dW <-1;
end
W =*accumulatedddo
end
Layered Autoencooer (Fashion MNIST)
Two Layered Autoencoder (MNIST)
00.155
random
disspating gradient cropout
0.045
random dropout
dispppatin gradienn dropout.
0.040
combination diapout
dropout
0.035
2 0.025
5 0.030
0.025
0.020
0.020
0.025
10
Sparsity
Sparstyr1
(a) Dataset: Fashion MNIST
(b) Dataset MNIST
Three Layered NK (Feshion MNIS1)
Three Layered NN (MNIS1)
07
04-
022
dicipating
cropout
50
93
<-ttarts
combination dropout
combinat on drocout
Spars ly1%)
Sparsis()))
(c) Dataset: Fashion MNIST
(d) Dataset : MNIST
Figure 2: Performance of dropout methods with increasing sparsity.
EXPERIMENTS AND RESULTS
The experiments performed on two datasets; MNIST Deng 2012 and Fashion MNIST Xiao et al. 2017). 7) The network architectures two layered Autoencoder (784- 11-1228-44, three layered NN 1184-12-1004 100-10) both with sigmoid activation and adam optimization functions.
The Architecture used for learning curves is single layered NN(784-10).

---

Under review as a conference paper at ICLR 2021
sppaise
p-0.0
pm0.0
D-0025
p.0.25
pi0.5
pi0d
pid 75
pa0.75
40
D-0.3 k-
p-235 k-10
(a) Dataset: Fashion MNIST
(b) Dataset MNIST
Figure 3: Effect of sample size on learning when varying the sparsity of the network with kstarts. Each trial involves running p-sparse network on fixed training set N examples from dataset for 2.5k iterations and is averaged over 10 trials each.
5.1 EFFECT OF INCREASING SPARSITY
As sparsity increases, overall performance reduces. Figure[ the behaviour of various dropout methods presented this paper.
In case of random dropout, it's indeed a random shot. Either no useful weight is eliminated or mul- tiple crucial weights are eliminated which decides how well does random dropout perform. Kstarts performs slightly better average with multiple start choices. Depending on how many indepen- dent p-sparse networks are present in the network that can learn well, one of them can be identified, given k is large enough and the fitness function is smartly decided by first examining the weights and gradients.
Dissipating gradients works well long as the network isn't learning very fast i.c. some weights are being updated in the consequent epochs. It's also most reliable. Combination works by far the best because it does not only rely eliminating weights that are not being updated but also uses kstarts. seems to achieve superior performance as long as p value chosen value that kstarts performs well on.
5.2 VARIATION IN SAMPLE SIZE
Figures 3a and 3b show relationship of varying sample size different sparsity values in a single
layer NN over 2.5k iterations.
The interesting result here that isolated from all other factors like number parameters, hidden units and various design choices, kstarts dropout performs better single layer network compared to even a fully connected network. The standard deviation also lower to fully connected network well as random dropout. kstarts dropout also learns faster than fully connected network. For instance, if the iterations for the experiments in figure [3 and 3b are increased, the fully connected network will eventually reach accuracy of the p-sparse network.
DESIGN PARAMETER CHOICES
There number of design parameter choices for the algorithm presented here. Some explained in detail in this section.
6.1 CHOICE OF FITNESS FUNCTION
Since the fitness function determines the individual being passed to the next generation and the individual being climinated. We had three choices for choosing the fitness of an individual each with it's own pros and cons.

---

Under review as a conference paper at ICLR 2021
Magnitude: As opted here, choose population shown in equation | and then select fitmess using equation 2 This skews the selection of new weights to the previously selected sparse matrix from KI and therefore, the initial sparse matrix will be propagated forward. This also renders elimination to be pointless it does not matter if or all other matrices in K are eliminated. Furthermore, the sparse matrix is picked awfully early in the experiments i.c. only after first 5 iterations or so and that is not when weights have reached a saturation point.
Gradient:
The second choice is to use gradient of the weights to create population as follows:
(5) doing so can have fitness totally dependant on the current update and the new weights are heavily skewed towards the performance of the current iterations which again doesn't 't seem that appropriate.
population 8W SparseMatrices
Sum of Gradients: The third option summing up the gradients for a number of iterations in the manner do for dissipating gradients and then those to create population:
population = oW) SppaseeMatricss
(6)
Doing so skews the network toward weights that are updated quickly and are increasing.
6.2 No. OF LAYERS AND NO. HIDDEN UNITS
We initially use single layer NN (784-10) to isolate the effects of kstarts algorithm from effects of other parameters that may have large impact performance as the size of the network grows. Those parameters i.e. number of hidden layers, regularization choices, types of layers may aid or adversely effect the performance of the algorithm and by performing the experiments unit of neural network we were able to concur that the method is effective in it's own right. After concluding the approach works tested on three layered NN and Autoencoder.
the basic
6.3 COST EFFECTIVESNESS,
One beneficial feature of pre-training pruning approaches is that the best p-sparse network is quickly identified. There a number of methods that exploit sparsity of a matrix for quicker multiplication i.e. Yuster & Zwick 2005). (Buluç & Gilbert 2012) which can be used to quickly retrain large networks. Although that out of scope for our findings.
DISCUSSION
7.1
EFFECT OF K
1. From the experiments done so far lower ( 10) outperforms or at least performs as well higher value of K (+s 100). This be because the times different matrix is chosen by the network, the more times the network has to adopt to learning with only that p-sparse network.
7.2 EFFECT OF
1. A sparse network can outperforms a fully connected network for lower number of iterations and smaller networks.
2. An appropriate value of p. for instance in these experiments p 0.5, seems work best for random dropout, kstarts dropout and combination dropout. A poor choice of p can't seem be remedied by a better choice of K.
3. p can be thought of as an information limiter. For better learning, if the network is only provided with particular features, might have an easier time learning the class specific features in different nodes but this only remains wishful speculation and requires further analysis. Table shows relationship between k.p and no of iterations a single layer NN (784-10)

---

Under review as a conference paper at ICLR 2021
Table 1: MNIST-Mean Accuracy and standard deviation averaged over 10 runs
p=0.0
p=0.5
k=1
k=10
k=50
k=100
iter 10
12.29 = 3.79
13.36 + 4.17
13.94 3.52
9.71 + 4.04
10.14
iter 100
28.41 = 7.42
28.96 3.03
50.53 = 10.7
46.19 10.10
iter
60.71
8.77
6.994
2.97
81.77 + 4.27
iter 10k
82.49 4.3
88.40
$9.07
89.08 + 0.38
89.21 + 0.38
8
LIMITATIONS
There are number of limitations to the approach and further investigation is required in a number of domains.
The NNs used are single and three layered feed forward networks and autoencoders. CNNs are not experimented upon.
2. Only classification tasks are considered and those only two datasets: MNIST and Fash- ion MNIST.
3. Ideally using sparse matrix should make for efficient computation but since the algorithms for that not used it at this point does not show what the time comparison of the ap- proaches will be.
CONCLUSIONS AND FINAL REMARKS
We present two methods for pruning weights pre-training in the first couple of epochs. The comparisons are made against random dropout and both approaches mostly perform better than the random dropout. We provide combination dropout approach that consistently outperforms other dropout approaches. A lot more analysis of the approach required on multiple datasets, learning tasks and network architectures but the basic methods seem to be effective.
REFERENCES
Simon Alford, Ryan Robinett, Lauren Milechin, and Jeremy Kepner. Training behavior of sparse neural network topologies. In 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2019.
Aydin Buluę and John R Gilbert. Parallel sparse matrix- matrix multiplication and indexing: Imple- mentation and experiments. SIAM Journal on Scientific Computing, C4(4)CC17,- 2012.
Li Deng. The mnist database of handwritten digit images for machine learning research [best the web]. IEEE Signal Processing Magazine, 29(6):11-1447 2012.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Pre- dicting parameters in deep learning. In Advances in neural information processing systems, pp. 2148-2156, 2013.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.0368 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.0857,, 2020.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
David E Goldberg and John Henry Holland. Genetic algorithms and machine learning. 1988.
8

---

Under review as a conference paper at ICLR 2021
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pp. 164-171, 1993.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 508-5598 605, 1990.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint vxXiv:/71..01333 2017
Oftadeh, MJ Mahjoob, and M Shariatpanahi. A novel meta-heuristic optimization algorithm inspired by group hunting of animals: Hunting search. Computers & Mathematics with Applica- tions, 60(7):2087-2098, 2010.
Wei Pan, Hao Dong. and Yike Guo. Dropneuron: Simplifying the structure of deep neural networks. arXiv preprint arXiv: /606.07326, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955) 2014.
Pradnya A Vikhar. Evolutionary algorithms: A critical review and its future prospects. In 2016 International conference on global trends in signal processing, information computing and com- munication (ICGTSPICC), pp. 261-265. IEEE, 2016.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In 2008 IEEE Congress Evolutionary Computation (IEEE World Congress Computational Intelligence), pp. 3381 5381-3387. IEEE, 2008
Han Xiao, Kashif Rasul, and Roland Vollgraf.
Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 178177170..07777, 2017.
Xin-She Yang. Firefly algorithm, levy flights and global optimization. In Research and development in intelligent systems XXVI, pp. 209-218. Springer, 2010.
Raphael Yuster and Uri Zwick. Fast sparse matrix multiplication. ACM Transactions On Algorithms (TALG), 111):2-13, 2005.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.000 2017.


Reviews:
[{"review_id": "azrEDsYxfQe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\n\nThis paper proposed two methods, K-starts and dissipating gradients approach to sparsity the network before training and achieve better performance than a random dropout.\n\n\nStrength bullets:\n1. The idea is interesting, but needs more effort to complete it.\n\n\nWeakness bullets:\n1. poor writing, for example, citation error in abstract line 3 //? exist in 2.1.2 line 2 and the last line of 3.1  // the explanation of sparsity is colloquial in 2.1.3 \n2. As for SparseMatrices in equation 1, does it mean the explanation of K? It's easy to confuse with a minus sign.\n3. because the classification on MNIST and FashionMNIST is too simple, the improvement of k-starts is marginal\n4. As dissipating gradient dropout needs training for a couple of epochs, it's unfair to compare with random dropout.\n5. The fatal limitation is the lack of comparison with previous methods and related works, need to compare with other pruning from scratch methods, like GraSP (https://arxiv.org/abs/2002.07376), SNIP (https://arxiv.org/abs/1810.02340).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YrnPZmVbppV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper looks at two new pruning methods, k-starts and dissipating gradients, as well as a combination of both. They both perform better than random, and the combination performs the best.\n\nPros\n* K-starts and dissipating gradients are novel techniques to my knowledge.\n* It makes sense that weights with zero gradient over time can be considered unimportant (though this could mean that you should freeze the weight at its current value rather than prune it and set it to zero). It is also a promising idea to prune early based on gradient info from the early epochs.\n\nCons\n* The results of this paper are limited to MLPs MNIST and Fashion MNIST. Methods that work on these small datasets do not always generalize well to other datasets. MNIST with MLPs is a particularly special case for pruning because of input pixels that are always zero. I would expect to see at least CIFAR-10 and some convolutional networks.\n* While the techniques are compared to each other and to random pruning, there are no comparisons to other existing pruning methods.\n* Some unclear points in writing: pruning seems to be conflated with “dropout”, even though they are two very different techniques. The authors say they use autoencoders, but I believe they should be MLPs if the task is MNIST.\n* Figure 2 should have error bars. Is this test or train loss/accuracy?\n* Section 7.2: performance at different sparsity levels has been studied by many other papers (e.g. https://arxiv.org/pdf/2003.03033.pdf), and it’s not a novel finding that accuracy can increase for some p, but large p causes performance to degrade.\n\nAdditional clarification questions:\n* 2.1.2: text description does not match equation 1, do you multiply the model weights with K and then subtract a matrix as in the equation, or multiply model weights with each of the K matrices as in the text?\n* What is the rationale behind the fitness function in 2.1.4? If you are optimizing for the sum of the individual’s weights (assuming that’s what ind[j] means), why not just optimize for that directly, without using multiple matrices and the evolutionary algorithm? Likewise, in algorithm 1, why not just prune by weight values directly?\n* Equation 3: it supposed to be a sum over the absolute values of the gradients? If not, weights with strong negative gradients should be treated the same as those with strong positive gradients.\n* Section 4: what are the details of “standard manner” and what is “batch update”?\n\nMinor: there are some latex errors, such as “cite” in the abstract, invalid equation references, and incorrect quote formatting. The abstract should probably be shorter. \n\nOverall: this paper poses two novel techniques for pruning, but do not evaluate them thoroughly on enough models and datasets and with comparisons to other pruning methods. Due to the limited scope, I do not recommend acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3tVtAYTFUYu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# No Rebuttal\n\nSince there is no rebuttal, I have not modified my score.\n\n# Overall\n\nThis paper presents some interesting ideas. In particular, this is the first time I have seen genetic algorithms used in the context of pruning, and I encourage the authors to explore this concept further. However, the paper has some severe weaknesses:\n\n(1) The paper doesn't appear to be complete. The latex is a mess, with broken references, missing formulas, citations that aren't in parentheticals, math notation that is broken, etc. Key aspects of the methodology (as mentioned in the notes below) are missing. This paper was not in submittable condition, and the authors should probably have continued to work on the paper and submitted it to another conference.\n\n(2) The two methods studied in the paper have little to do with each other. It's unclear why they're both included in the same paper, and they should be studied separately. The dissipated gradients method has little justification itself, and it is simply declared.\n\n(3) The paper only studies MNIST and FashionMNIST, which are known to allow for many findings that do not scale to larger networks. I encourage the authors to extend their method to a larger-scale setting (e.g., ResNet-20 on CIFAR-10) to show that the results also work there.\n\n(4) The paper does not include any baselines, including other early pruning methods, making it impossible to evaluate whether these results are trivial or SOTA in any way. The authors cannot plead ignorance on this matter: they cite a paper (Frankle et al., 2020) in their abstract that explicitly surveys methods for pruning at initialization. In fact, one of the primary findings of Frankle et al. 2020 is that it's important to include baselines.\n\nI cannot recommend this paper for acceptance, and I suggest that the authors re-submit the paper at a time when it is in more complete shape.\n\n# Notes\n\n## Abstract\n\nThe citations in the abstract are garbled.\n\nThe abstract mentions two approaches but three are given.\n\nMNIST and FashionMNIST are not sufficient datasets to justify any pruning results. Many pruning results (e.g., lottery tickets) work on MNIST but not on larger-scale settings. The authors will need to include a larger-scale setting (e.g., CIFAR-10 and ResNet-20) to substantiate any results in this paper.\n\n## Intro\n\nThe authors should use the \"\\citep\" macro to ensure that citations are parenthetical rather than appearing in the middle of the text.\n\nPruning was generally claimed as a way to reduce overfitting in the 1980s and early 1990s, but it generally has not been used in this way in almost 30 years. Today, it is used solely to reduce costs (e.g., storage, running time, FLOPs, etc.).\n\nPruning is used on all sorts of networks, not just fully-connected networks.\n\nThe citations and sentence structure are garbled throughout this section.\n\n## Section 2\n\nI have no idea what Equation 2 means, but I assume you're keeping population members that lead to the highest accuracy before any training occurs?\n\nI assume that, once an individual makes it to the next generation, its weights are perturbed in some way to create new members of that generation? This doesn't seem to be explained in the text.\n\nWhat is the justification for dissipated gradients? Why this design and not other designs.\n\nDissipated gradients and the genetic algorithm (GA) approach are completely unrelated. Why are they in the same paper? Perhaps each should receive consideration in a separate paper.\n\nWhy combine these two approaches and not any two other approaches?\n\n## Section 5\n\nThe paper frequently uses the word \"Dropout\" when it means \"Pruning.\" Dropout is something different entirely.\n\n## Section 6:\n\nWhat does the magnitude heuristic mean? Why don't these fitness heuristics appear in Section 2?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2nByxB4bQkz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose two approaches for pruning: (a) \"Evolution-style\": start with K random masks associated with the weights, update weights on gradient descent corresponding to those active in the “fittest” mask, and overtime throw away all but one masks which are less fit. (b) \"Dissipating-gradients”: Here those weights are removed which are not being updated as much, measured by their sum of gradients over a number of iterations. This is shown for elementary networks on MNIST datasets without any serious experiments or comparisons or even presentation. \n\n\nPros:\n-It is quite possible that these nuggets of “ideas” proposed in this paper could be useful, in some shape. But right now their form is *as rough as a draft can possibly get*. \n-Another thing which is mentioned rightfully in their paper is the “Limitations” section, of which are plenty.\n\nCons:\nEmpirical analysis\n-No serious experiments or comparisons to any baseline apart from random pruning. Even so, all networks used comprise of a few layers on MNIST & Fashion MNIST. \n-The performance of their proposed methods is basically on par with random pruning, sometimes even worse (Figure 2)!\n-Why do you sum over gradients? Doesn’t it make more sense to sum the absolute value of gradients? (There could be a weight which is changing a lot, but at the time when you decided to prune, its sum of updates was very small.)\n-What happens in Figure 3a and 3b if you use all the samples from MNIST?\n-Given the extremely rough shape of the paper, I don’t think if the other results should even be taken seriously!\n\nPresentation:\n-The authors keep saying post-training “dropout”, instead of post-training “pruning”. Dropout is just used as a verb in place of pruning. It’s not that authors do not know that Dropout and they cite the corresponding paper. Yet, it is used by the authors as if dropout and pruning are the same things.\n-The equations are so poorly written, why even bother?\n-The writing is absolutely dismal, I don’t even think I could enumerate over all the issues here. \n-There are several vague statements: \"It is also averse to retraining losses.”, \"which can be only 2 if the image is very monochrome”, etc.\n-There is one paragraph on the connectivity factor, which is trivial. It’s basically what fraction of weights are pruned. Funnily, this concept is accompanied with diagrams. \n\n\nSorry if this whole review sounds rude, but to be honest I am amazed that authors thought about submitting a paper in this state to ICLR.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work appears to be a promising start to a research direction. However, as the reviewers noted, the work does not compare to alternative approaches and the presentation of the work overall is incomplete.
Decision: Reject