What to Prune and What Not to Prune at Initialization

1 INTRODUCTION . Computational complexity and overfitting in neural networks is a well established problem Frankle & Carbin ( 2018 ) , Han et al . ( 2015 ) , LeCun et al . ( 1990 ) , Denil et al . ( 2013 ) . We utilize pruning approaches for the following two reasons : 1 ) To reduce the computational cost of a fully connected neural network . 2 ) To reduce overfitting in the network . Given a large number of post-training pruning approaches Srivastava et al . ( 2014 ) , Geman et al . ( 1992 ) , Pan et al . ( 2016 ) , the paper attempts to propose two pre-training pruning approaches : kstarts and dissipating gradients . Moreover , it appears to be the case that when isolated from other factors sparse networks outperform fully connected networks . When not isolated they perform at least as well up to a percentage of sparsity depending on the number of parameters in the said network . kstarts and dissipating gradients provide are simple nevertheless effective methods to quickly look for best sparse networks to . The approaches exploit the knowledge that a network has multiple underlying p-sparse networks that perform just as well and in some cases even better when contrasted with their fully connected counterparts Frankle & Carbin ( 2018 ) . What percentage of sparsity is realized , depends largely on the number of parameters originally present in the network . Such sparse networks are potent in preventing over-fitting and reducing computational cost . The poset-training pruning has several approaches in place such as adding various regularization schemes to prune the network Louizos et al . ( 2017 ) , Pan et al . ( 2016 ) or using second derivative or hessian of the weights for dropout LeCun et al . ( 1990 ) , Hassibi & Stork ( 1993 ) . Han et al . ( 2015 ) , Alford et al . ( 2019 ) , Zhu & Gupta ( 2017 ) use an efficient iterative pruning method to iteratively increase sparsity . Srivastava et al . ( 2014 ) dropout random hidden units with p probability instead of weights to avoid overfitting in general . Each of these approaches is effective and achieves good sparsity post-training . We use a simple intuitive models that achieve good results and exploits the fact that a number of sub networks in a Neural Network has the potential to individually learn the input Srivastava et al . ( 2014 ) . We decide on a sparse network early on based on the dropout method and use only that for training . This provides an edge for faster computation , quicker elimination of excess weights and reduced generalization error . The sparsity achieved is superior to random dropout . Section II gives a general introduction to all the methods , section III defines p-sparsity , section IV provides the algorithm for both approaches , section V describes experimental setup and results , section VI discusses various design choices , section VII gives a general discussion of results , section VIII discusses limitations of the approach and section IX provides conclusions and final remarks . 2 PRUNING METHODS . 2.1 KSTARTS . 2.1.1 KSTARTS AND EVOLUTIONARY ALGORITHMS . We take the concept of k random starts from Evolutionary Algorithms ( Vikhar , 2016 ) that use a fitness function or heuristic to perform ” natural selection ” in optimization and search based problems ( Goldberg & Holland , 1988 ) . It is relatively simple to fit genetic algorithms to the problem at hand . Other method that would be equally effective with a little bit of modification are Hunting Search ( Oftadeh et al. , 2010 ) , Natural Adaptation Strategies ( Wierstra et al. , 2008 ) , firefly algorithm ( Yang , 2010 ) etc . The basic components of the algorithm are : ( 1 ) Population : A product of network weights and sparse matrices . ( 2 ) Individual : An instance of the population . ( 3 ) Fitness Function : The heuristic chosen for evaluation of the population . 2.1.2 POPULATION . We first initialize K sparse matrices , a single instance of these K sparse matrices can be seen in equation ? ? . In every iteration we multiply model weights W of the Network layer in question with every instance of the K sparse matrices . The resulting set of matrices is our population for that iteration . Each iteration is referred to as a new generation . population =W ∗K − SparseMatrices ( 1 ) 2.1.3 INDIVIDUAL . Each individual , in a population of K instances , is a sparse matrix of size equal to the size of network weights , W. The number of 0 ’ s and 1 ’ s in the sparsity matrix are determined by the connectivity factor p which is further described in section 3 . An sparse matrix of p ≈ 0.5 will have ≈ 50 % 0 ’ s and ≈ 50 % 1s . 2.1.4 EVALUATION/FITNESS FUNCTION . The fitness of an individual is ranked by determining the sum of each individual in population as given in 1 such that the fittest individual in a generation is given by equation 2. fittest = argmax ind i∗c∑ j=1 ind [ j ] ( of population ) ( 2 ) where i ∗ c is the size of each individual and ind refers to the individual in population . 2.1.5 NEXT GENERATION SELECTION . Assuming each iteration is the next generation . In each generation the fit individual is favoured so : • The fittest individual is passed of as weight to the next generation . • Every 5 generations or as per the decided elimination frequency , the individual with lowest fitness is discarded from the population . 2.2 DISSIPATING GRADIENTS . Magnitude and gradient based pruning approaches are popular in post-training pruning Srivastava et al . ( 2014 ) , LeCun et al . ( 1990 ) . It does not make much sense to employ them pre-training becauseof the randomly generated weights . But in order to reduce error , any network aims to target updating weights that influence the results most . Based upon that hypothesis , in each epoch we sum gradients over and eliminate weights whose weights are not getting updated . In equation 3 N is the total number of iterations for an epoch . In equation 4 epsilon is 1e-6 for all experiments . Accumulated dw = N∑ i=1 dW ( 3 ) W [ Accumulated dw < ] = 0 ( 4 ) One consideration in this approach is to not do this for too many epochs which can be only 2 if the image is very monochrome and more than 2 if the gradients are dissipating more slowly . Moreover , once specific weights have reached their optimal learning , their gradients will dissipate and we don ’ t want to eliminate them . 2.3 COMBINATION DROPOUT . Combination dropout is merely combining Kstarts with dissipating gradients . The weights eliminated use both approaches . We fix p for Kstarts to a certain value of minimum sparsity and further eliminate weights that dissipating gradients method will eliminate as well . The approach achieves better performance than either methods . 3 DEFINING P-SPARSITY . In a p-sparse network layer approximately p percent of connection between two layers are eliminated . Figure 1 shows a fully connected conventional neural network ( figure 1a ) and three sparsely connected networks with different values of p ( figures 1b , 1c , 1d ) . 3.1 CONNECTIVITY FACTOR . Connectivity factor , p , determines the percentage of connections to be removed between any two layers of the network . For instance if p = 0.0 than the network is be fully connected as shown in figure 1a , if on the opposite extreme , p=1.0 , then there will be no connections between two layers of neurons . If p=0.5 , figure 1b , only approximately 50 % of the connection in the network remain . If p=0.3 , figure 1c , approximately 70 % of the connection still exist and if p=0.7 , figure 1d , a mere 30 % of the connections are active between two network layers . In short p determines percentage of 0 ’ s in an individual sparse matrix shown in equation ? ? . 4 ALGORITHM . The autoencoder , two and three layered neural networks are trained in a standard manner with adam optimizer and batch update . 4.1 KSTARTS ALGORITHM . Algorithm 1 gives the kstarts method : 1 . We have K number of sparse matrices as shown in equation ? ? generated and we call them KI in algorithm 1 . 2 . Every time the weight W needs to be updated instead of the usual gradient update we add one step further and using the fitness function , pass the fittest individual as the new W which biases the network towards that sparse matrix . 3 . Every 5 iterations , the individual with lowest fitness is dropped . Algorithm 1 : k Random starts input : Data , params , K , p output : W , b initialize W , b ; KI ← K individuals with approximately 1-p percent active connections ; for maxiterations do Run Neural Network ; Update weights ; if One individual in KI is left then W← individual ; else W← individual with maximum sum ( of weights ) ; for every 5 iterations do pop individual with minimum sum ( of weights ) from KI ; end end end 4.2 DISSIPATING GRADIENTS ALGORITHM . Algorithm 2 is more simple and just eliminates weights with sum of gradients equal to zero in the first 1-4 epochs depending on the desired sparsity . Algorithm 2 : Dissipating Gradients input : Data , params output : W , b initialize W , b ; for maxepochs do for Maxiterations do Run Neural Network ; accumulated dW← accumulated dW+ dW ; Update weights ; end if accumulated dW < 0.0001 then accumulated dW←0 ; else accumulated dW←1 ; end W←W * accumulated dW ; end 5 EXPERIMENTS AND RESULTS . The experiments performed on two datasets ; MNIST Deng ( 2012 ) and Fashion MNIST Xiao et al . ( 2017 ) . The network architectures are two layered Autoencoder ( 784-128-64 ) , a three layered NN ( 784-128-100-10 ) both with sigmoid activation and adam optimization functions . The Architecture used for learning curves is a single layered NN ( 784-10 ) . 5.1 EFFECT OF INCREASING SPARSITY . As sparsity increases , overall performance reduces . Figure 2 shows the behaviour of various dropout methods presented in this paper . In case of random dropout , it ’ s indeed a random shot . Either no useful weight is eliminated or multiple crucial weights are eliminated which decides how well does random dropout perform . Kstarts performs slightly better on average with multiple start choices . Depending on how many independent p-sparse networks are present in the network that can learn well , one of them can be identified , given k is large enough and the fitness function is smartly decided by first examining the weights and gradients . Dissipating gradients works well as long as the network isn ’ t learning very fast i.e . some weights are being updated in the consequent epochs . It ’ s also most reliable . Combination works by far the best because it does not only rely on eliminating weights that are not being updated but also uses kstarts . It seems to achieve superior performance as long as p value chosen is a value that kstarts performs well on .

Summary: The authors propose two approaches for pruning: (a) "Evolution-style": start with K random masks associated with the weights, update weights on gradient descent corresponding to those active in the “fittest” mask, and overtime throw away all but one masks which are less fit. (b) "Dissipating-gradients”: Here those weights are removed which are not being updated as much, measured by their sum of gradients over a number of iterations. This is shown for elementary networks on MNIST datasets without any serious experiments or comparisons or even presentation.