Revisiting the Lottery Ticket Hypothesis: A Ramanujan Graph Perspective

1 INTRODUCTION AND RELATED WORK . Neural Network ( NN ) and its recent advancements have made a significant contribution to solve various machine learning applications . The power of an over-parameterized NN lies in its capability to learn simple patterns and memorize the noise in the data ( Neyshabur et al. , 2018 ) . However , the training of such networks requires enormous computational resources , and often the deployment onto low-resource environments such as mobile devices , or embedded systems becomes difficult . Recent trend in research to reduce training time of deep neural networks has shifted towards pretraining following the introduction of a remarkable contribution , named the Lottery Ticket Hypothesis ( LTH ) , which hypothesize the existence of a highly sparse subnetwork and weight initialization to reduce the training resources as well ( Frankle & Carbin , 2019 ) . It uses a simple iterative , magnitudebased pruning algorithm , and empirically shows that even after removing approximately 90 % of the weights , the subnetwork can preserve the original generalization error . In the subsequent studies , the focus goes on finding this lottery ticket for more competitive tasks by pruning with weight rewinding ( Frankle et al. , 2019a ) , fine tuning the learning rates ( Renda et al. , 2020 ) , more efficient training ( You et al. , 2019 ; Brix et al. , 2020 ; Girish et al. , 2021 ) . Neural network pruning involves sparsification of the network ( LeCun et al. , 1990 ; Blalock et al. , 2020 ) . It identifies the weight parameters , removal of which incurs minimal effect on the generalization error . There exists different categories of pruning based on ( i ) how the pruning is performed , for instance based on the weight magnitude ( Han et al. , 2015 ; Zhu & Gupta , 2017 ) , gradient in the backpropagation , hessian of the weight ( Hassibi et al. , 1993 ; Dong et al. , 2017 ; Lee et al. , 2018 ) , etc ; ( ii ) whether the pruning is global or local ; ( iii ) how often pruning should be applied like one-shot Lee et al . ( 2018 ) ; Wang et al . ( 2020 ) , iterative Tanaka et al . ( 2020 ) . One of the primary goals in the literature has been to reduce the computational footprint at the time of prediction , i.e. , during post-training . In recent LTH studies , the victim weights are determined by their value at the ini- tialization , gradient of the error , and network topology Lee et al . ( 2019 ) ; Tanaka et al . ( 2020 ) . To understand weight initialization , Malach et al . ( 2020 ) show that pruning makes a stronger hypothesis with bounded weight distribution . The sparsity of the network is reduced from polynomial to a logarithmic factor of the number of training variables ( Orseau et al. , 2020 ) . Mocanu et al . ( 2018 ) suggest to consider the topology of the network from a network science point of view . The pruning algorithm starts from a random Erdős Rényi graph and returns a scale-free network of a high sparsity factor based on the number of neurons in each layer . The method is further evolved for convolution layers considering both the magnitude and gradient of the weights ( Evci et al. , 2020a ) . Various analysis for explaining the LTH have been attempted in the past . Researchers Evci et al . ( 2020b ) explain empirically why the LTH works through gradient flow at different stages of the training . Despite previous attempts to explain why the Lottery Ticket Hypothesis works , the underlying phenomenon associated with the hypothesis still remains ill-understood . All of these studies related to LTH identify that a sparse sub-network can be trained instead of a complete network and the network needs to be connected from input to output layers . However , none of them try to explain the LTH and the properties of the pruned network through the lens of spectral graph theory . The network connectivity can be described from the graph expansion point of view , where any subset of vertices of size less than or equal to half of the number of vertices in a graph , is adjacent to at least a fraction of the number of vertices in that set ; for details , see ( Lubotzky , 2010 ) . Graphs satisfying this property are known as expander graphs . The Ramanujan Graph is a special graph in a bounded degree expander family , where the eigenbound is maximal ( Nilli , 1991 ) . This leads to a maximum possible sparsity of a network while preserving the connectivity . In this paper , we initiate a study to observe the characteristics of a pruned sub-network from the spectral properties of its adjacency matrix , which , has not been reported previously . We represent a feed-forward neural network as a series of connected bipartite graphs . Both weighted and unweighted bi-adjacency matrices are considered . The Ramanujan graph properties of each of the bipartite layers are studied . We use the results of Hoory ( 2005 ) on the bound of spectral gap for the weight matrix of a pruned network . It is empirically observed that existence of winning tickets in a pruned network is dependent on the Ramanujan graph properties of the bipartite layers . As network sparsity increases with more aggressive pruning , we obtain regions where test set accuracy do not decrease much and the bipartite layers satisfy Ramanujan graph property . Subsequently we obtain regions where the Ramanujan graph properties are lost for all the layers and test accuracy decreases sharply . Also , we observe that the impact of noise in the data , on test set accuracy is more adverse when some of the layers lose their Ramanujan graph properties . Experimental results are presented for the Lenet architecture on the MNIST dataset and the Conv4 architecture on the CIFAR10 dataset . Results for other popular feed-forward network are presented in the Appendix . We suggest that preservation of Ramanujan graph properties may benefit existing network pruning algorithms . We propose a modified pruning algorithm that uses the spectral bounds . The algorithm identifies network layers that may still be pruned further , while avoiding pruning in layers that have already lost their Ramanujan graph property . Neural network weight score functions suggested by various pruning algorithms are used to represent the bipartite layers as weighted graphs . Spectral bounds for these graphs are used to verify the Ramanujan property . For a number of popular pruning algorithms , we experimentally demonstrate significant improvement in accuracy for sparse networks by using these connectivity criteria . Contributions : The contributions of the paper are the followings : 1 . We propose a methodology for analyzing winning lottery tickets with respect to their spectral properties . 2 . We empirically observe that winning lottery tickets often satisfy layerwise bipartite Ramanujan graph property representing a sparse but resiliently connected global network . The property is checked using spectral bounds that generalize to irregular networks . We also notice better noise robustness when all the layers of the pruned sparse networks preserve the Ramanujan graph property . 3 . Based on the above property we propose a modified iterative network pruning algorithm that attempts to preserve Ramanujan graph property for all the layers even at low network densities . It identifies layers that are still amenable to pruning while avoiding further pruning in layers that have lost their Ramanujan graph property . We report significant performance improvement for a number of popular pruning algorithms modified using this criteria . 2 THE LOTTERY TICKET HYPOTHESIS AND NETWORK PRUNING . The lottery ticket hypothesis states that a randomly initialized , dense neural network contains a subnetwork which when trained independently using the same initialization achieves a test accuracy close to the original network after training for less or at most the same number of iterations Frankle & Carbin ( 2019 ) . These subnetworks , denoted as ‘ winning tickets ’ , can be uncovered by network pruning algorithms . Weight pruning is one such simple strategy . Let the original dense network be represented as the function N ( x ; θ ) , where x is the input and θ are the weights . The weights have an initialization of θ0 . Weight pruning generates a mask m ∈ { 0 , 1 } |θ| such that the pruned network can be represented by N ( x ; m⊙ θ ) with initialization m⊙ θ0 . Pruning algorithms that are used to obtain the winning tickets can be either one-shot or iterative . In one-shot pruning the original network is trained to convergence , then p % of weights are pruned and the surviving weights are re-initialized to their values in θ0 followed retraining/fine tuning the subnetwork . Here , the network training and pruning are not simultaneously performed and pruning occurs only after convergence is reached . Iterative pruning repeats one-shot pruning over several iteration . This often leads to higher pruning percentage while retaining test set accuracy . However , iterative pruning is more time consuming than one-shot pruning . After pruning the surviving weights may alternately be initialized to their values in θi , i denoting a small iteration number , rather than their initial values in θ0 . This process , as illustrated in Figure 1 ( Frankle et al. , 2019b ) , is denoted as rewinding and is effective for large networks . We adopt this in our study . Various scoring functions are used to prioritize the weights for pruning . They may be based on weight magnitudes , gradient , information flow ( Blalock et al. , 2020 ; Hoefler et al. , 2021 ) or saliency Tanaka et al . ( 2020 ) . Magnitude pruning provides a simple method for obtaining the pruning mask by retaining the top p % weights wi ∈ θ that have the highest values of |wi| . The role of the weights in local computation in the network layers is not considered . A higher pruning efficiency may be achieved by algorithms that account for connectivity structures in the individual layers . In the next section we describe graph parameters of the network that determine such connectivity . 3 EXPANDERS AND RAMANUJAN GRAPHS . Expanders are highly connected , and yet sparse graphs . In this work , we shall be considering finite , connected , undirected , but not necessarily regular graphs . Recall that the degree of a vertex v in a graph is the number of half edges emanating from v. Definition 1 ( ( n , d , ε ) -expander ) Let ε > 0 . An ( n , d , ε ) -expander is a graph G = ( V , E ) on |V | = n vertices , having maximal degree d , such that for every set ∅ ≠ U ⊆ V satisfying |U | ≤ n2 , |δ ( U ) | ≥ ε|U | holds . Here , δ ( U ) denotes the vertex boundary of U . The quantity |δ ( U ) ||U | measures the rate of expansion and the infimum |δ ( U ) ||U | as U varies among the non-empty subsets of V with |U | ≤ |V | 2 is called the ( vertex ) Cheeger constant h ( G ) of the graph G. The higher the value of h ( G ) , the more expansion property it exhibits and vice versa . Expansion and the Cheeger constant quantifies the connectivity properties of a graph as a high value of h ( G ) signifies that the graph is strongly connected . This ensures that information can flow freely without much bottlenecks . Definition 2 ( Expander family ) A sequence of finite , connected graphs { Gi = ( Vi , Ei ) } i=1,2 , ··· on Vi vertices and Ei edges is called an expander family if there exists an uniform ϵ > 0 such that each graph in the sequence is an ( |Vi| , di , ϵ ) expander for some di ’ s . The study of expansion properties of graphs is closely related to the study of the spectrum ( distribution of eigenvalues ) of the adjacency operator defined on them . Given a finite r-regular graph of size |V | = n , the eigenvalues ti of the adjacency matrix are all real and they satisfy , −r ≤ tn ≤ . . . ≤ t2 ≤ t1 = r. The graph is connected iff t2 < t1 and is bipartite iff ti ’ s are symmetric about 0 ( in particular tn = −r ) . The quantity t1 − t2 is known as the spectral gap and it is related to the Cheeger constant via the discrete Cheeger-Buser inequality , discovered independently by Dodziuk ( 1984 ) and by Alon & Milman ( 1985 ) . In our context , we consider a stronger notion of the spectral gap ( but it is equivalent for bipartite graphs ) . Let t : = max { |ti| : 1 < i ≤ n , |ti| < t1 } . In here , the quantity t1 − t will denote the spectral gap . The more connected a graph is , the larger is the spectral gap and ideally , a graph with strong expansion properties has a very large spectral gap . However , for a bounded degree expander family , this spectral gap can not be arbitrarily large . This brings us to the notion of Ramanujan graphs . Definition 3 ( Ramanujan graph ) Let G be a r-regular graph on n vertices , with adjacency eigenvalues { ti } i=1,2 , ... n , satisfying −r ≤ tn ≤ . . . ≤ t2 ≤ t1 = r. Let t ( G ) : = max { |ti| : 1 < i ≤ n , |ti| < t1 } . Then G is a Ramanujan graph if t ( G ) ≤ 2 √ r − 1 = 2 √ t1 − 1 . The fact that in a bounded degree expander family , the eigenvalue bound in Ramanujan graphs is maximal can be deduced from the following result due to Alon , see Nilli ( 1991 ) , t ( G ) ≥ 2 √ r − 1− 2 √ r−1−1 ⌊m/2⌋ where m denotes the diameter of the graph . As m → +∞ and r is bounded , we obtain the result ( this also follows from the Alon-Boppana theorem ) . For our applications , we shall encounter not necessarily regular graphs , thus we need a notion of irregular version of Ramanujan graphs . The two ways we shall exploit Definition 3 for irregular graphs will be to : 1 . Use the average degree davg in place of the regularity . 2 . For weighted graphs , use t1 , the largest eigenvalue of the adjacency matrix . Note that a motivation for considering the above bounds comes from the following generalisation of the definition of Ramanujan graphs to irregular graphs . For a finite , connected graph G ( not necessarily regular ) consider its universal cover G̃ , for details see ( Hoory et al. , 2006 , sec . 6 ) . A Ramanujan graph is a graph satisfying t ( G ) ≤ ρ ( G̃ ) where ρ ( G̃ ) denotes the spectral radius of G̃ . See also Marcus–Spielman–Srivastava ( Marcus et al. , 2015 , sec . 2.3 ) . A result of Hoory , see Hoory ( 2005 ) implies that 2 √ davg − 1 ≤ ρ ( G̃ ) . Thus , it makes sense to consider t ( G ) ≤ 2 √ davg − 1 ≤ ρ ( G̃ ) . The above consideration also result in extremal families , Hoory ( 2005 ) . Further , for irregular bipartite graphs , with minimal degree at least two and an average degree davgL on the left and davgR on the right , we can further consider the sharper estimate t ( G ) ≤ √ davgL − 1 + √ davgR − 1 ≤ ρ ( G̃ ) see Hoory ( 2005 ) . Upto now , we had only discussed about unweighted graphs . A weighted graph is a graph with a weight function w : E → R≥0 attached on the edges . It can be looked upon as a generalisation of unweighted graphs , in the sense that in the unweighted case , the function w takes values in the set { 0 , 1 } . In the case of weighted networks , we shall use the absolute values of the weight functions according to the architecture for the corresponding dataset . This also means that in the case of weighted graphs , we need to modify the definition of the edge set of the graph to incorporate multiple ( as well as fractional ) edges . The theory of characterisation of weighted Ramanujan graphs is not well developed . However , characterisation of weighted expanders ( with positive weights ) exist , due to the Cheeger inequality for such graphs , see ( Chung , 1996 , sec . 5 ) and we use the largest eigenvalue of the adjacency matrix in place of the regularity . In the case of regular graphs , it coincides with the notion of Ramanujan graphs and even in the general case , by the Cheeger inequality , it ensures a large expansion which in turn implies that there is no bottleneck to the free flow of information . This forms the theoretical basis of our work .

Summary: This paper revisits the Lottery Ticket Hypotheis (LTH), where aggressively pruning a neural network in a smart way allows it to retain its convergence properties, at a lesser computational cost.  The authors advance an explanation relating this LTH to expanders, graphs that have the property that information circulates really well between the vertices, and then to Ramanujan graphs. They then propose an algorithm built on this premise, that prunes the graph as long as the ramanujan property is still satisfied.  Finally, a wealth of experiments is shown, showing the relation between the Ramanujan property and the performance of the pruned network. The authors also test the performance of their Ramanujan pruning algorithm against the same datasets.