Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity

1 INTRODUCTION . Recent successful advances of Deep Neural Networks are commonly attributed to their high architectural complexity and excessive size ( over-parametrization ) ( Denton et al. , 2014 ; Neyshabur et al. , 2019 ; Arora et al. , 2018 ) . Modern state-of-the-art architectures exhibit enormous parameter overhead , requiring prohibitive amounts of resources during both training and inference and leaving a significant environmental footprint ( Shoeybi et al. , 2019 ) . In response to these challenges , much attention has turned to compression of neural networks and , in particular , parameter pruning . While initial approaches mostly focused on pruning models after training ( LeCun et al. , 1990 ; Hassibi et al. , 1993 ) , contemporary algorithms optimize the sparsity structure of a network while training its parameters ( Mocanu et al. , 2018 ; Evci et al. , 2020 ) or even remove connections before any training whatsoever ( Lee et al. , 2019 ; Wang et al. , 2020 ) . Compression rates usually considered in the pruning literature usually fall between 10× and 100× of the size of the original model . However , as contemporary model sizes grow into the billions of parameters , studying higher compression regimes becomes increasingly important . Recently , a new bold sparsity benchmark was set by Tanaka et al . ( 2020 ) with Iterative Synaptic Flow ( SynFlow ) , a data-agnostic algorithm for pruning at initialization . Reportedly , it is capable of removing all but only a few hundreds of parameters ( a 100 , 000× compression for VGG-16 ) and still produce trainable subnetworks , while other pruning methods disconnect networks at much lower sparsity levels ( Tanaka et al. , 2020 ) . Related work by de Jorge et al . ( 2021 ) proposes an iterative version of one-shot pruning algorithm , Single-shot Network Pruning ( SNIP ) ( Lee et al. , 2019 ) , and evaluates it in a similar high sparsity regime , reaching more than 10 , 000× compression ratio . Effective sparsity . This increased focus on extreme sparsity leads us to consider what sparsity is meant to represent in neural networks and computational graphs at large . In the context of neural network pruning , sparsity to date is computed straightforwardly as the fraction of removed connections ( direct sparsity ) —and compression as the inverse fraction of unpruned connections ( direct compression ) . We observe that this definition does not distinguish between connections that have actually been pruned , and those that have become effectively pruned because they have disconnected from the computational flow . In this work , we propose to instead focus on effective sparsity—the fraction of inactivated connections , be it through direct pruning or through otherwise disconnecting from either input or output of a network ( see Figure 1 for an illustration ) . In this work , we advocate that effective sparsity ( effective compression ) be used universally in place of its direct counterpart since it more accurately depicts what one would reasonably consider the network ’ s sparsity state . Using the lens of effective compression for benchmarking allows for a fairer comparison between different unstructured pruning algorithms . Note that effective compression is lower bounded by direct compression , which means that some pruning algorithms will give improved sparsity-accuracy trade-offs in this new framework . In Section 3 , we critically reexamine a plethora of recent pruning algorithms for a variety of architectures to find that , in this refined framework , conclusions drawn in previous works appear overstated or incorrect . Figure 2 gives a sneak-preview of this effect for three ab-initio pruning algorithms : SynFlow ( Tanaka et al. , 2020 ) , SNIP ( Lee et al. , 2019 ) and plain random pruning for LeNet-300-100 on MNIST . While SynFlow appears superior to other methods when evaluated against direct compression , it loses its advantage in the effective framework . Such radical performance changes are partly explained by differing gaps between effective and direct compression inherent to different pruning algorithms ( Figure 2 ) . We can see that significant departure between direct and effective compression kicks in at relatively low rates below 100× , making our work relevant even in these moderate regimes . For example , using random pruning to compress LeNet-300-100 by 100× ( sparsity 99 % ) results in ∼ 1 , 000× effective compression ; yet , removing the same number of parameters with SynFlow yields an unchanged 100× effective compression . What makes certain iterative algorithms like SynFlow less likely to amass dis- connected edges ? In Section 3 , we show that they are fortuitously designed to achieve a close convergence of direct and effective sparsity , hinting that preserving connectivity is an important aspect in the strong performance of high-compression pruning algorithms ( Tanaka et al. , 2020 ; de Jorge et al. , 2021 ) . Moreover , the lens of effective compression gives access to more extreme compression regimes for some pruning algorithms , which appear to disconnect much earlier when not accounting for inactive connections . For these high effective compression ratios all three pruning methods from Figure 2 perform surprisingly similar , even though they use varying degrees of information on data and parameter values . Layerwise Sparsity Quotas ( LSQ ) and Ideal Gas Quotas ( IGQ ) . A recent thread of research by Frankle et al . ( 2021 ) and Su et al . ( 2020 ) shows that performance of trained subnetworks produced by algorithms for pruning at initialization is robust to randomly reshuffling unpruned edges within layers before training . This observation led to the conjecture that these algorithms essentially generate successful distributions of sparsity across layers , while the exact connectivity patterns are unimportant . In Section 4 , we reexamine this conjecture through the lens of effective sparsity , confirm it for moderate compression regimes ( 10×–100× ) studied by Frankle et al . ( 2021 ) and Su et al . ( 2020 ) , but find the truth to be more nuanced at higher compression rates . Nonetheless , this re- sult highlights the importance of algorithms that carefully engineer layerwise sparsity quotas ( LSQ ) to obtain very simple and adequately performing pruning algorithms that are data- and parameteragnostic . Another important motivation to search for good LSQ is that global pruning algorithms frequently remove entire layers prematurely ( Lee et al. , 2020 ) ( cf . layer-collapse ( Tanaka et al. , 2020 ) ) , even before any significant differences between direct and effective sparsity emerge . Wellengineered LSQ could avoid this and enforce proper redistribution of compression across layers ( see ( Gale et al. , 2019 ; Mocanu et al. , 2018 ; Evci et al. , 2020 ) for existing baselines ) . In Section 4 , we propose a novel LSQ coined Ideal Gas Quotas ( IGQ ) by drawing intuitive analogies from physics . Effortlessly computable for any network-sparsity combination , IGQ performs similarly or better than any other baseline in the context of random pruning at initialization and of magnitude pruning after training . Effective pruning . Pruning to any desired direct sparsity is straightforward : one simply needs to mask out the corresponding number of parameters from a network . Effective sparsity , unfortunately , is more unpredictable and difficult to control . In particular , several known pruning algorithms suffer from layer-collapse once reaching a certain sparsity level , leading to unstable effective sparsity just before the disconnection . As a result , most pruning methods are unable to deliver certain values of effective sparsity regardless of how many connections are pruned . When possible , however , one needs to carefully tune the number of pruned parameters so that effective sparsity lands near a desired value . In Section 5 , we suggest a simple extension to algorithms for pruning at initialization or after training that helps bring effective sparsity close to any predefined achievable value while incurring costs that are at most logarithmic in model size . Our contributions . In this study , we ( i ) clearly formulate and illustrate the importance of effective sparsity by reevaluating several recent pruning strategies ; ( ii ) provide algorithms to prune according to and compute effective sparsity ; ( iii ) reconfirm the hypothesis from ( Frankle et al. , 2021 ) in the new framework and design efficient layerwise sparsity quotas IGQ that perform consistently well across all sparsity regimes . 2 RELATED WORK . Neural network compression encompasses a number of orthogonal approaches such as parameter regularization ( Lebedev & Lempitsky , 2016 ; Louizos et al. , 2018 ) , variational dropout ( Molchanov et al. , 2017 ) , vector quantization and parameter sharing ( Gong et al. , 2014 ; Chen et al. , 2015 ; Han et al. , 2016 ) , low-rank matrix decomposition ( Denton et al. , 2014 ; Jaderberg et al. , 2014 ) , and knowledge distillation ( Buciluǎ et al. , 2006 ; Hinton et al. , 2015 ) . Network pruning , however , is by far the most common technique for model compression , and can be partitioned into structured ( at the level of entire neurons/units ) and unstructured ( at the level of individual connections ) . While the former offers resource efficiency unconditioned on use of specialized hardware ( Liu et al. , 2019 ) and constitutes a fruitful research area ( Li et al. , 2017 ; Liu et al. , 2017 ) , we focus on the more actively studied unstructured pruning , which is where differences between effective and direct sparsity emerge . In what follows we give a quick overview , naturally grouping pruning methods by the time they are applied relative to training ( see ( Frankle & Carbin , 2019 ) and ( Wang et al. , 2020 ) for a similar taxonomy ) . Pruning after training . These earliest pruning techniques were designed to remove the least “ salient ” learned connections without sacrificing predictive performance . Optimal Brain Damage ( LeCun et al. , 1990 ) and its sequel Optimal Brain Surgeon ( Hassibi et al. , 1993 ) use the Hessian of the loss to estimate sensitivity to removal of individual parameters . Han et al . ( 2015 ) popularized magnitude as a simple and effective pruning criterion . It proved to be especially successful when applied alternately with several finetuning cycles , which is commonly referred to as Iterative Magnitude Pruning ( IMP ) , a modification of which was used by Frankle & Carbin ( 2019 ) to discover lottery tickets . Later , Dong et al . ( 2017 ) showed that magnitude-based pruning minimizes ` 2 distortion of each layer ’ s output incurred by parameter removal . Recently , Lee et al . ( 2021 ) extend this idea and propose Layer-Adaptive Magnitude-Based Pruning ( LAMP ) , which approximately minimizes the upper bound of the ` 2 distortion of the entire network . While equivalent to magnitude pruning within individual layers , LAMP automatically discovers state-of-the-art layerwise sparsity quotas ( see Section 4 ) that yield better performance ( as a function of direct compression ) than existing alternatives in the context of IMP . Pruning during training . Algorithms in this category learn sparsity structures together with parameter values , hoping that continued training will correct for damage incurred by pruning . To avoid inefficient prune-retrain cycles inherent to IMP , Narang et al . ( 2017 ) introduce gradual magnitude pruning over a single training round . Subsequently , Zhu & Gupta ( 2018 ) modify this algorithm by introducing a simpler pruning schedule and keeping layerwise sparsities uniform throughout training . Sparse Evolutionary Training ( SET ) ( Mocanu et al. , 2018 ) starts with an already sparse subnetwork and restructures it during training by pruning and randomly reviving connections . Unlike SET , Mostafa & Wang ( 2019 ) allow redistribution of sparsity across layers , while Dettmers & Zettlemoyer ( 2019 ) use gradient momentum as the criterion for parameter regrowth . Evci et al . ( 2020 ) rely on the instantaneous gradient to revive weights but follow SET to maintain the initial layerwise sparsity distribution during training . A different body of works tackle the general optimization problem with an intractable ` 0 parameter sparsity constraint by designing and solving related continuous problems ( Zhou et al. , 2021 ; Savarese et al. , 2020 ; Kusupati et al. , 2020 ) . For example , Continuous Sparsification ( CS ) by Savarese et al . ( 2020 ) uses a sigmoid of learnable continuous variables as mask values and applies ` 1 regularization , effectively forcing them to either 0 or 1 in training . Pruning before training . Pruning at initialization is especially alluring to deep learning practitioners as it promises lower costs of both optimization and inference . While this may seem too ambitious , the Lottery Ticket Hypothesis ( LTH ) postulates that randomly initialized dense networks do indeed contain highly trainable and equally well-performing sparse subnetworks ( Frankle & Carbin , 2019 ) . Inspired by the LTH , Lee et al . ( 2019 ) design SNIP , which uses connection sensitivity as a parameter saliency score . Wang et al . ( 2020 ) notice that SNIP creates bottlenecks or even removes entire layers and propose Gradient Signal Preservation ( GraSP ) as an alternative that aims to maximize gradient flow in a pruned network . de Jorge et al . ( 2021 ) improve SNIP by applying it iteratively , allowing for reassessment of saliency scores during pruning and helping networks stay connected at higher compression rates . A truly new compression benchmark was set by Tanaka et al . ( 2020 ) ; their algorithm , SynFlow , iteratively prunes subsets of parameters according to their ` 1 path norm and helps networks reach maximum compression without disconnecting . For example , SynFlow achieves non-random test accuracy on CIFAR-10 with a 100 , 000× compressed VGG-16 , while SNIP and GraSP fail already at 100× and 1 , 000× , respectively . An extensive ablation study by Frankle et al . ( 2021 ) examines SNIP , GraSP and SynFlow within moderate compression rates ( up to 100× ) and reveals that performance of subnetworks produced by these methods is stable under layerwise rearrange prior to training . Later , this result was independently confirmed by Su et al . ( 2020 ) for SNIP and GraSP only . This observation suggests that these algorithms perform as well as random pruning with corresponding layerwise quotas , putting the spotlight on designing competitive LSQ ( Mocanu et al. , 2018 ; Gale et al. , 2019 ; Lee et al. , 2021 ) .

Summary: The paper points out that in pruning, there is often a difference between the number of zero parameters and the number effective zero parameters due to disconnection during pruning. A thorough analysis is done comparing many pruning methods across MNIST, Cifar-10, and TinyImageNet using various architectures. The authors introduce a new physics inspired layerwise sparsity quota  baseline the "Ideal Gas quota" which sets a sparsity per layer.