Lottery Tickets can have Structural Sparsity

The lottery ticket hypothesis ( LTH ) has shown that dense models contain highly sparse subnetworks ( i.e. , winning tickets ) that can be trained in isolation to match full accuracy . Despite many exciting efforts being made , there is one “ commonsense ” seldomly challenged : a winning ticket is found by iterative magnitude pruning ( IMP ) and hence the resultant pruned subnetworks have only unstructured sparsity . That gap limits the appeal of winning tickets in practice , since the highly irregular sparse patterns are challenging to accelerate on hardware . Meanwhile , directly substituting structured pruning for unstructured pruning in IMP damages performance more severely and is usually unable to locate winning tickets . In this paper , we demonstrate the first positive result that a structurally sparse winning ticket can be effectively found in general . The core idea is to append “ post-processing techniques ” after each round of ( unstructured ) IMP , to enforce the formation of structural sparsity . Specifically , we first “ re-fill ” pruned elements back in some channels deemed to be important , and then “ re-group ” non-zero elements to create flexible group-wise structural patterns . Both our identified channel- and group-wise structural subnetworks win the lottery , with substantial inference speedups readily supported by practical hardware . Extensive experiments , conducted on diverse datasets across multiple network backbones , consistently validate our proposal , showing that the hardware acceleration roadblock of LTH is now removed . Specifically , the structural winning tickets obtain up to { 64.93 % , 64.84 % , 64.84 % } running time savings at { 36 % ∼ 80 % , 74 % , 58 % } sparsity on { CIFAR , Tiny-ImageNet , ImageNet } , while maintaining comparable accuracy . All the codes and pre-trained models will be publicly released . 1 INTRODUCTION . Recently , the machine learning research community has devoted considerable efforts and financial outlay to scaling deep neural networks ( DNNs ) to enormous sizes ( 175 billion parameter-counts in GPT-3 ( Brown et al. , 2020 ) ) . Although such overparameterization simplifies the training of DNNs and dramatically improves their generalization ( Bartlett et al. , 2021 ; Du et al. , 2018 ; Kaplan et al. , 2020 ) , it may severely obstruct the practical usage on resource-limited platforms like mobile devices , due to its large memory footprint and inference time ( Hoefler et al. , 2021 ) . Pruning as one of the effective remedies can be dated back to LeCun et al . ( 1990 ) : it can eliminate substantial redundant model parameters and boost the computational and storage efficiency of DNNs . Such benefits drive numerous interests in designing model pruning algorithms ( Han et al. , 2015a ; b ; Ren et al. , 2018 ; He et al. , 2017 ; Liu et al. , 2017 ) . Among this huge family , an emerging representative studies the prospect of training sparse subnetworks in lieu of the full dense models without impacting performance ( Frankle & Carbin , 2019 ; Chen et al. , 2020b ) . For instance , Frankle & Carbin ( 2019 ) demonstrates that dense models contain sparse , matching subnetworks ( Frankle et al. , 2020a ) ( a.k.a . winning tickets ) capable of training in isolation from the original initialization to match or even surpass the full accuracy . This phenomenon is referred to as the lottery tickets hypothesis ( LTH ) , which indicates several impressive observations : ( i ) usually extreme sparsity levels ( e.g. , 90 % , 95 % ) can be achieved without sacrificing the test accuracy ; ( ii ) the located winning ticket maintains undamaged expressive power as its dense counterpart , and can be easily trained from scratch or early-epoch weights ( Renda et al. , 2020 ; Frankle et al. , 2020a ) to recover the full performance . These advances are positive signs about the substantial potential of sparse DNNs . 1 However , almost all LTH literature investigates unstructured sparsity only . In practical scenarios , it brings little hardware efficiency benefits due to the poor data locality and low parallelism ( He et al. , 2017 ; Mao et al. , 2017 ; Wen et al. , 2016 ) caused by highly irregular sparse patterns . Meanwhile , most of the accelerators are optimized for dense matrix operations ( Han et al. , 2016 ) , which means there is limited speedup for unstructured pruned subnetworks even the sparsity level exceeds 95 % ( Wen et al. , 2016 ) . Structural pruning ( He et al. , 2017 ; Liu et al. , 2017 ) as an alternative to exploring sparse subnetworks , removes the entire filter or channel in DNNs to gain more computational efficiency at the cost of ( more ) accuracy degradation . As shown in Figure 1 , traditional channel-wise structural pruning approaches ( He et al. , 2017 ; Liu et al. , 2017 ; Bartoldson et al. , 2019 ; Molchanov et al. , 2019 ) quickly degrade performance and can not lead to winning tickets , which was also echoed in You et al . ( 2020 ) . In our paper , we present the first study into the structural lottery tickets , which explores hardwarefriendly structural sparsity ( including channel-wise and group-wise patterns ) in order to find lottery tickets . Specifically , we start from unstructured sparse subnetworks , and then adopt proposed refilling techniques to create channel-wise structural sparsity by growing back the pruned elements within the most important channels and abandoning the rest . Our results ( Section 4 ) show such refined channel-wise structural subnetworks win the lottery at a moderate sparsity level with∼ 50 % running time savings on an Nvidia 2080 TI . In order to push the compression ratio higher , we introduce a regrouping algorithm based on hypergraph partitioning ( Rumi et al. , 2020 ) to establish group-wise structural patterns which are more amenable to pruning due to the shape flexibility of grouped dense blocks . These group-wise structural winning tickets achieve ∼ 60 % running time savings at 50 % ∼ 80 % sparsity without any performance degradation compared to the dense models . Our main contributions lie in the following aspects : • To our best knowledge , we are the first to demonstrate the existence of structurally sparse winning tickets at non-trivial sparsity levels ( i.e. , > 30 % ) , and with both channel-wise and group-wise sparse patterns . • We propose the refilling technique and introduce the regrouping algorithm to form channelwise and group-wise structural sparsity , respectively . Such refined structural subnetworks match the trainability and expressiveness of dense networks , while enabling the inference speedup on practical hardware platforms like GPU machines . • Extensive experiments validate our proposal on diverse datasets ( i.e. , CIFAR-10/100 , TinyImageNet , and ImageNet ) across multiple network architectures , including ResNets , VGG , and MobileNet . Specifically , our structural winning tickets achieve 53.75 % ∼ 64.93 % GPU running time savings at 45 % ∼ 80 % channel- and group-wise sparsity . 2 RELATED WORK . Pruning . Network pruning is a technique that aims at eliminating the unnecessary model parameters ( Blalock et al. , 2020 ) , which can effectively shrink models for the deployment on resourceconstrained devices ( LeCun et al. , 1990 ; Hanson & Pratt , 1988 ) . Pruning algorithms are roughly categorized into two groups : ( 1 ) unstructured pruning ( LeCun et al. , 1990 ; Han et al. , 2015a ; b ; Ren et al. , 2018 ; Zhang et al. , 2018 ) with irregular sparse patterns ; ( 2 ) structural pruning ( He et al. , 2017 ; Liu et al. , 2017 ; Li et al. , 2016 ; Hu et al. , 2016 ; Wen et al. , 2016 ; Hong et al. , 2018 ) with structural sparse patterns such as channel-wise , block-wise , column-wise , etc .. Within the group of unstructured pruning , Han et al . ( 2015a ; b ) remove insignificant connections of models in the post-training stage , with respect to certain heuristics like weight/gradient magnitudes ; during training sparsification is also another popular trend for pruning by leveraging ` 0 regularization ( Louizos et al. , 2017 ) or alternating direction method of multipliers ( ADMM ) ( Ren et al. , 2 2018 ; Zhang et al. , 2018 ) . Recently , several pruning-at-initialization methods ( Wang et al. , 2020 ; Lee et al. , 2019b ; Tanaka et al. , 2020 ) are proposed to identify critical unstructured connections for gradient-flow preserving , without any training . Although the unstructured sparse model has superior performance , it usually suffers from poor data locality and low parallelism ( He et al. , 2017 ; Mao et al. , 2017 ; Wen et al. , 2016 ) , which make it hard to speed up in real-world applications . On the contrary , structural pruning is more hardware-friendly at the cost of notable accuracy loss when the compression ratio increases . He et al . ( 2017 ) ; Liu et al . ( 2017 ) slim the network channels via ` 1 regularization , and Bartoldson et al . ( 2019 ) selects important channels according to heuristics of feature maps . To combine the benefits of structural and unstructured pruning , hybrid pruning strategies have been introduced to pursue more general structural spares patterns which are also capable of acceleration . For example , convolution kernels with half regular sparsity ( Chen et al. , 2018 ) or pattern-based structural sparsity ( Ma et al. , 2020 ) or vector-wise ( Zhu et al. , 2019 ) and group-wise ( Rumi et al. , 2020 ) regular sparsity . The lottery tickets hypothesis ( LTH ) . The lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) conjectures that there exists a sparse subnetwork called winning ticket within a dense network , whose performance can match with the dense network when training from the same initialization . With the assistance of weight rewinding techniques ( Renda et al. , 2020 ; Frankle et al. , 2020a ) , the original LTH can be scaled up to larger networks and datasets . The existence of winning tickets are broadly verified under diverse contexts , such as image classification ( Frankle & Carbin , 2019 ; Liu et al. , 2019 ; Wang et al. , 2020 ; Evci et al. , 2019 ; Frankle et al. , 2020b ; Savarese et al. , 2020 ; You et al. , 2020 ; Ma et al. , 2021a ; Chen et al. , 2020a ) , object detection Girish et al . ( 2020 ) , natural language processing Gale et al . ( 2019 ) ; Yu et al . ( 2020 ) ; Prasanna et al . ( 2020 ) ; Chen et al . ( 2020b ; c ) , generative adversarial networks Chen et al . ( 2021d ) ; Kalibhat et al . ( 2020 ) ; Chen et al . ( 2021a ) , graph neural networks Chen et al . ( 2021b ) , reinforcement learning Yu et al . ( 2020 ) , and life-long learning Chen et al . ( 2021c ) . However , all of the above LTH literature only locate unstructured sparse winning tickets , which can hardly bring hardware efficiency boost on real-world applications . As the most related work , You et al . ( 2020 ) finds structural winning tickets at only low sparsity levels around 30 % in few cases . It again reveals the complication and difficulty of identifying computation-friendly sparse patterns . Another concurrent work ( Alabdulmohsin et al. , 2021 ) investigates a generalized LTH with weight space factorization , which is orthogonal to our work . Sparse convolutional neural network ( CNN ) acceleration on GPU . Previous works have explored the acceleration of sparse convolution operations in two different directions . One direction is to design efficient implementation of unstructured pruned networks for improved data locality and utilization of hardware ( Chen , 2018 ; Park et al. , 2016 ) . For example , Dong et al . ( 2019 ) proposes “ Acorns ” to accelerate the sparse computations of convolution kernels with an input sparsity . Peng et al . ( 2017 ) has proposed a matrix splitting algorithm for efficient CNN inference . Nvidia ’ s cuSPARSE1 library contains various efficient sparse matrix computation algorithms like SpMM on GPUs , drawing great attention in efficient scientific computing . Furthermore , advanced approaches are developed based on SpMM , such as Adaptive Sparse Tiling ( ASpT ) ( Hong et al. , 2019 ) . ASpT significantly improves the data usage of SpMM and achieves the current state-of-the-art performance among SpMM implementation variants . Another direction focuses on more hardware-friendly pruning methods ( Chen et al. , 2018 ; Ma et al. , 2020 ; Niu et al. , 2020 ) . During the model pruning , these works aim to maintain certain regular sparse patterns , which benefit the hardware processing/computing of corresponding sparse matrices . However , Chen et al . ( 2018 ) achieves unsatisfactory compression ratio , while the pruning methods used in Ma et al . ( 2020 ) and Niu et al . ( 2020 ) require dedicated compiler optimization to accelerate network execution .

Summary: This work proposes a method to effectively find structurally sparse winning tickets. It consists of some post-processing techniques that can be added to each round of standard iterative magnitude pruning (IMP) methods. Starting from unstructured sparse sub-networks, the method uses a "re-filling and re-grouping" manner to enforce the formation of structural sparsity.