Lottery Tickets can have Structural Sparsity

The lottery ticket hypothesis ( LTH ) has shown that dense models contain highly sparse subnetworks ( i.e. , winning tickets ) that can be trained in isolation to match full accuracy . Despite many exciting efforts being made , there is one “ commonsense ” seldomly challenged : a winning ticket is found by iterative magnitude pruning ( IMP ) and hence the resultant pruned subnetworks have only unstructured sparsity . That gap limits the appeal of winning tickets in practice , since the highly irregular sparse patterns are challenging to accelerate on hardware . Meanwhile , directly substituting structured pruning for unstructured pruning in IMP damages performance more severely and is usually unable to locate winning tickets . In this paper , we demonstrate the first positive result that a structurally sparse winning ticket can be effectively found in general . The core idea is to append “ post-processing techniques ” after each round of ( unstructured ) IMP , to enforce the formation of structural sparsity . Specifically , we first “ re-fill ” pruned elements back in some channels deemed to be important , and then “ re-group ” non-zero elements to create flexible group-wise structural patterns . Both our identified channel- and group-wise structural subnetworks win the lottery , with substantial inference speedups readily supported by practical hardware . Extensive experiments , conducted on diverse datasets across multiple network backbones , consistently validate our proposal , showing that the hardware acceleration roadblock of LTH is now removed . Specifically , the structural winning tickets obtain up to { 64.93 % , 64.84 % , 64.84 % } running time savings at { 36 % ∼ 80 % , 74 % , 58 % } sparsity on { CIFAR , Tiny-ImageNet , ImageNet } , while maintaining comparable accuracy . All the codes and pre-trained models will be publicly released . 1 INTRODUCTION . Recently , the machine learning research community has devoted considerable efforts and financial outlay to scaling deep neural networks ( DNNs ) to enormous sizes ( 175 billion parameter-counts in GPT-3 ( Brown et al. , 2020 ) ) . Although such overparameterization simplifies the training of DNNs and dramatically improves their generalization ( Bartlett et al. , 2021 ; Du et al. , 2018 ; Kaplan et al. , 2020 ) , it may severely obstruct the practical usage on resource-limited platforms like mobile devices , due to its large memory footprint and inference time ( Hoefler et al. , 2021 ) . Pruning as one of the effective remedies can be dated back to LeCun et al . ( 1990 ) : it can eliminate substantial redundant model parameters and boost the computational and storage efficiency of DNNs . Such benefits drive numerous interests in designing model pruning algorithms ( Han et al. , 2015a ; b ; Ren et al. , 2018 ; He et al. , 2017 ; Liu et al. , 2017 ) . Among this huge family , an emerging representative studies the prospect of training sparse subnetworks in lieu of the full dense models without impacting performance ( Frankle & Carbin , 2019 ; Chen et al. , 2020b ) . For instance , Frankle & Carbin ( 2019 ) demonstrates that dense models contain sparse , matching subnetworks ( Frankle et al. , 2020a ) ( a.k.a . winning tickets ) capable of training in isolation from the original initialization to match or even surpass the full accuracy . This phenomenon is referred to as the lottery tickets hypothesis ( LTH ) , which indicates several impressive observations : ( i ) usually extreme sparsity levels ( e.g. , 90 % , 95 % ) can be achieved without sacrificing the test accuracy ; ( ii ) the located winning ticket maintains undamaged expressive power as its dense counterpart , and can be easily trained from scratch or early-epoch weights ( Renda et al. , 2020 ; Frankle et al. , 2020a ) to recover the full performance . These advances are positive signs about the substantial potential of sparse DNNs . 1 However , almost all LTH literature investigates unstructured sparsity only . In practical scenarios , it brings little hardware efficiency benefits due to the poor data locality and low parallelism ( He et al. , 2017 ; Mao et al. , 2017 ; Wen et al. , 2016 ) caused by highly irregular sparse patterns . Meanwhile , most of the accelerators are optimized for dense matrix operations ( Han et al. , 2016 ) , which means there is limited speedup for unstructured pruned subnetworks even the sparsity level exceeds 95 % ( Wen et al. , 2016 ) . Structural pruning ( He et al. , 2017 ; Liu et al. , 2017 ) as an alternative to exploring sparse subnetworks , removes the entire filter or channel in DNNs to gain more computational efficiency at the cost of ( more ) accuracy degradation . As shown in Figure 1 , traditional channel-wise structural pruning approaches ( He et al. , 2017 ; Liu et al. , 2017 ; Bartoldson et al. , 2019 ; Molchanov et al. , 2019 ) quickly degrade performance and can not lead to winning tickets , which was also echoed in You et al . ( 2020 ) . In our paper , we present the first study into the structural lottery tickets , which explores hardwarefriendly structural sparsity ( including channel-wise and group-wise patterns ) in order to find lottery tickets . Specifically , we start from unstructured sparse subnetworks , and then adopt proposed refilling techniques to create channel-wise structural sparsity by growing back the pruned elements within the most important channels and abandoning the rest . Our results ( Section 4 ) show such refined channel-wise structural subnetworks win the lottery at a moderate sparsity level with∼ 50 % running time savings on an Nvidia 2080 TI . In order to push the compression ratio higher , we introduce a regrouping algorithm based on hypergraph partitioning ( Rumi et al. , 2020 ) to establish group-wise structural patterns which are more amenable to pruning due to the shape flexibility of grouped dense blocks . These group-wise structural winning tickets achieve ∼ 60 % running time savings at 50 % ∼ 80 % sparsity without any performance degradation compared to the dense models . Our main contributions lie in the following aspects : • To our best knowledge , we are the first to demonstrate the existence of structurally sparse winning tickets at non-trivial sparsity levels ( i.e. , > 30 % ) , and with both channel-wise and group-wise sparse patterns . • We propose the refilling technique and introduce the regrouping algorithm to form channelwise and group-wise structural sparsity , respectively . Such refined structural subnetworks match the trainability and expressiveness of dense networks , while enabling the inference speedup on practical hardware platforms like GPU machines . • Extensive experiments validate our proposal on diverse datasets ( i.e. , CIFAR-10/100 , TinyImageNet , and ImageNet ) across multiple network architectures , including ResNets , VGG , and MobileNet . Specifically , our structural winning tickets achieve 53.75 % ∼ 64.93 % GPU running time savings at 45 % ∼ 80 % channel- and group-wise sparsity . 2 RELATED WORK . Pruning . Network pruning is a technique that aims at eliminating the unnecessary model parameters ( Blalock et al. , 2020 ) , which can effectively shrink models for the deployment on resourceconstrained devices ( LeCun et al. , 1990 ; Hanson & Pratt , 1988 ) . Pruning algorithms are roughly categorized into two groups : ( 1 ) unstructured pruning ( LeCun et al. , 1990 ; Han et al. , 2015a ; b ; Ren et al. , 2018 ; Zhang et al. , 2018 ) with irregular sparse patterns ; ( 2 ) structural pruning ( He et al. , 2017 ; Liu et al. , 2017 ; Li et al. , 2016 ; Hu et al. , 2016 ; Wen et al. , 2016 ; Hong et al. , 2018 ) with structural sparse patterns such as channel-wise , block-wise , column-wise , etc .. Within the group of unstructured pruning , Han et al . ( 2015a ; b ) remove insignificant connections of models in the post-training stage , with respect to certain heuristics like weight/gradient magnitudes ; during training sparsification is also another popular trend for pruning by leveraging ` 0 regularization ( Louizos et al. , 2017 ) or alternating direction method of multipliers ( ADMM ) ( Ren et al. , 2 2018 ; Zhang et al. , 2018 ) . Recently , several pruning-at-initialization methods ( Wang et al. , 2020 ; Lee et al. , 2019b ; Tanaka et al. , 2020 ) are proposed to identify critical unstructured connections for gradient-flow preserving , without any training . Although the unstructured sparse model has superior performance , it usually suffers from poor data locality and low parallelism ( He et al. , 2017 ; Mao et al. , 2017 ; Wen et al. , 2016 ) , which make it hard to speed up in real-world applications . On the contrary , structural pruning is more hardware-friendly at the cost of notable accuracy loss when the compression ratio increases . He et al . ( 2017 ) ; Liu et al . ( 2017 ) slim the network channels via ` 1 regularization , and Bartoldson et al . ( 2019 ) selects important channels according to heuristics of feature maps . To combine the benefits of structural and unstructured pruning , hybrid pruning strategies have been introduced to pursue more general structural spares patterns which are also capable of acceleration . For example , convolution kernels with half regular sparsity ( Chen et al. , 2018 ) or pattern-based structural sparsity ( Ma et al. , 2020 ) or vector-wise ( Zhu et al. , 2019 ) and group-wise ( Rumi et al. , 2020 ) regular sparsity . The lottery tickets hypothesis ( LTH ) . The lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) conjectures that there exists a sparse subnetwork called winning ticket within a dense network , whose performance can match with the dense network when training from the same initialization . With the assistance of weight rewinding techniques ( Renda et al. , 2020 ; Frankle et al. , 2020a ) , the original LTH can be scaled up to larger networks and datasets . The existence of winning tickets are broadly verified under diverse contexts , such as image classification ( Frankle & Carbin , 2019 ; Liu et al. , 2019 ; Wang et al. , 2020 ; Evci et al. , 2019 ; Frankle et al. , 2020b ; Savarese et al. , 2020 ; You et al. , 2020 ; Ma et al. , 2021a ; Chen et al. , 2020a ) , object detection Girish et al . ( 2020 ) , natural language processing Gale et al . ( 2019 ) ; Yu et al . ( 2020 ) ; Prasanna et al . ( 2020 ) ; Chen et al . ( 2020b ; c ) , generative adversarial networks Chen et al . ( 2021d ) ; Kalibhat et al . ( 2020 ) ; Chen et al . ( 2021a ) , graph neural networks Chen et al . ( 2021b ) , reinforcement learning Yu et al . ( 2020 ) , and life-long learning Chen et al . ( 2021c ) . However , all of the above LTH literature only locate unstructured sparse winning tickets , which can hardly bring hardware efficiency boost on real-world applications . As the most related work , You et al . ( 2020 ) finds structural winning tickets at only low sparsity levels around 30 % in few cases . It again reveals the complication and difficulty of identifying computation-friendly sparse patterns . Another concurrent work ( Alabdulmohsin et al. , 2021 ) investigates a generalized LTH with weight space factorization , which is orthogonal to our work . Sparse convolutional neural network ( CNN ) acceleration on GPU . Previous works have explored the acceleration of sparse convolution operations in two different directions . One direction is to design efficient implementation of unstructured pruned networks for improved data locality and utilization of hardware ( Chen , 2018 ; Park et al. , 2016 ) . For example , Dong et al . ( 2019 ) proposes “ Acorns ” to accelerate the sparse computations of convolution kernels with an input sparsity . Peng et al . ( 2017 ) has proposed a matrix splitting algorithm for efficient CNN inference . Nvidia ’ s cuSPARSE1 library contains various efficient sparse matrix computation algorithms like SpMM on GPUs , drawing great attention in efficient scientific computing . Furthermore , advanced approaches are developed based on SpMM , such as Adaptive Sparse Tiling ( ASpT ) ( Hong et al. , 2019 ) . ASpT significantly improves the data usage of SpMM and achieves the current state-of-the-art performance among SpMM implementation variants . Another direction focuses on more hardware-friendly pruning methods ( Chen et al. , 2018 ; Ma et al. , 2020 ; Niu et al. , 2020 ) . During the model pruning , these works aim to maintain certain regular sparse patterns , which benefit the hardware processing/computing of corresponding sparse matrices . However , Chen et al . ( 2018 ) achieves unsatisfactory compression ratio , while the pruning methods used in Ma et al . ( 2020 ) and Niu et al . ( 2020 ) require dedicated compiler optimization to accelerate network execution .

Summary: LTH (lottery ticket hypothesis) was mainly proposed for unstructured pruning. This paper shows it can also be validated on structured pruning, for the first time. The key for them to achieve so is the newly proposed post-processing techniques, refilling(+) and regrouping. They show by these techniques, structural winning tickets can be found, with up to 6.67x on hardware platforms.