Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions

1 INTRODUCTION . The applications of convolutional neural networks ( CNNs ) have demonstrated proven success in various computer vision tasks ( Voulodimos et al. , 2018 ) . However , with modern CNN architectures being increasingly deeper and wider , over-parameterization has become one of the major challenges of deploying such models to devices with limited computational resources and memory capacity ( Frankle & Carbin , 2019 ) . Therefore , the study of network pruning—the technique of removing redundant weights from the originally trained network without significantly sacrificing accuracy—has been an important subject both for practical concerns ( Mozer & Smolensky , 1989 ) and to better understand the properties and mechanisms of neural networks ( Arora et al. , 2018 ) . In the realm of CNN pruning , a spectrum of techniques have been studied where the two ends are populated by structured pruning and unstructured pruning methods ( Mao et al. , 2017 ) . Methods from the former end often propose to remove redundant weights in groups while following some geometrical constraints—such as removing a certain filter or layer . The methods from the latter end , on the other hand , prune the network with a more fine-grained view where they evaluate every weight individually . Yet , there are many other methods lay in between of the two ends . Methods that are more “ unstructured ” are believed to be capable of yielding better accuracy retention with a commensurate amount of parameters pruned , due to having a higher degree of freedom on how and where to introduce sparsity to the originally dense network . Empirical findings also support this claim ( Liu et al. , 2019 ; Mao et al. , 2017 ) . Despite having advantages on accuracy retention , the resultant networks from methods closer to the unstructured end will be more sparse and less regulated on where to introduce sparsity . Which may not provide actual compression and acceleration without relying on custom-indexing , sparse convolution libraries , or even dedicated hardware devices ; thus , limits the deployability of such methods ( Mao et al. , 2017 ) . Meanwhile , methods closer to the structured end , by preserving a more regulated resultant network , are more likely to be library/hardware-friendly . The most deployable structured pruning method may deliver a densely resultant network and therefore gain immediate compression and acceleration benefits . We denote this kind of pruning method as densely structured . Naturally , many scholars want to develop new methods within the realm of densely structured pruning but with better accuracy retention . The majority of densely structured pruning methods focus on pruning the original network at a filter- or layer-level . We argue that the blanket removal of an entire filter or layer may harm the representation power of the network and result in undesired accuracy loss — as removing a filter would consequently remove all feature maps generated by such filter . Even worse , removing a layer would eliminate more feature maps and even face the danger of layer-collapse , a phenomenon of having an untrainable pruned network due to premature pruning of an entire layer ( Tanaka et al. , 2020 ) . In this paper , we revisit the idea of kernel pruning ( to only prune one or several k × k kernels from a 3D-filter , instead of an entire one ) as an alternative and less aggressive pruning approach . We hypothesize that by not removing the entire filter , the representation power of the original network will be better preserved . Although the idea of kernel pruning is nothing too novel ( as it is simply a special case of individual weights pruning with 100 % of weights of a kernel pruned ) , it is mostly applied under the context of unstructured pruning or structured pruning methods which may not deliver a dense pruned network ( Mao et al. , 2017 ; Ma et al. , 2020 ) . This is because a direct implementation of kernel pruning with no constraint would introduce sparsity across the network and therefore making the pruned network no longer dense . We address this problem by proposing a versatile grouped pruning framework , where we : 1 . Cluster similar filters from each convolutional layer into a ( predefined ) number of equal sized filter groups . 2 . For each filter group , identify a certain portion of grouped kernels to prune according to the required pruning ratio . 3 . Permute the remaining filters to form a densely grouped convolutional architecture according to the number of groups used in step 1 . Like most other post-train grouped pruning methods , we face the challenge of determining which clustering schemes and which importance metrics to use in step 1 and 2 of the above procedure . Upon investigations and experiments , we discovered that a classical clustering scoring system ( e.g. , the Silhouette score ) might not capture the better clustering scheme in regard to accuracy retention . Yet many filter importance metrics require sophisticated procedures , which are not computationally friendly or easy to execute when applied at a kernel-level . We address the first challenge by consulting model-generated information—in this case , the empirical findings on Lottery Ticket Hypothesis and related literature on weights shifting—to develop a scoring system that identifies the optimal clustering scheme among options per each convolutional layer ( Frankle & Carbin , 2019 ; Renda et al. , 2020 ) . For the second challenge , we design a simple and cost-efficient greedy algorithm with multiple restarts to generate multiple candidate kernel selection queues and identify the one queue where the preserved kernels are most “ distinctive ” from each other yet “ similar ” to the pruned kernels . The main contributions and advantages of our method are : • We brought attention to the heavily overlooked approach of kernel pruning under the context of densely structured pruning . • We developed a framework that is compatible with further-developed/discovered clustering schemes and inductive biases , or more advanced variations upon them . This overcomes one of the major drawbacks of many filter pruning methods : as most of them propose different filter importance metrics that are largely incompatible with each other either in terms of their procedures or computational requirements . • The resultant network of our method is structured as a densely grouped convolution , which enables parallel computing capability and greatly increases the practical deployability of our methods : as we can now share the required computation and memory footprint across multiple end-user devices , where most of them have very limited said resources individually ( e.g. , IoT devices , mobile phones , and wearable technologies ) . • Even by just applying well-understood classical mathematical tools , extensive experiments demonstrate our method outperforms comparable SOTA methods across different networks and datasets . Additionally , our method often needs less data augmentation , a smaller finetuning budget , and it executes without requiring any custom retraining , special fine-tuning , or iterative prune-train cycles — which is rare for the approaches relying on Lottery Ticket Hypothesis-related studies . 2 RELATED WORK . Many prior arts have explored the possibility of obtaining a smaller model with comparable performance by removing redundant weights ( Zhu & Gupta , 2018 ; Han et al. , 2016 ) , filters ( Molchanov et al. , 2017 ; Yu et al. , 2018 ; He et al. , 2019 ; Wang et al. , 2019a ) , layers ( Wang et al. , 2019b ; Lin et al. , 2019 ) , image input ( Howard et al. , 2017 ; Han et al. , 2020 ) , or from all three dimensions ( Wang et al. , 2021 ) . It is clear that filter pruning attracts the most attention among all structured pruning approaches . Our method is inspired by grouped convolution , a widely adopted convolutional architecture which could be implemented efficiently on common devices ( Iandola et al. , 2017 ) . Although kernel pruning used together with filter clustering is not a popular trend , we have seen such a combination in Yu et al . ( 2017 ) . However , the proposed method by Yu requires iterative analysis of many different intermediate feature maps per layer , involves a complex knowledge distillation application during the fine-tuning stage , and lacks comparable experiment results to recent pruning literature . Where our method delivers beyond-SOTA performance with a straightforward one-shot pruning and standard fine-tuning procedure . In addition , our method consults empirical findings on the lottery ticket hypothesis and its derived literature regarding weights masking and weights shifting ( Frankle & Carbin , 2019 ; Renda et al. , 2020 ; Zhou et al. , 2019 ) . More on this in the Proposed Method section . 3 PROPOSED METHOD . 3.1 PRELIMINARIES . Assume a convolutional neural network W has L convolutional layers , we denote the W ` to be the ` -th convolutional layer of W ( for ` ∈ { Z+ | [ 1 , L ] } ) . Therefore , we shall have a 4-D tensor W ` ∈ RC ` out×C ` in×H ` ×W ` where C ` out represents the number of filters in W ` ( also known as the number of output channels in some literature ) , C ` in represents the number of kernels per filter ( a.k.a . number of input channels ) , and H ` ×W ` represents the size of each kernel . The overall procedure of our method can be mainly divided into four stages : 1 ) Clustering filters into n equal-sized groups , where the best clustering scheme for each convolutional layer is determined using the tickets magnitude increase score derived from prior arts on lottery ticket hypothesis and weight-shifting ; 2 ) Evaluating several candidate grouped kernel pruning strategies generated by a greedy approximation algorithm with multiple restarts , where the strategy with preserved grouped kernels that are most distinctive from each other , yet most similar to the pruned grouped kernels gets selected ; 3 ) Permuting the preserved filters to form a grouped convolutional architecture with n groups ; 4 ) Fine-tuning the pruned and grouped network to recover accuracy lost from pruning . 3.2 CLUSTERING FILTERS INTO GROUPS . The first step of our method is to cluster filters from the same convolutional layer into n equal-sized groups . Known that for a layer W ` we have C ` out filters , there shall be C ` out/n filters inside each equal-sized filter group . We denotes F ` i to be the i-th filter in W ` ( namely , F ` i = W ` [ i , : , : , : ] ) for i ∈ { Z+ | [ 1 , C ` out ] } . Filter clustering is a maturely adopted technique in network pruning since it is a widely accepted assumption that when similar filters are clustered together , the representation power of some filters can be covered by the rest of the filters in the same group ( which therefore enables the potential of pruning ) . Additionally , this technique drastically decomposes the scope of the problem , as we may now proceed to evaluate in a group-by-group fashion instead of evaluating all of the filters from W ` at the same time . Many prior arts have developed methods on filter clustering with linearity assumptions ( Guo et al. , 2020 ) , via retraining with a custom loss function ( Wu et al. , 2018 ) , or through an iterative process ( Yu et al. , 2017 ) . We argue that since each filter F ` i is a tensor of C ` in ×H ` ×W ` , considerations regarding non-linearity and high-dimensional relationships should be added . Therefore , we utilize the following three combinations of proven mathematical tools on dimensionality reduction and clustering in order to cluster filters from each layer into n equal-sized groups .

Summary: The authors present a new metric to determine the similarity between different grouped kernels and prune the unimportant $k\times k$ slices out of a 3D filter. They utilize the Lottery Ticket Hypothesis and propose a greedy search strategy to overcome the challenge of a huge search space. The experiment results show that the one-shot scheme can still be comparable to two-stage leading methods on the CIFAR-10 dataset, with a slightly lower training cost. The empirical success of this paper may serve as proof of the existence of the Lottery Ticket Hypothesis.