Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions

1 INTRODUCTION . The applications of convolutional neural networks ( CNNs ) have demonstrated proven success in various computer vision tasks ( Voulodimos et al. , 2018 ) . However , with modern CNN architectures being increasingly deeper and wider , over-parameterization has become one of the major challenges of deploying such models to devices with limited computational resources and memory capacity ( Frankle & Carbin , 2019 ) . Therefore , the study of network pruning—the technique of removing redundant weights from the originally trained network without significantly sacrificing accuracy—has been an important subject both for practical concerns ( Mozer & Smolensky , 1989 ) and to better understand the properties and mechanisms of neural networks ( Arora et al. , 2018 ) . In the realm of CNN pruning , a spectrum of techniques have been studied where the two ends are populated by structured pruning and unstructured pruning methods ( Mao et al. , 2017 ) . Methods from the former end often propose to remove redundant weights in groups while following some geometrical constraints—such as removing a certain filter or layer . The methods from the latter end , on the other hand , prune the network with a more fine-grained view where they evaluate every weight individually . Yet , there are many other methods lay in between of the two ends . Methods that are more “ unstructured ” are believed to be capable of yielding better accuracy retention with a commensurate amount of parameters pruned , due to having a higher degree of freedom on how and where to introduce sparsity to the originally dense network . Empirical findings also support this claim ( Liu et al. , 2019 ; Mao et al. , 2017 ) . Despite having advantages on accuracy retention , the resultant networks from methods closer to the unstructured end will be more sparse and less regulated on where to introduce sparsity . Which may not provide actual compression and acceleration without relying on custom-indexing , sparse convolution libraries , or even dedicated hardware devices ; thus , limits the deployability of such methods ( Mao et al. , 2017 ) . Meanwhile , methods closer to the structured end , by preserving a more regulated resultant network , are more likely to be library/hardware-friendly . The most deployable structured pruning method may deliver a densely resultant network and therefore gain immediate compression and acceleration benefits . We denote this kind of pruning method as densely structured . Naturally , many scholars want to develop new methods within the realm of densely structured pruning but with better accuracy retention . The majority of densely structured pruning methods focus on pruning the original network at a filter- or layer-level . We argue that the blanket removal of an entire filter or layer may harm the representation power of the network and result in undesired accuracy loss — as removing a filter would consequently remove all feature maps generated by such filter . Even worse , removing a layer would eliminate more feature maps and even face the danger of layer-collapse , a phenomenon of having an untrainable pruned network due to premature pruning of an entire layer ( Tanaka et al. , 2020 ) . In this paper , we revisit the idea of kernel pruning ( to only prune one or several k × k kernels from a 3D-filter , instead of an entire one ) as an alternative and less aggressive pruning approach . We hypothesize that by not removing the entire filter , the representation power of the original network will be better preserved . Although the idea of kernel pruning is nothing too novel ( as it is simply a special case of individual weights pruning with 100 % of weights of a kernel pruned ) , it is mostly applied under the context of unstructured pruning or structured pruning methods which may not deliver a dense pruned network ( Mao et al. , 2017 ; Ma et al. , 2020 ) . This is because a direct implementation of kernel pruning with no constraint would introduce sparsity across the network and therefore making the pruned network no longer dense . We address this problem by proposing a versatile grouped pruning framework , where we : 1 . Cluster similar filters from each convolutional layer into a ( predefined ) number of equal sized filter groups . 2 . For each filter group , identify a certain portion of grouped kernels to prune according to the required pruning ratio . 3 . Permute the remaining filters to form a densely grouped convolutional architecture according to the number of groups used in step 1 . Like most other post-train grouped pruning methods , we face the challenge of determining which clustering schemes and which importance metrics to use in step 1 and 2 of the above procedure . Upon investigations and experiments , we discovered that a classical clustering scoring system ( e.g. , the Silhouette score ) might not capture the better clustering scheme in regard to accuracy retention . Yet many filter importance metrics require sophisticated procedures , which are not computationally friendly or easy to execute when applied at a kernel-level . We address the first challenge by consulting model-generated information—in this case , the empirical findings on Lottery Ticket Hypothesis and related literature on weights shifting—to develop a scoring system that identifies the optimal clustering scheme among options per each convolutional layer ( Frankle & Carbin , 2019 ; Renda et al. , 2020 ) . For the second challenge , we design a simple and cost-efficient greedy algorithm with multiple restarts to generate multiple candidate kernel selection queues and identify the one queue where the preserved kernels are most “ distinctive ” from each other yet “ similar ” to the pruned kernels . The main contributions and advantages of our method are : • We brought attention to the heavily overlooked approach of kernel pruning under the context of densely structured pruning . • We developed a framework that is compatible with further-developed/discovered clustering schemes and inductive biases , or more advanced variations upon them . This overcomes one of the major drawbacks of many filter pruning methods : as most of them propose different filter importance metrics that are largely incompatible with each other either in terms of their procedures or computational requirements . • The resultant network of our method is structured as a densely grouped convolution , which enables parallel computing capability and greatly increases the practical deployability of our methods : as we can now share the required computation and memory footprint across multiple end-user devices , where most of them have very limited said resources individually ( e.g. , IoT devices , mobile phones , and wearable technologies ) . • Even by just applying well-understood classical mathematical tools , extensive experiments demonstrate our method outperforms comparable SOTA methods across different networks and datasets . Additionally , our method often needs less data augmentation , a smaller finetuning budget , and it executes without requiring any custom retraining , special fine-tuning , or iterative prune-train cycles — which is rare for the approaches relying on Lottery Ticket Hypothesis-related studies . 2 RELATED WORK . Many prior arts have explored the possibility of obtaining a smaller model with comparable performance by removing redundant weights ( Zhu & Gupta , 2018 ; Han et al. , 2016 ) , filters ( Molchanov et al. , 2017 ; Yu et al. , 2018 ; He et al. , 2019 ; Wang et al. , 2019a ) , layers ( Wang et al. , 2019b ; Lin et al. , 2019 ) , image input ( Howard et al. , 2017 ; Han et al. , 2020 ) , or from all three dimensions ( Wang et al. , 2021 ) . It is clear that filter pruning attracts the most attention among all structured pruning approaches . Our method is inspired by grouped convolution , a widely adopted convolutional architecture which could be implemented efficiently on common devices ( Iandola et al. , 2017 ) . Although kernel pruning used together with filter clustering is not a popular trend , we have seen such a combination in Yu et al . ( 2017 ) . However , the proposed method by Yu requires iterative analysis of many different intermediate feature maps per layer , involves a complex knowledge distillation application during the fine-tuning stage , and lacks comparable experiment results to recent pruning literature . Where our method delivers beyond-SOTA performance with a straightforward one-shot pruning and standard fine-tuning procedure . In addition , our method consults empirical findings on the lottery ticket hypothesis and its derived literature regarding weights masking and weights shifting ( Frankle & Carbin , 2019 ; Renda et al. , 2020 ; Zhou et al. , 2019 ) . More on this in the Proposed Method section . 3 PROPOSED METHOD . 3.1 PRELIMINARIES . Assume a convolutional neural network W has L convolutional layers , we denote the W ` to be the ` -th convolutional layer of W ( for ` ∈ { Z+ | [ 1 , L ] } ) . Therefore , we shall have a 4-D tensor W ` ∈ RC ` out×C ` in×H ` ×W ` where C ` out represents the number of filters in W ` ( also known as the number of output channels in some literature ) , C ` in represents the number of kernels per filter ( a.k.a . number of input channels ) , and H ` ×W ` represents the size of each kernel . The overall procedure of our method can be mainly divided into four stages : 1 ) Clustering filters into n equal-sized groups , where the best clustering scheme for each convolutional layer is determined using the tickets magnitude increase score derived from prior arts on lottery ticket hypothesis and weight-shifting ; 2 ) Evaluating several candidate grouped kernel pruning strategies generated by a greedy approximation algorithm with multiple restarts , where the strategy with preserved grouped kernels that are most distinctive from each other , yet most similar to the pruned grouped kernels gets selected ; 3 ) Permuting the preserved filters to form a grouped convolutional architecture with n groups ; 4 ) Fine-tuning the pruned and grouped network to recover accuracy lost from pruning . 3.2 CLUSTERING FILTERS INTO GROUPS . The first step of our method is to cluster filters from the same convolutional layer into n equal-sized groups . Known that for a layer W ` we have C ` out filters , there shall be C ` out/n filters inside each equal-sized filter group . We denotes F ` i to be the i-th filter in W ` ( namely , F ` i = W ` [ i , : , : , : ] ) for i ∈ { Z+ | [ 1 , C ` out ] } . Filter clustering is a maturely adopted technique in network pruning since it is a widely accepted assumption that when similar filters are clustered together , the representation power of some filters can be covered by the rest of the filters in the same group ( which therefore enables the potential of pruning ) . Additionally , this technique drastically decomposes the scope of the problem , as we may now proceed to evaluate in a group-by-group fashion instead of evaluating all of the filters from W ` at the same time . Many prior arts have developed methods on filter clustering with linearity assumptions ( Guo et al. , 2020 ) , via retraining with a custom loss function ( Wu et al. , 2018 ) , or through an iterative process ( Yu et al. , 2017 ) . We argue that since each filter F ` i is a tensor of C ` in ×H ` ×W ` , considerations regarding non-linearity and high-dimensional relationships should be added . Therefore , we utilize the following three combinations of proven mathematical tools on dimensionality reduction and clustering in order to cluster filters from each layer into n equal-sized groups .

Summary: The proposed work focuses on kernel pruning. The core idea revolves around grouping filters using a similarity criterion and removing common convolutional kernels with the group. The proposed work explores optimal grouping schemes for filters and after pruning unwanted filters, the retained filters are restructured and the network can be fine-tuned to recoup prediction accuracy.