Pruning Compact ConvNets For Efficient Inference

1 INTRODUCTION . Neural networks frequently suffer from the problem of over-parameterization , such that the model can be compressed by a large factor to drastically reduce memory footprint , computation as well as energy consumption while maintaining similar performance . This is especially pronounced for models for computer vision ( Simonyan & Zisserman , 2014 ) , speech recognition ( Pratap et al. , 2020 ) and large text understanding models such as BERT ( Devlin et al. , 2018 ) . The improvements obtained from intelligently reducing the number of model parameters has several benefits , such as reduction in datacenter power consumption , faster inference and reduced memory footprint on edge devices such as mobile phones which also enable decentralized techniques ex . federated learning ( Kairouz et al. , 2019 ) . There are several techniques to reduce model size while maintaining similar generalization performance , such as model quantization ( Polino et al. , 2018 ) , NAS ( Neural Architecture Search ) ( Elsken et al. , 2019 ) and model distillation through teacher-student networks ( Gou et al. , 2021 ) . For the scope of this paper , we consider pruning as a technique to remove trainable weights in the network , and save on computation costs for the FBNet family of models . The motivations for this are two-fold . Firstly , state-of-the-art models such as FBNet ( Wu et al. , 2019 ) already adopt the best practices in the area of efficient hardware-aware design of convolutional neural network based models , and are widely used across different vision tasks . This makes them suitable baselines to understand whether pruning can offer any performance gain over their already optimized behavior . While there has been limited work on pruning for efficient convolution network models they investigate older architectures such as EfficientNet and MobileNet ( Aflalo et al. , 2020 ) or integrate pruning into expensive techniques such as joint prune-and-architecture search ( Wang et al. , 2020 ) . For each of the constituent models of the FBNetV3 family ( FBNetV3A , FBNetV3B , ... , FBNetV3G ) we reduce the number of parameters using two pruning based approaches : ( 1 ) Global magnitudebased pruning : Starting with the pre-trained model , we prune all weights whose magnitude is below a threshold chosen in order to achieve a target number of FLOPs for the pruned model ; ( 2 ) Uniform magnitude-based pruning : Starting with the pre-trained model , we prune weights in each layer whose magnitude is below a level-specific threshold in order to yield a pruned model achieving a target number of FLOPs with the same sparsity in each layer . After either pruning method is applied , we fine-tune the pruned model for a certain number of epochs until convergence is reached . Within the scope of our study in this paper , we are mostly interested in the following research questions : • RQ1 : Pruning to improve computation vs. performance tradeoff . Can a model obtained by pruning a larger FBNetV3 model M1 ( optimized using NAS ) achieve higher generalization performance than a smaller FBNetV3 model M2 when the pruned model has the same number of FLOPs as M2 ? • RQ2 : Pruning as an efficient paradigm . When a larger FBNetV3 model M1 is available and computational resources are limited , is pruning a faster and less computationally expensive approach to obtain a model with higher accuracy at a desired computation level ( FLOPs ) than running a full-fledged architecture search ? Pruning to improve computation vs. performance tradeoff ( RQ1 ) . There have been recent research advances in the area of building hardware-aware efficient models ( Deng et al. , 2020 ) . These can provide good generalization performance while adhering to constraints on memory , inference latency and battery power , which are often dictated by the hardware environment where inference happens . Experiments described in existing work on efficient vision models such as ChamNet ( Dai et al. , 2019 ) , MobileNet ( Howard et al. , 2017 ) , EfficientNet ( Tan & Le , 2019 ) and FBNetV2 ( Wan et al. , 2020 ) have shown that it is possible to achieve even higher performances on standard image recognition tasks such as ImageNet ( Deng et al. , 2009 ) at a certain level of FLOPs . However the efficient design of these models does not solve the over-parameterization problem completely , and none of these approaches study how model pruning can be performed to obtain even better trade-offs between computation and model accuracy . This paper is the first of its kind to understand how we can improve on the state-of-the-art in this problem space . Pruning as an efficient paradigm ( RQ2 ) . In addition to achieving state-of-the-art performance with reduced FLOPs , we are also interested in understanding how such pruned models can be obtained inexpensively with limited resources that are generally available to a machine learning practitioner who has access to existing optimized models but limited computing resources . For example , the FBNetV3 models are freely available through Facebook ’ s Mobile Model Zoo1 , while EfficientNet models can be obtained at GitHub2 . While the techniques needed to obtain computation- and latencyfriendly models have been democratized through open-sourcing the source code as well as the models themselves , fully applying these techniques necessitates costly operations such as finding an optimal network topology through meta-learning approaches ( You et al. , 2020 ) and search algorithms such as Genetic Algorithms ( GAs ) ( Goldberg & Deb , 1991 ) . Given the high-degree of intractability of this problem , expensive computational resources are often needed in this case , easily exceeding the budget available to a university research laboratory or an angel-stage startup ( Zoph & Le , 2016 ) . When a starting model is already available , for example through open-sourcing , the best option would be to perform a cheap modification of the model to fit a certain target FLOPs/latency requirement . In this paper we have compared the NAS approaches for training FBNetV3 models with our pruning techniques on a computational complexity metric ( GPU-hours ) to effectively answer RQ2 . Benchmark results . In addition to experimental outcomes for answering RQ1 and RQ2 , we also benchmark pruned FBNetV3 models using available open-sourced quantized sparse kernels and conduct ablation studies to obtain additional insights into pruning performance . These results augment our main observations and demonstrate that with existing hardware support , it is possible to deploy pruned cutting-edge computer vision models with practical latency reductions and improve further beyond the performance vs. FLOPs trade-off . We conduct our experiments on ImageNet , which is an object-recognition task on a large training dataset of 1.2 million images . We show that computationally less intensive techniques such as uniform and global magnitude-based pruning of larger FBNetV3 models can yield higher test accuracies than small models while having the same number of FLOPs . Given a target computation budget for an efficient model , we show that it is more practically advantageous ( both in terms of performance and running time ) to simply prune the larger model than run a neural architecture search to find the target model from scratch . 1FBNetV3 models available here http : //https : //github.com/facebookresearch/ mobile_cv/model_zoo/models/model_info/fbnet_v2/model_info_fbnet_v3.json 2EfficientNet models available here https : //github.com/mingxingtan/efficientnet The technique we have employed for pruning ( unstructured sparsity ) is already tried and tested , however our novelty lies in studying whether efficient image recognition models such as FBNetV3 can be optimized further to improve on the FLOPs-accuracy curve , and the contributions are two fold : ( 1 ) FBNets are themselves state-of-the-art in efficient vision models and we achieve better accuracy-FLOPs tradeoff over these models and ( 2 ) from the standpoint of computational overhead , we significantly reduce the amount of GPU hours required to obtain such models . Pruning a publicly available NAS optimized model incurs ≈4x less GPU hours to achieve a target FLOPs level , compared to training a full-fledged NAS to obtain a model which has less accuracy at the same FLOPs level . Paper organization . The remainder of this paper is organized as follows . In Section 2 , we describe related work in the area of efficient vision model design and also provide an introduction to different pruning techniques . In Section 3 , we discuss our experimental setup , including a description of the baseline models and the global and uniform pruning approaches we have employed . Section 4 describes our main findings and we conclude the paper in Section 5 . 2 RELATED WORK . We discuss related literature in the areas of computationally efficient vision models and model pruning . Within the scope of our work , we mainly focus on inference efficiency of models in contrast to training efficiency . Computationally efficient vision models : Neural networks for computer vision are generally characterized by convolutional layers and fully-connected layers , along with blocks such as residual or skip connections . This makes such networks resource intensive in terms of FLOPs , which affects the memory storage and power consumed , and also leads to increased latency . It is of paramount importance to design more efficient networks which can provide higher performance for the same FLOPs or latency level , or even to optimize them appropriately to provide the same performance at reduced FLOPs/latency . This can be performed either through the design of new simplified layers , for example in deep residual learning ( He et al. , 2016 ) or though explicit model compression as in weight quantization ( Polino et al. , 2018 ) . Extremely deep networks for image recognition often suffer from not only high complexity and inference latency , but also from the issue of vanishing gradients ( Pascanu et al. , 2013 ) . This was addressed through deep residual networks which effectively simplified network design through skip-connections . MobileNets ( Howard et al. , 2017 ) are one of the earlier approaches to building small low-latency networks by using depthwise separable convolutions with two parameters , width and resolution multipliers . They demonstrate the effectiveness of MobileNets across different vision tasks , such as face embeddings and object detection . MobileNetV2 ( Sandler et al. , 2018 ) extends MobileNets by utilizing inverted residual filter structures and linear bottlenecks , obtaining improvements on state-of-the-art models both in terms of accuracy and computational complexity . ShuffleNets ( Zhang et al. , 2018 ) propose dedicated residual units where 1×1 convolutions are replaced with pointwise group convolutions and channel shuffling reducing FLOPs computations . More recently , the focus on building efficient neural network models has shifted to techniques that treat the design of efficient networks as a search problem , falling under the umbrella of Neural Architecture Search ( NAS ) . EfficientNets ( Tan & Le , 2019 ) propose a novel scaling method which adjusts the network ’ s length , width , and resolution to optimize performance subject to target memory and FLOPs constraints . They also define a novel baseline that is optimized by a multi-objective neural architecture search . The FBNet collections of models—FBNet ( Wu et al. , 2019 ) , FBNetV2 ( Wan et al. , 2020 ) and FBNetV3 ( Dai et al. , 2021 ) —employ neural architecture search to obtain highly-optimized models that improve on the state-of-the-art for different visual understanding tasks . FBNet frames the architecture search as a differentiable meta-learning problem with gradient based techniques , namely DNAS—Differentiable Neural Architecture Search—by Wu et al . ( 2019 ) , and avoids selecting the optimized model over a discrete set . The subsequent entry in this collection , FBNetV2 , expands the search space over conventional DNAS , and employs a masking scheme to maintain the same level of computational complexity while searching over this expanded space . FBNetV3 further improves on the state-of-the-art by employing Neural Architecture Recipe Search ( NARS ) and searching over the space of not only architectures , but also corresponding recipes ( which are generally hyper-parameters ) . In this paper , we consider FBNetV3 models as our baselines as they are state-of-the-art . We are interested in understanding if they are overparameterized and evaluate how much model pruning can improve performance at a certain FLOPs level over the state-of-the-art in this family of models . Model Pruning : Modern neural networks , particularly those processing complex sensory inputs ( such as speech , vision and language ) for perception applications , are often over-parameterized . It is only to be expected that we should be able to compress such networks significantly to maintain the same level of performance at decreased level of computation ( fewer weights and reduced FLOPs ) , memory footprint and power consumption . Foundational efforts in this space include the Optimal Brain Surgeon ( Hassibi & Stork , 1993 ) and Optimal Brain Damage ( LeCun et al. , 1990 ) . Recently the idea of network pruning has been formalized through the lottery ticket hypothesis ( Frankle & Carbin , 2018 ) , which claims that randomly initialized , feed-forward networks have winning sub-networks that perform just as well as the original network on an unseen test dataset . Model pruning is generally of two types : unstructured and structured pruning . Unstructured pruning , as the name suggests , doesn ’ t adhere to any structure and prunes neurons based on chosen criteria ( such as magnitude ) . This has the advantage of providing higher performance , but is difficult to implement in hardware , as it needs dedicated support for efficient sparse matrix multiplications . Meanwhile , structured pruning is the practice of removing entire groups of neurons ( e.g. , blocks within the weight matrix , or channels in convolutional neural networks ) . This is easy to implement without dedicated hardware support , but has the issue of lower generalization performance than unstructured pruning ( Yao et al. , 2019 ) . In the literature , there have also been several studies , for example investigating whether rewinding ( training from scratch with a fixed mask ) can perform just as well as the fine-tuning on top of the original unpruned network ( Renda et al. , 2020 ) . Blalock et al . ( 2020 ) provide an overview survey of recent advances and open problems in neural network pruning . In the research area of designing efficient networks for computer vision , there has not been much focus on understanding how pruning can be applied to the current generation of models . Most literature on pruning is based on older networks such as VGGNet , ResNet ( He et al. , 2016 ) , and MobileNet ( Sandler et al. , 2018 ) . Our work improves upon these existing studies by understanding how pruning can improve the FLOPs-accuracy tradeoff over existing state-of-the-art networks .

Summary: This paper applies conventional pruning-and-finetuning techniques to further compress the networks searched by NAS. The experiments and evaluations are based on the family of FBNetV3. The authors show that by pruning large FBNetV3 model to small one, the accuracy of pruned model may be slightly better than the original target (small) model, achieving better tradeoff between computational complexity and accuracy. Also, the authors show that pruning is more training-efficient than searching by NAS to achieve a compact FBNetV3 model.