Pruning Compact ConvNets For Efficient Inference

1 INTRODUCTION . Neural networks frequently suffer from the problem of over-parameterization , such that the model can be compressed by a large factor to drastically reduce memory footprint , computation as well as energy consumption while maintaining similar performance . This is especially pronounced for models for computer vision ( Simonyan & Zisserman , 2014 ) , speech recognition ( Pratap et al. , 2020 ) and large text understanding models such as BERT ( Devlin et al. , 2018 ) . The improvements obtained from intelligently reducing the number of model parameters has several benefits , such as reduction in datacenter power consumption , faster inference and reduced memory footprint on edge devices such as mobile phones which also enable decentralized techniques ex . federated learning ( Kairouz et al. , 2019 ) . There are several techniques to reduce model size while maintaining similar generalization performance , such as model quantization ( Polino et al. , 2018 ) , NAS ( Neural Architecture Search ) ( Elsken et al. , 2019 ) and model distillation through teacher-student networks ( Gou et al. , 2021 ) . For the scope of this paper , we consider pruning as a technique to remove trainable weights in the network , and save on computation costs for the FBNet family of models . The motivations for this are two-fold . Firstly , state-of-the-art models such as FBNet ( Wu et al. , 2019 ) already adopt the best practices in the area of efficient hardware-aware design of convolutional neural network based models , and are widely used across different vision tasks . This makes them suitable baselines to understand whether pruning can offer any performance gain over their already optimized behavior . While there has been limited work on pruning for efficient convolution network models they investigate older architectures such as EfficientNet and MobileNet ( Aflalo et al. , 2020 ) or integrate pruning into expensive techniques such as joint prune-and-architecture search ( Wang et al. , 2020 ) . For each of the constituent models of the FBNetV3 family ( FBNetV3A , FBNetV3B , ... , FBNetV3G ) we reduce the number of parameters using two pruning based approaches : ( 1 ) Global magnitudebased pruning : Starting with the pre-trained model , we prune all weights whose magnitude is below a threshold chosen in order to achieve a target number of FLOPs for the pruned model ; ( 2 ) Uniform magnitude-based pruning : Starting with the pre-trained model , we prune weights in each layer whose magnitude is below a level-specific threshold in order to yield a pruned model achieving a target number of FLOPs with the same sparsity in each layer . After either pruning method is applied , we fine-tune the pruned model for a certain number of epochs until convergence is reached . Within the scope of our study in this paper , we are mostly interested in the following research questions : • RQ1 : Pruning to improve computation vs. performance tradeoff . Can a model obtained by pruning a larger FBNetV3 model M1 ( optimized using NAS ) achieve higher generalization performance than a smaller FBNetV3 model M2 when the pruned model has the same number of FLOPs as M2 ? • RQ2 : Pruning as an efficient paradigm . When a larger FBNetV3 model M1 is available and computational resources are limited , is pruning a faster and less computationally expensive approach to obtain a model with higher accuracy at a desired computation level ( FLOPs ) than running a full-fledged architecture search ? Pruning to improve computation vs. performance tradeoff ( RQ1 ) . There have been recent research advances in the area of building hardware-aware efficient models ( Deng et al. , 2020 ) . These can provide good generalization performance while adhering to constraints on memory , inference latency and battery power , which are often dictated by the hardware environment where inference happens . Experiments described in existing work on efficient vision models such as ChamNet ( Dai et al. , 2019 ) , MobileNet ( Howard et al. , 2017 ) , EfficientNet ( Tan & Le , 2019 ) and FBNetV2 ( Wan et al. , 2020 ) have shown that it is possible to achieve even higher performances on standard image recognition tasks such as ImageNet ( Deng et al. , 2009 ) at a certain level of FLOPs . However the efficient design of these models does not solve the over-parameterization problem completely , and none of these approaches study how model pruning can be performed to obtain even better trade-offs between computation and model accuracy . This paper is the first of its kind to understand how we can improve on the state-of-the-art in this problem space . Pruning as an efficient paradigm ( RQ2 ) . In addition to achieving state-of-the-art performance with reduced FLOPs , we are also interested in understanding how such pruned models can be obtained inexpensively with limited resources that are generally available to a machine learning practitioner who has access to existing optimized models but limited computing resources . For example , the FBNetV3 models are freely available through Facebook ’ s Mobile Model Zoo1 , while EfficientNet models can be obtained at GitHub2 . While the techniques needed to obtain computation- and latencyfriendly models have been democratized through open-sourcing the source code as well as the models themselves , fully applying these techniques necessitates costly operations such as finding an optimal network topology through meta-learning approaches ( You et al. , 2020 ) and search algorithms such as Genetic Algorithms ( GAs ) ( Goldberg & Deb , 1991 ) . Given the high-degree of intractability of this problem , expensive computational resources are often needed in this case , easily exceeding the budget available to a university research laboratory or an angel-stage startup ( Zoph & Le , 2016 ) . When a starting model is already available , for example through open-sourcing , the best option would be to perform a cheap modification of the model to fit a certain target FLOPs/latency requirement . In this paper we have compared the NAS approaches for training FBNetV3 models with our pruning techniques on a computational complexity metric ( GPU-hours ) to effectively answer RQ2 . Benchmark results . In addition to experimental outcomes for answering RQ1 and RQ2 , we also benchmark pruned FBNetV3 models using available open-sourced quantized sparse kernels and conduct ablation studies to obtain additional insights into pruning performance . These results augment our main observations and demonstrate that with existing hardware support , it is possible to deploy pruned cutting-edge computer vision models with practical latency reductions and improve further beyond the performance vs. FLOPs trade-off . We conduct our experiments on ImageNet , which is an object-recognition task on a large training dataset of 1.2 million images . We show that computationally less intensive techniques such as uniform and global magnitude-based pruning of larger FBNetV3 models can yield higher test accuracies than small models while having the same number of FLOPs . Given a target computation budget for an efficient model , we show that it is more practically advantageous ( both in terms of performance and running time ) to simply prune the larger model than run a neural architecture search to find the target model from scratch . 1FBNetV3 models available here http : //https : //github.com/facebookresearch/ mobile_cv/model_zoo/models/model_info/fbnet_v2/model_info_fbnet_v3.json 2EfficientNet models available here https : //github.com/mingxingtan/efficientnet The technique we have employed for pruning ( unstructured sparsity ) is already tried and tested , however our novelty lies in studying whether efficient image recognition models such as FBNetV3 can be optimized further to improve on the FLOPs-accuracy curve , and the contributions are two fold : ( 1 ) FBNets are themselves state-of-the-art in efficient vision models and we achieve better accuracy-FLOPs tradeoff over these models and ( 2 ) from the standpoint of computational overhead , we significantly reduce the amount of GPU hours required to obtain such models . Pruning a publicly available NAS optimized model incurs ≈4x less GPU hours to achieve a target FLOPs level , compared to training a full-fledged NAS to obtain a model which has less accuracy at the same FLOPs level . Paper organization . The remainder of this paper is organized as follows . In Section 2 , we describe related work in the area of efficient vision model design and also provide an introduction to different pruning techniques . In Section 3 , we discuss our experimental setup , including a description of the baseline models and the global and uniform pruning approaches we have employed . Section 4 describes our main findings and we conclude the paper in Section 5 . 2 RELATED WORK . We discuss related literature in the areas of computationally efficient vision models and model pruning . Within the scope of our work , we mainly focus on inference efficiency of models in contrast to training efficiency . Computationally efficient vision models : Neural networks for computer vision are generally characterized by convolutional layers and fully-connected layers , along with blocks such as residual or skip connections . This makes such networks resource intensive in terms of FLOPs , which affects the memory storage and power consumed , and also leads to increased latency . It is of paramount importance to design more efficient networks which can provide higher performance for the same FLOPs or latency level , or even to optimize them appropriately to provide the same performance at reduced FLOPs/latency . This can be performed either through the design of new simplified layers , for example in deep residual learning ( He et al. , 2016 ) or though explicit model compression as in weight quantization ( Polino et al. , 2018 ) . Extremely deep networks for image recognition often suffer from not only high complexity and inference latency , but also from the issue of vanishing gradients ( Pascanu et al. , 2013 ) . This was addressed through deep residual networks which effectively simplified network design through skip-connections . MobileNets ( Howard et al. , 2017 ) are one of the earlier approaches to building small low-latency networks by using depthwise separable convolutions with two parameters , width and resolution multipliers . They demonstrate the effectiveness of MobileNets across different vision tasks , such as face embeddings and object detection . MobileNetV2 ( Sandler et al. , 2018 ) extends MobileNets by utilizing inverted residual filter structures and linear bottlenecks , obtaining improvements on state-of-the-art models both in terms of accuracy and computational complexity . ShuffleNets ( Zhang et al. , 2018 ) propose dedicated residual units where 1×1 convolutions are replaced with pointwise group convolutions and channel shuffling reducing FLOPs computations . More recently , the focus on building efficient neural network models has shifted to techniques that treat the design of efficient networks as a search problem , falling under the umbrella of Neural Architecture Search ( NAS ) . EfficientNets ( Tan & Le , 2019 ) propose a novel scaling method which adjusts the network ’ s length , width , and resolution to optimize performance subject to target memory and FLOPs constraints . They also define a novel baseline that is optimized by a multi-objective neural architecture search . The FBNet collections of models—FBNet ( Wu et al. , 2019 ) , FBNetV2 ( Wan et al. , 2020 ) and FBNetV3 ( Dai et al. , 2021 ) —employ neural architecture search to obtain highly-optimized models that improve on the state-of-the-art for different visual understanding tasks . FBNet frames the architecture search as a differentiable meta-learning problem with gradient based techniques , namely DNAS—Differentiable Neural Architecture Search—by Wu et al . ( 2019 ) , and avoids selecting the optimized model over a discrete set . The subsequent entry in this collection , FBNetV2 , expands the search space over conventional DNAS , and employs a masking scheme to maintain the same level of computational complexity while searching over this expanded space . FBNetV3 further improves on the state-of-the-art by employing Neural Architecture Recipe Search ( NARS ) and searching over the space of not only architectures , but also corresponding recipes ( which are generally hyper-parameters ) . In this paper , we consider FBNetV3 models as our baselines as they are state-of-the-art . We are interested in understanding if they are overparameterized and evaluate how much model pruning can improve performance at a certain FLOPs level over the state-of-the-art in this family of models . Model Pruning : Modern neural networks , particularly those processing complex sensory inputs ( such as speech , vision and language ) for perception applications , are often over-parameterized . It is only to be expected that we should be able to compress such networks significantly to maintain the same level of performance at decreased level of computation ( fewer weights and reduced FLOPs ) , memory footprint and power consumption . Foundational efforts in this space include the Optimal Brain Surgeon ( Hassibi & Stork , 1993 ) and Optimal Brain Damage ( LeCun et al. , 1990 ) . Recently the idea of network pruning has been formalized through the lottery ticket hypothesis ( Frankle & Carbin , 2018 ) , which claims that randomly initialized , feed-forward networks have winning sub-networks that perform just as well as the original network on an unseen test dataset . Model pruning is generally of two types : unstructured and structured pruning . Unstructured pruning , as the name suggests , doesn ’ t adhere to any structure and prunes neurons based on chosen criteria ( such as magnitude ) . This has the advantage of providing higher performance , but is difficult to implement in hardware , as it needs dedicated support for efficient sparse matrix multiplications . Meanwhile , structured pruning is the practice of removing entire groups of neurons ( e.g. , blocks within the weight matrix , or channels in convolutional neural networks ) . This is easy to implement without dedicated hardware support , but has the issue of lower generalization performance than unstructured pruning ( Yao et al. , 2019 ) . In the literature , there have also been several studies , for example investigating whether rewinding ( training from scratch with a fixed mask ) can perform just as well as the fine-tuning on top of the original unpruned network ( Renda et al. , 2020 ) . Blalock et al . ( 2020 ) provide an overview survey of recent advances and open problems in neural network pruning . In the research area of designing efficient networks for computer vision , there has not been much focus on understanding how pruning can be applied to the current generation of models . Most literature on pruning is based on older networks such as VGGNet , ResNet ( He et al. , 2016 ) , and MobileNet ( Sandler et al. , 2018 ) . Our work improves upon these existing studies by understanding how pruning can improve the FLOPs-accuracy tradeoff over existing state-of-the-art networks .

Summary: The paper presents an experimental evaluation of simple pruning techniques applied to modern architectures that are designed to be inherently resource-efficient. It is shown that pruning large models in the FBNetV3 family achieves better accuracy-FLOPS trade-offs than smaller models in the FBNetV3 family. It is also shown that pruning large models is faster than performing NAS to find the smaller models directly.