Pruning Compact ConvNets For Efficient Inference
Authors: Sayan Ghosh
Karthik Prasad
Xiaoliang Dai
Peizhao Zhang
Bichen Wu
Graham Cormode
Peter Vajda
Venue: ICLR 2022 Submitted (2022)
Keywords: pruning
neural networks
computations
latency
imagenet

Abstract: Neural network pruning is frequently used to compress over-parameterized networks by large amounts, while incurring only marginal drops in generalization performance. However, the impact of pruning on networks that have been highly optimized for efficient inference has not received the same level of attention. In this paper, we analyze the effect of pruning for computer vision, and study state-of-the-art FBNetV3 family of models. We show that model pruning approaches can be used to further optimize networks trained through NAS (Neural Architecture Search). The resulting family of pruned models can consistently obtain better performance than existing FBNetV3 models at the same level of computation, and thus provide state-of-the-art results when trading off between computational complexity and generalization performance on the ImageNet benchmark. In addition to better generalization performance, we also demonstrate that when limited computation resources are available, pruning FBNetV3 models incur only a fraction of GPU-hours involved in running a full-scale NAS (Neural Architecture Search).

Under review as a conference paper at ICLR 2022
PRUNING COMPACT CONVNETS FOR EFFICIENT INFER- ENCE
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural network pruning is frequently used to compress over- over-pareee parameterized net- works by large amounts, while incurring only marginal drops in generalization performance. However, the impact of pruning on networks that have been highly optimized for efficient inference has not received the same level of attention. this paper, we analyze the effect of pruning for computer vision, and study state-of- the-art ConvNets, such as the FBNetV3 family of models. We show that model pruning approaches can be used to further optimize networks trained through NAS (Neural Architecture Search). The resulting family pruned models can consis- tently obtain better performance than existing FBNetV3 models the same level of computation, and thus provide state-of-the-ard results when trading off between computational complexity and generalization performance the ImageNet bench- mark. In addition to better generalization performance, we also demonstrate that when limited computation resources are available, pruning FBNetV3 models incur only fraction of GPU-hours involved in running a full-scale NAS.
INTRODUCTION
Neural networks frequently suffer from the problem of peer-parameterization, such that the model can be compressed by large factor to drastically reduce memory footprint, computation well as energy consumption while maintaining similar performance. This is especially pronounced for models for computer vision (Simonyan & Zisserman 2014)), speech recogmition (Pratap al.) (0220) and large text understanding models such as BERT (Devlin et al.) 2018). The improvements obtained from intelligently reducing the number of model parameters has several benefits, such as reduction in datacenter power consumption, faster inference and reduced memory footprint on edge devices such as mobile phones which also enable decentralized techniques ex. federated learning (Kairouz et al. 2019).
There are several techniques to reduce model size while maintaining similar generalization perfor- mance, such model quantization (Polino al. 2018). NAS (Neural Architecture Search) (Elsken) ct al. 2019) and model distillation through ceacher-student networks (Gou et al. 202I). ). For the scope of this paper, consider pruning on computation costs for the FBNet family of models. The motivations for this are two-fold. Firstly, state-of-the-ar models such as FBNet (Wu et al. 2019) already adopt the best practices in the area of efficient hardware-aware design of convolutional neural network based models, and are widely used across different vision tasks. This makes them suitable baselines to understand whether pruning can offer any performance gain over their already optimized behavior. While there has been limited work on pruming for efficient convolution network models they investigate older architectures such as EfficientNet and MobileNet (Aflalo et al. (020) or integrate pruning into expensive techniques such as joint prune-and -aw--andiaaattttureeeerr search (Wang et al. 2020).
technique remove trainable weights in the network, and save
For each of the constituent models of the F3NetV3 family (FBNeIV3A, FBNetV3B.... FBNe(V3G)
we reduce the number of parameters using two pruning based approaches: (I) Global magnitude- based pruning: Starting with the pre-trained model, we prune all weights whose magnitude is below a threshold chosen order to achieve a target number of FLOPs for the pruned model; (2) Uniform magnitude-bass -bsedddeeebdeddd--ee pruning: Starting with the pre- treitrained model. magnitude is below a level-specific threshold in order to yield a pruned model achieving a target number of FLOPPs with the same sparsity in each layer. After either pruning method is applied,
prune weights in each layer whose

---

Under review as a conference paper at ICLR 2022
fine-tune the pruned model for certain number of epochs until convergence is reached. Within the scope of our study in this paper, we are mostly interested in the following research questions:
RQL: Pruning to improve computation vs. performance tradeoff. Can a model obtained by pruning a larger VBNetV3 model M1 (optimized using NAS) achieve higher generalization performance than a smaller FBNetV3 model M2 when the pruned model has the same number of FLOPs as M2?
RQ2: Pruning as an efficient paradigm. When a larger FBNeIV3 model M1 is available and computational resources are limited, is pruning a faster and less computationally expensive approach to obtain model with higher accuracy at a desired computation level (FLOPs) than running a full-fledged architecture search?
Pruning to improve computation vs. performance tradeoff (RQI). There have been recent research advances in the area of building hardware-awan efficient models (Deng et al., 2020). These can provide good generalization performance while adhering to constraints on memory, inference latency and battery power, which are often dictated by the hardware environment where inference happens. Experiments described in existing work on efficient vision models such as ChamNet (Dai et al. 2019), MobileNet (Howard et al. 2017). EfficientNet (Tan & Le| 2019)) and FBNetV2 (Wan et al. 2020) have shown that it is possible to achieve even higher performances on standard image recognition tasks such ImageNet (Deng et al.) (2009) at certain level of FLOPs. However the efficient design of these models does not solve the over-parameetrrizatoon problem completely, and none of these approaches study how model pruning can be performed to obtain even better trade-offs between computation and model accuracy. This paper is the first of its kind to understand how we can improve on the state-of-the -trtttrrrttttt art in this problem space.
Pruming as an efficient paradigm (RQ2). In addition to achieving state-of-the-ar performance with reduced FLOPs, we are also interested in understanding how such pruned models can be obtained inexpensively with limited resources that are generally available to machine learning practitioner who has access to existing optimized models but limited computing resources. For example, the 33 models are freely available through Facebook's 's Mobile Model Zod while EfficientNet models can be obtained at GitHull? While the techniques needed to obtain computation- and latency- friendly models have been democratized through open-sourcing the source code as well as the models themselves, fully applying these techniques necessitates costly operations such as finding an optimal network topology through meta-leaming approaches Yoou et al. 2020) and search algorithms such as Genetic Algorithms (GAs) (Goldberg & Deb 1991).
Given the high-degree of intractability of this problem, expensive computational resources are often needed in this case, easily exceeding the budget available to university research laboratory or an angel-stage startup (Zoph Le 2016). When starting model is already available, for example through open-soureing, the best option would be to perform a cheap modification of the model to fit a certain target FLOPs/latency requirement. In this paper we have compared the NAS approaches for training FBNetV3 models with our pruning techniques on a computational complexity metric (GPU-hours) to effectively answer RQ2.
Benchmark results. In addition to experimental outcomes for answering RQ1 and RQ2, we also benchmark pruned FBNetV3 models using available open-sourced quantized sparse kernels and conduct ablation studies to obtain additional insights into pruning performance. These results augment our main observations and demonstrate that with existing hardware support, is possible to deploy pruned cutting-edi computer vision models with practical latency reductions and improve further beyond the performance vs. FLOPs trade-off.
We conduct our experiments ImageNet, which is an object-recognition task on large training dataset of 1.2 million images. We show that computationally less intensive techniques such as uniform and global magnitude- based pruning of larger FBNetV3 models can yield higher test accuracies than small models while having the same number of FLOPs. Given a target computation budget for an efficient model, we show that it is more practically advantageous (both in terms of performance and running time) to simply prune the larger model than run a neural architecture search to find the target model from scratch.
FBBNeV33 models available here http:///ttess//gtttoooo////ooeee http:///foooeee_eeeeeeeeeee_e__ fbnet_ 13.oon
mobi _e_cv/ /mmee l_aeell_ooooooooo mooo/mooellooo 1 s http:///aaaaaaa/////o//oo//////a "fffiiientNet models available here
2

---

Under review as a conference paper at ICLR 2022
The technique we have employed for pruning (unstructured sparsity) is already tried and tested, however our novelty lies in studying whether efficient image recogmition models such as FBNetV3 can be optimized further to improve the FLOPs-accuracy curve, and the contributions are two fold: : (1) FBNets are themselves state-of-the-a in efficient vision models and we achieve better accuracy- FLOOPs tradeoff over these models and (2) from the standpoint of computational overhead, we significantly reduce the amount of GPU hours required to obtain such models. Pruning publicly available NAS optimized model incurs 6:4x less GPU hours to achieve a target FLOPs level, compared to training full-fledged NAS to obtain model which has less accuracy at the same FLOPs level.
Paper organization. The remainder of this paper is organized as follows. In Section 2 we describe related work in the area of efficient vision model design and also provide an introduction to different pruning techniques.
Section 3 we discuss our experimental setup, including a description of the baseline models and the global and uniform pruning approaches we have employed. Section describes our main findings and we conclude the paper in Section
RELATED WORK
We discuss related literature in the areas of computationally efficient vision models and model pruming. Within the scope of our work, we mainly focus on inference efficiency of models in contrast to training efficiency.
Computationally efficient vision models: Neural networks for computer vision are generally char- acterized by convolutional layers and fully-connectoo layers, along with blocks such as residual or skip connections. This makes such networks resource intensive in terms of FLOPs, which affects the memory storage and power consumed, and also leads to increased latency. It is of paramount importance to design more efficient networks which can provide higher performance for the same FLOPs or latency level, or even to optimize them appropriately to provide the same performance at reduced FLOP FLOPs/latency. This can be performed either through the design of new simplified layers, for example in deep residual learning (He et al. 2016) or though explicit model compression as in weight quantization (Polino et al., (0188) Extremely deep networks for image recognition often suffer from not only high complexity and inference latency, but also from the issue of vanishing gradients (Pas canu al. 2013). This was addressed through deep residual networks which effectively simplified network design through skip-connections. MobileNets (Howard et al. 2017) are one of the earlier approaches to building small low low-natecyy networks by using depthwise separable convolutions with two parameters, width and resolution multipliers. They demonstrate the effectiveness of MobileNets across different vision tasks, such as face embeddings and object detection. MobileNetV2 (Sandler et al. 2018) extends MobileNets by utilizing inverted residual filter structures and linear bottlenecks, obtaining improvements on state-of of models both in terms of accuracy and computational com- plexity. ShuffleNets (Zhang al. 2018) propose dedicated residual units where I Ix] I convolutions are replaced with pointwise group convolutions and channel shuffling reducing FLOPs computations.
More recently, the focus on building efficient neural network models has shifted to techniques that treat the design of efficient networks as a scarch problem, falling under the umbrella of Neural Architecture Search (NAS). EfficientNets (Tan & Le 2019) propose novel scaling method which adjusts the network's ' length, width, and resolution to optimize performance subject to target memory and FLOPS constraints. They also define novel baseline that is optimized by multi-objective neural architecture search. The FBNet collections of models-FBNe (Wu et al. 2019). FBNeIV2 (Wan et al., neural architecture search to obtain highly-optimized
2020) and 3BNetV3 (Dai al. 2021)-emin models that improve on the state-of-the-art for different visual understanding tasks. FBNet frames the architecture search as a differentiable meta-learning problem with gradient based techniques, namely Diff---ifrreettabbll Neural Architecture Search- by Wu et al. (2019). and avoids selecting the optimized model over a discrete set. The subsequent entry in this collection, FBNetV2, expands the search space over conventional DNAS, and employs masking scheme to maintain the same level of computational complexity while searching over this expanded space. FBNetV3 further improves on the state-of- oftt---rt by employing Neural Architecture Recipe Search (NARS) and searching over the space of not only architectures, but also corresponding recipes (which are generally hyper-parametert In this paper, we consider FBNetV3 models our baselines they ofate-of-the-ar We are interested in understanding if they are overparameterized and evaluate how much model pruning can improve performance at certain FL.PPs level over the -fate-of-the-art in this family of models.
3

---

Under review as a conference paper at ICLR 2022
Model Pruming: Modern neural networks, particularly those processing complex sensory inputs (such as speech, vision and language) for perception applications, are often over parameterized. It is only to be expected that should be able to compress such networks significantly to maintain the same level of performance at decreased level of computation (fewer weights and reduced FI FLOOss, memory footprint and power consumption. Foundational efforts in this space include the Optimal Brain Surgeon (Hassibi & Stork 1993) and Optimal Brain Damage (LeCun et al., 1990). Recently the idea of network pruning has been formalized through the lottery ticket hypothesis (Franlle & Carbin 2018), which claims that randomly initialized, feed- wordard networks have winning sub-networks that perform just as well as the original network on an unseen test dataset. Model pruning is generally of two types: unstructured and structured pruning. Unstructured pruning, as the name suggests, doesn't t adhere to any structure and prunes neurons based on chosen criteria (such as magnitude). This has the advantage of providing higher performance, but is difficult to implement in hardware, as it needs dedicated support for efficient sparse matrix multiplications. Meanwhile, structured pruning is the practice of removing entire groups of neurons (e.g., blocks within the weight matrix, or channels in convolutional neural networks). This is easy to implement without dedicated hardware support, but has the issue of lower generalization performance than unstructured pruning (Yao al.] 2019). In the literature, there have also been several studies, for example investigating whether rewinding (training from scratch with fixed mask) can perform just well as the fine-tuning on top of the original unpruned network (Renda et al., 2020 Blalock et al. (2020) provide an overview survey of recent advances and open problems neural network pruning.
In the research area of designing efficient networks for computer vision, there has not been much focus on understanding how pruning can be applied to the current generation of models. Most literature on pruning is based on older networks such VGGNet, ResNet (He et al. 2016), and MobileNet (Sandler et al., 2018). Our work improves upon these existing studies by understanding how pruning can improve the FLOPs-accuracyy tradeoff over existing state-of-the-art networks.
3 PRUNING TECHNIQUES AND SETUP
In this section, describe the main components of techniques and experimental setup, including Baseline Models, Pruning Techniques, Latency Measurement and Metrics. We have mainly used standard splits of the ImageNet dataset, further details are in Section A.l the appendix.
3.1 BASELINE MODELS
Dai al. (2020) address the previous limitations of NAS-based architecture search where these approaches can only search over architectures given a training recipe (set of hyperparameters), and thus cannot optimize over both. As described in Section 2 the most recent state-of-the-ard models are based on NARS (Neural Architecture-Resee Recipe Search), which we select as baseline models. Table lists the accuracy of FBNetV3 models (Dai et al., 202T) on the ImageNet classification task, along with the number of model parameters and computation complexity in terms of FLOPs.
Each baseline model consists of multiple IRF (Inverted Residual Filter) blocks, which contain convolutional layers of different kernel sizes. For our experiments, we are mostly interested in 1 x convolutions as potentially prunable, since within each FBNetV3 model, the 1xl convolution layers constitute >80% of total model FLOPs for all models in the family, and the open-sourced sparsity kernel support we use for latency benchmarking is available only for fully connected layers. A 1 convolution can be transformed into an equivalent fully connected layer with few tensor reshape operations without any significant loss of performance or latency.
For each initial and target FBNetV3 model X and Y. where X is larger than Y, prune X to sparsity level of S so that the FLOP count is the same as for Y. The number of FLOPS consumed by linear layer of sparsity S is proportional to the number of sparse matrix multiplications performed and is given by S F, where F' is the corresponding dense FLOPs. Thus if Fix (X) is the number of FLOPs consumed by the 1 Xx] convolution layers and F(x) is the total number of FLOPs consumed by model X, we have:
(1) Hence, sparsity measures the fraction of 1x1 convolution weights removed, and so higher sparsity indicates smaller model. For the uniform pruning senario, [ the amount of sparsity
S= =F((X) - F(Y))
4

---

Under review as a conference paper at ICLR 2022
required to prune each larger FBNetV3 model to smaller one based on Eq. IT. For global pruning, (I) does not hold, and we compute the target sparsities empirically from the layer shapes instead with details provided in Section A.2 We prune each larger FBNetV3 model to discrete FLOPs target based on a defined set of smaller models in the family, and not to a continuous range of FLOPPs values, it makes it easier compare models directly based on a given computation budget. If we can demonstrate that for the same computation level, the pruned larger FBNetV3 model has higher performance than a smaller model with the same FLOPs, it is sufficient to demonstrate that we can improve on the FLOPs- accuracy curve over the state-of-the-an of-the-art
3.2 PRUNING TECHNIQUES
In this paper, we utilize a pre-trained FBNetV3 model with higher number of FLOPss without training an image classification model from scratch with sparsity, which would be time consuming and computationally intensive. There are several approaches in the literature such as prune-and-fing tune (Han et al. 2015) and iterative pruning with sparsity scheduling (Frankle Carbin 2018). We have utilized the former for our experiments, as although studies have shown that iterative and incremental pruning approaches lead better generalization performance, they typically require training for high number of epochs, need tuning and selection of optimal sparsity schedules and are computationally resource intensive. We have therefore not considered them our experiments. For prune and fine-tune experiments, we have used 8-GPU boxes, with each box having Nvidia V100 (Volta) 32G GPUs. As described in Section []] experiments. For the latency benchmarking, we also perform magnitude-ban based uniform pruning with a sparse block size of x 4 as explained in Section 3.3
we perform both global and magnitude- based pruning
We have conducted hyper- parameerr tuning for the learning rate parameter, with LR values in the set (4e-5, 8e-5, 1. 1.6e-4), as fine-tuning generally admits smaller learning rates than training from scratch. We have found that using the same learning rate for all models, along with the same hyper-parameter settings used for training the seed model is sufficient to obtain pruned networks which are superior to the bascline FBNetV3 models. Hence minimal hyper-paramette tuning was required for our experiments and we have used values of settings such as weight decay and momentum to be the same as those used for training the baseline FBNetV3 models. During fine-tuning after pruning, we have used smoothed validation loss to stop the process early after convergence tolerance (0.01 %) reached between two consecutive epochs. Generally, we have observed fine-tuning to converge around 250 epochs.
3.3 LATENCY MFASUREMENTS AND METRICS
We are interested not only in the sparsity level of our pruned models and the image recognition performance, but also in metrics which potentially improve due to model sparsity, such as number of parameters, the FLOP count and the model latency. For reporting model performance under pruning, we use standard image recognition metrics such as Top-1 and Top-5 test accuracies. We measure overall model sparsity, which is different to the layer sparsity since only prune 1 1 convolution layers, as explained in Section 3.1 We report the model FLOPs, because this metric captures the computational footprint of the model and its power consumption.
Last, we record the total latency (in ms.) under pruning. The sparse kernels used in our experiments already in open-source and released under the PyTorch sparse quantization library using these kernels, we perform uniform layer-wise block-based pruning with block sizes of 1 x 4. Magnitude based pruning is implemented block level, and the model is quantized 8-bit integers (int8) before latency benchmarking, which is performed on Intel CPUs designed using the Skylake aecro-architecture While we would expect sparsity to translate to tangible inference speedups, this is highly dependent on the sparse kernel support provided by hardware. Current hardware is not well- suited for unstructured randomly sparse matrix multiplications and tend to do better with structured sparsity in models (Anwar et al. 2017). We have utilized block sparsity within the weight matrix for latency experiments. However this often tends to come at a decreased level of model performance. The design of highly performant sparse models under structured sparsity with reasonable inference speedups remains an important research topic outside the scope of this paper.
3 Prior to
5

---

Under review as a conference paper at ICLR 2022
Table Sparsity level (in percentage) and performance of pruned FBNetV3 networks on ImageNet dataset for different target MFLOPs, The best accuracy obtained at each target FLOPs level is highlighted in bold.
Seed
Target network
Target MFLOPs
Baseline Accuracy
Uniform pruning
Global pruning
network
Sparsity
Top-1
Sparsity level(%)
Top-
Gain %
Gain %
33NetV3
3BNetV3
level(%)
Acc.
Acc.
B
A
356.6
79.6
26.59
80.308
0.887
39.5
80.232
0.793
A
356.6
79.6
40.7
80.738
1.43
57.9
80.476
1.1
B
461.6
80.2
19.4
80.996
0.992
28.9
80.998
0.985
D
B
461.6
80.2
31.47
81.116
1.142
43.7
81.08
1.097
D
C
557.0
80.8
15.04
81.278
0.591
21.5
81.208
1.256
C
557.0
80.8
31.0
81.282
0.596
43.6
81.184
0.475
E
D
644.4
81.0
17.8
81.118
0.145
25.8
81.388
0.479
D
644.4
81.0
38.2
82.00
1.234
67.8
81,484
0.597
E
762.0
81.3
29.8
82.19
1.094
54.7
81.97
0.824
G
E
762.0
81.3
71.67
81.166
-0.16
85.5
79.934
-1.68
F
1181.6
82.0
49.69
82.528
0.643
63.8
82.454
0.553
4 RESULTS
4.1 PRUNED FBNETV3 MODEL PERFORMANCE
To answer RQI, we consider the family of FBNetV3 models baselines and seed models for further pruning. For each pair of models X, Y in the family, we calculate the amount of sparsity required to prune the larger model X model that consumes the same number of FLOPs the target smaller model Y, via Equation [ There are 21 potential seed and target model pairs, however we conduct pruning experiments only for a depth of for tractability. For example, given FBNetV3E. as the seed, we only prune it to FLOOPs targets corresponding to FBNetV3D and FBNetV3C. Table presents the accuracy and number of parameters of the pruned models each target FLOPs level. The improvement in performance is apparent even at lower FLOPs targets, where we might expect baseline models such as FBNetV3A to not be over over-parameeteeee porr-raamateeizedd. For example, pruning F3C to target of 356.6 MFLOPs obtains network which is 1.43% better than FBNetV 33. Figure plots the Top-1 ImageNet testing accuracy vs. FLOPs for the best pruned models as seen from Table I This clearly shows that pruning FBNetV3 models with minimal fine-tuning
significantly improve
on the state-of-the-art for FL.OPs vs. Top-1 accuracy trade-off. This analysis is performed for both uniform layer-wise and global magnitude-based prune with fine-tune settings. Global pruning ranks the weights of the entire network in contrast to uniform layer-wise pruning, which ranks each layer's s weights to determine the sparsity mask. It would be expected that global pruning performs better than uniform pruning for the same target sparsity level or number of non-sparse parameters. However in our experiments we determine the pruning threshold based on FLOPs targets, and find global pruning to require higher sparsity levels, which results in uniform pruning outperforming global pruning in Top-1 ImageNet accuracy most cases.
4.2 PRUNING COMPLEXITY
In addition to demonstrating the improvement over state-of-the-art obtained by pruning FBNetV3 models, is also important quantify the reduction in computational complexity obtained pruning larger FBNetV3 model compared to training an FBNetV3 model directly through NAS (Network Architecture Search). RQ2 (pruning for efficient model search) asks if the pruning and subsequent fine-tuning approach in Section 4. is faster than full-fledged neural architecture search. During pruning and subsequent fine-tuning, we train the pruned networks till the validation loss converges to within pre-specified tolerance, as described in Section 3.2 The time needed is generally less than when training the original FBNetV3 models, which runs for 400 epochs. The number of GPU-hours computed (number of training GPU nodes) (number of GPUs per node) * (training time

---

Under review as a conference paper at ICLR 2022
F>E
F-D
E->C
D->B
D
C->A
8
Premifesell eaCeN)
FLOP-0M
Figure 1: FLOPs vs. performance (ImageNet Top-1 acc.) for different pruned FBNetV3 networks. For comparison, the existing FBNetV3 networks are also shown here.
Table 2: Computation speedup in term of GPU-hours when comparing NAS (neural Architecture Search) with pruning and fine-tuning approaches. The selected seed networks are drawn from those in Table I with the best performance at target FLOPs.
GPU-hours in pruning and fine-tuning
Target FLOPs (FBNetV3 Model)
GPU-hours in NAS
Computational cost speedup
356.6 (FBNetV3A)
10.7k
2.240k
4.77
557.0 (FBNetV3C)
10.7k
2.496k
4.28
762.0 (FBNeIV3E)
10.7k
3.456k
3.09
convergence) for each network. In Table for each of the best performing uniformly-prunce models in Section [ we report the number of GPU-hours consumed by the prune and fine-tune strategy, along with the GPU-hours consumed when obtaining FBNetV3 model through architecture search using the method described in [Dai et al. (2020). The results are quite conclusive- we not only obtain pruned models superior in performance to the original neural search optimized models, but also as described in Section D computational cost is significantly lower when starting from pre-trained model with higher FLOPs. Given the performance improvements obtained with lower computational resources, this approach is beneficial for an experimental setting where researchers have access to open- sourced pre-trained models and limited GPU resources, for example academic environment. We observe that the degree of speedup reduces as the network size gets bigger (e.g., in FBNetV3A vs. FBNetV3C) due higher training time to convergence. Nevertheless, we still obtain speedup of 3-5 times compared to full NAS (Neural Architecture Search).
small startup or an
4.3 LATENCY EXPERIMENTS
We also measure the latency- performacee tradeoff for the pruned FBNetV3G models. FBNeIV3G is the largest model in the family and so is expected to have the best generalization performance under high sparsity levels. As described in Section 3.3 we prune the network using block sparsity (where the block size is 1 x 4) to sparsity levels in the set (40%, 50%, 60%). We have not utilized lower sparsity levels, as we have observed that for the selected kernels we need at least 40% sparsity yield any significant latency benefits. We have pruned all I Ix1 convolution layers uniformly here and subsequently converted them to fully-connected layers for compatibility with the quantized sparse kernels. In Figure 2a we present the Top-1 ImageNet accuracy vs. latency curve
7

---

Under review as a conference paper at ICLR 2022
(a) Latency vs. Top-1 accuracy on ImageNet
(b) Layer-wise sparsity pattern FBNe(V3E
Top 1 Accuracy (ImageNeg
60
8
20
PW
PWL
Type conv layer
(c) Layer-wise FLOPs distribution for 3BNetV3B
(d) Performance distribution for layer type
Figure 2: Latency benchmarking FBNetV3G for different sparsity levels f40%, 50%, 60%) and layer-wise spansity/FLLOPPsaccrnaa sensitivity for a pruned FBNetV3E network.
after pruning the 3BNetV3G network for different sparsity levels. The pruned FBNeVV3G models show marked performance reduction with lower latency as expected, with a sparsity level 60% translating to around 7% absolute accuracy reduction with a latency reduction of 18 ms (16% relative). While the xx convolution layers account for >80% of FL.OPs, they only constitute 25% of overall network latency. This is consistent with previous literature (Dudziak et al. (020) which shows that computational complexity (ex. FLOPs) and latency are not well-correlated, and indeed the latter more dependent on layer shapes. This result underscores the need to develop more latency-friendly pruning techniques which can potentially improve on the state-of-the-art in this domain.
4.4 INSIGHTS INTO PRUNING EXPERIMENTS
Our pruning experiments demonstrate that we can improve the state-of-the-art FBNetV3 models in generalization performance for a given FLOPs level. In this subsection, we obtain an insight into (1) the sparsity pattern under global magnitude- brasetuuee-beedd pruning and (2) the sensitivity of each layer when pruned in isolation under uniform layer-wise magnitude pruming (sparsity level of 95%). For (1) in Figure 2b plot the amount of sparsity obtained per 1 x convolution layer. The model being considered an FBNetV3E network pruned to a sparsity level of 43.6%, to the same FL.OPs level as FBNetV3C and subsequently fine-tuned. We note that the sparsity level in lower layers is lower which is potentially required for maintaining the performance. Higher sparsity can be admitted in upper layers of the network where has learnt more redundant representations. SE (Squeeze and Excitation) 1x] 1 convolution layers generally tend to get pruned more compared to other layers, with the sparsity being >99% for two such SE layers in stage fí 0. This indicates that we can also consider revisiting SE layer role in F3NetV3 networks, and even remove entire layers in future work to yield additional latency and FL.OPs benefits.
For analysis (2) we prune each 1 Xx] 1 convolution layer in isolation at sparsity target of 95% and record the Top-1 test accuracy obtained on ImageNet dataset. For each type of layer, PWrexpansion, PWL: bottleneck, SE: Sxxitti-n we plot the distribution of accuracies in Figure 2d We observe that the PW and PWL layers are most sensitive to high sparsity, while SE layers are able to retain performance adequately. We could also avoid pruning the most sensitive layers (appearing as outliers in the figure) to maintain generalization performance. This observation corroborates findings from analysis (1), and motivates us revisit the role of squeeze-excitation layers in future work.
8

---

Under review as a conference paper at ICLR 2022
5 CONCLUSIONS
this paper, we have investigated the problem of improving on the current state-of-the-ar FLOPs vs. performance trade-off for FBNets which have been pre-optimized by NAS (Neural Architecture Search). We have employed network pruning techniques, and our results demonstrate that we can further improve on performance over FBNetV3 at given FLOPs target through global as well as uniform magnitude-based pruning. This happens not only for relatively over-parameterized networks such FBNetV3G. but also smaller networks such as FBNetV3A which have lower computational complexity. On average, the GPU-hours incurred during pruning is about ~ 4x less than that consumed by full-scale NAS. We have also performed latency measurements on the FBNetV3G model and conducted an analysis to understand the sparsity patterns and sensitivity of different FBNetV3 layers to pruning. For future work, we plan investigate squeeze-excitation layers in more detail, and explore structured pruning approaches such as channel and layer pruning to further improve on the latency-performant tradeoff for this family models.
REFERENCES
Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik. Knapsack pruning with inner distillation. arXiv preprint wxXiv:2002.082555 2020.
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks, ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-18. 2017.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprini arXiv:2003.00033, 2020.
Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, et al. Chamnet: Towards efficient network design through platform-aware model adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11398-11407, 2019.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, al. Fbnetv3: Joint architecture-recipe search using neural acquisition function. arXiv preprint wxXiv:2006.02049 2020.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using predic- tor pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (6276-16285, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Lei Deng, Guoqi Li. Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Lukasz. Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-based nas using gens. arXiv preprint xxXiv:2007.08668 2020.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey. J. Mach. Learn. Res., 20(55):1-21, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural 2018.
networks. arXiv preprint arXiv:
David E Goldberg and Kalyanmoy Deb. A comparative analysis of selection schemes used in genetic algorithms. In Foundations of genetic algorithms, volume 1, pp. 69-93. Elsevier, 1991.

---

Under review as a conference paper at ICLR 2022
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):189-18111) 2021.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections
for efficient neural networks. In Proceedings of the 28th International Conference Neural Information Processing Systems-Volume pp. 1135-1143, 2015.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /00x:://00404861,, 2017.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint arXiv: 1912::911:2 1912,9997, 2019.
Yann LeCun, John S Denker, and Sara A Solla, Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint arXiv: /80200668. 2018
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Massively multilingual asr: 50 languages, model, 1 billion parameters. arXiv preprint arXiv:2007.03001, 2020.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprini urXiv:2003.02388 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Jian-CChiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 45200, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. 2014.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14. PMLR, 2019.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12965-12974, 2020.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2078-2087, 2020.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiablee neural architecture search. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. Balanced sparsity for efficient dnn inference on gpu. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5676-5683, 2019.
10

---

Under review as a conference paper at ICLR 2022
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernt.. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1999-2008 2020.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.0157,, 2016.

---

Under review as a conference paper at ICLR 2022
Table Baseline FBNetV3 models chosen for our experiments
No. of parameters (in millions)
Top-1 Accuracy (ImageNet)
Top-5 Accuracy (ImageNet)
Baseline
MFLOPs
3B
8.5
356.6
79.6
94.7
3B
8.5
461.6
80.2
94.9
FBNetV3C
9.9
557.0
80.8
95.3
FBNetV3D
I0.2
644.4
81.0
95.4
FBNeLV3E
10.7
762.0
81.3
95.5
FBNetV3F
13.8
1181.6
82.5
95.9
FBNetV3G
16.5
2129.7
83.2
96.3
A APPENDIX
A.I DATASET
Our pruning experiments are conducted on the ImageNet dataset, which is commonly used in the literature to evaluate performance of image classification models. is a collection of millions of images, where there is defined taxonomy based the WordNet hierarchy. The taxonomy comprises of approximately 22,000 visual subcategories, making this a large-scale classification problem. ImageNet was first introduced by Deng al. (2009) and has been adopted by the machine learning and computer vision communities to benchmark image classification models. We use the entire dataset consisting of 14 million images for experiments, and we utilize both the training set and the validation set. We split the training set to also create a smaller validation set (of 50,000 images evenly distributed across all image categories) for parameter tuning and setting the training convergence criterion. The ImageNet validation set is used in our experiments as an testing set for reporting model generalization performance.
A.2 ÖBTAINING SPARSITY LEVELS FOR GLOBAL MAGNITUDE BASED PRUNING
While the sparsity level required to prune the 1x 1 convolution layers can be obtained from Equation I for the uniform pruning case, for global magnitude- based pruning all weights in such layers are ranked globally and then thresholded determine the sparsity mask. There is closed form expression we can use to determine the sparsity level given the FLOPs target for this scenario. To obtain the sparsity level, have used the pre-trained seed model and pruned it to sparsity level We can calculate the overall FLOPss consumed due to sparsity 8 by plugging in the layer-wise shapes and sparsity levels. Extrapolating backwards from target FLOPs, we can easily find out which sparsity level corresponds to this. It is important to note that since we prune and fine-tune with a fixed sparsity mask, the number of FLOPs estimated sparsity level s does not change even after the network is fine-tuned.
12


Reviews:
[{"review_id": "p790pHodnnw", "reviewer": "Reviewer_gdjt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This paper is the among the first if not the first to demonstrate that FBNetv3 (a state-of-the-art compact network architecture searched with a sophisticated NAS method) can be pruned to achieve better computation-accuracy performance, than searching for the architecture at the same complexity. It also shows that pruning the larger FBNetv3 model gets to a higher accuracy more quickly than running architecture search.\n- The experimental results are good and they show that at different FLOP points, pruned FBNetv3 models always perform better than their original searched counterparts. \n\nWeaknesses\n- The paper has limited novelty and does not present any ideas on its own in terms of the pruning method. It merely uses some existing pruning methods on the FBNetv3 architecture in a straightforward manner.\n- It fails to make experimental comparison to NAS methods that jointly search and prune, such as the cited paper `(Wang et al., 2020)`. \n- Only FBNetv3 is evaluated in the paper. How about other compact NAS architectures? It seems like the authors tried with other architectures but find that they do not work well with pruning. It could be that the claims in the paper are only narrowly applicable to FBNetv3.\n", "summary_of_the_paper": "This paper describes the details on how the authors prune FBNetV3 (which is already a compact network architecture) models with existing methods to make them more compact. The resulting models have better FLOP-accuracy tradeoffs than the original FBNetV3.", "summary_of_the_review": "This paper has some good empirical contributions but those are all it has to offer. For a paper like this, I wish to see a stronger version of the paper where pruning is generally applicable to all sorts of compact NAS architectures.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UmeS_ecRxlJ", "reviewer": "Reviewer_tgou", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1) The paper is generally well-written and easy to follow. \n2) The authors provide valuable evaluations to show that pruning can be useful even for optimized network architectures, i.e. those searched by NAS.\n\nCons:\n1) The technical contribution and novelty are limited. This paper only applied previous pruning methods to the network obtained by NAS and made a series of evaluations and comparisons. Technically, this paper didn't make novel contributions. And considering the experimental evaluations, the conclusions are not surprising to me, i.e. they didn't indicate or inspire new scientific problems. So generally, I think the contributions are quite limited. \n\n2) There are many typos in Table 1. The gains shown in Table 1 seem incorrect. It is strange that the shown numbers are not equal to (accuracy of pruned models - accuracy of baselines). And in the last line of Table 1, the baseline accuracy seems incorrect, i.e. it should be 82.5. \n\n3) Some details are not clear. For example, in Fig. 1, what are the types of baseline models and corresponding seed networks? For Table 2, what are the selected seed networks? Are the selected seed networks the same in Table 2? \n\n4) The evaluation is only based on FBNetV3 architecture. I just doubt the generality of the conclusions. ", "summary_of_the_paper": "This paper applies conventional pruning-and-finetuning techniques to further compress the networks searched by NAS. The experiments and evaluations are based on the family of FBNetV3. The authors show that by pruning large FBNetV3 model to small one, the accuracy of pruned model may be slightly better than the original target (small) model, achieving better tradeoff between computational complexity and accuracy. Also, the authors show that pruning is more training-efficient than searching by NAS to achieve a compact FBNetV3 model. ", "summary_of_the_review": "I made my recommendations mainly considering the limited contribution and limited practical significance made by this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "U0p0aP6qjQ", "reviewer": "Reviewer_F7aS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and clear in most aspects. Applying pruning techniques to recent architectures, especially architectures that are designed to be resource-efficient, is a task that is rarely addressed in the literature. The two main contributions as stated in the paper summary above are novel.\n\nMy main concern is about the practical relevance of the presented work. The paper is mainly an experimental contribution that shows the potential of reducing the number of FLOPS. However, reducing the number of FLOPS appears to be mostly of theoretical relevance. It is also mentioned in the paper (end of Section 4.3) that FLOPS are not well correlated with latency. I wonder if FLOPS can then be seen as a good proxy for computational footprint and power consumption as mentioned in Section 3.3. From what I know, sparsity in matrices gives a decent practical improvement only at high levels of sparsity. Special sparse-matrix structures typically store positions and values of non-zero entries and, therefore, for small amounts of sparsity the memory footprint even increases when using sparse-matrix formats. This is also supported by Section 4.3 where it is reported that sparsities below 40% do not provide a benefit in terms of latency. Consequently, I do not see a real benefit of the achieved sparsity levels in Table 1 which are often below 40%. In this view, I believe that comparing a small dense matrix with larger sparse matrices having the same number of non-zero entries is unfair. In summary, I doubt that the proposed method of pruning a larger model is \"practically advantageous\" as promised in the introduction.\n\nThe paper only shows experiments where pruning works reasonably well. I am missing experiments where we can see that the pruning breaks down, i.e., how far can we go until the accuracy degrades drastically on some specific model. This is important to study the impact on pruning for architectures that are inherently resource-efficient and to understand how \"overparameterized\" these models really are.\n\nThe second main contribution states that pruning a larger model is faster than finding a smaller model directly by means of NAS. Although the reduction in training time compared with NAS is substantial (approx. 4x), the process still seems to take around 2000--3500 GPU hours. I am not sure whether this amount of GPU hours can be considered as \"can be obtained inexpensively with limited resources that are generally available to a machine learning practitioner\" as mentioned in the introduction. What hardware was used for these experiments and for the latency experiments.\n\nMinor:\nIt appears that F_{1x1} in Eq. (1) is the number of FLOPS of *all* 1x1 layers of the whole network. Please emphasize the \"all\".\n", "summary_of_the_paper": "The paper presents an experimental evaluation of simple pruning techniques applied to modern architectures that are designed to be inherently resource-efficient. It is shown that pruning large models in the FBNetV3 family achieves better accuracy-FLOPS trade-offs than smaller models in the FBNetV3 family. It is also shown that pruning large models is faster than performing NAS to find the smaller models directly.", "summary_of_the_review": "The paper is largely experimental and, in my opinion, shows improvements in FLOPS that are not practically relevant. However, the paper promises a few times that the proposed methods provide practical benefits. I am not sure, what to take away of the paper: in my opinion there are hardly any practical benefits and there is no clear takeaway message from the theoretically relevant decrease in FLOPS.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H_ipPyLeLpd", "reviewer": "Reviewer_haMF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper is primary focused on empirical results. Applied technique, magnitude pruning, is a well known techniques. \n\nTwo research questions that paper tries to answer are interesting, but have been answered multiple times. \n\nIt is known that pruning a larger model to the level of smaller will give a more accurate model. Researchers have been showing this by pruning Resnets from 101 to 50 to 34 to 18 etc. \n\nPruning is more efficient to get a model derivative than running a new NAS targeted to a smaller model, also models like EfficientNet provide a scaling rule that doesn't require to run NAS again for a different computational budget.\n\nFor an overview of recent work in pruning please refer to [R1]. MobileNet and EfficientNet are popular model to benchmark a pruning technique. For this kind of study, it is highly recommended to apply multiple pruning techniques to get technique agnostic insights.\n\nR1) Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, John V. Guttag, What is the State of Neural Network Pruning? MLSys 2020", "summary_of_the_paper": "Paper focuses on pruning 1x1 convolutions in FBNet with existing technique (magnitude pruning). Authors tend to answer 2 questions: a) how a larger pruned model compare to smaller model trained from scratch; b) is pruning an efficient way to get NAS model derivatives.  ", "summary_of_the_review": "Paper presents an empirical study on a well studied problem. The setup, method and results are not significant and are well known to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study which shows that pruning FBNets with larger capacity results in a model with higher accuracy than one searched via neural architecture search. The below are pros and cons of the paper mentioned by the reviewers:

Pros
- The observation that optimized architectures such as FBNets can benefit from pruning is interesting.
- The paper is well-written and easy to follow.

Cons
- It is trivially known that training larger model and then pruning it will yield a better performing model, than training a smaller model from scratch.  
- The authors do not propose a novel pruning technique for optimized CNN architectures, and use existing pruning techniques for all experiments. 
- The experimental validation is only done with FBNets on ImageNet, and it does not show when pruning starts to break down. 

All reviewers unanimously voted for rejection, especially since the main “findings” of this paper that compact architectures can be further pruned down for improved accuracy/efficiency tradeoff, and that pruning a larger compact model results in models that outperform smaller models trained from scratch, have been already shown in many of the previous works on neural pruning. In fact, compact networks such as MobileNets and EfficientNets are the standard architectures for measuring the effectiveness of pruning techniques, and thus the contribution of this work reduces down to showing that the same results can be obtained with FBNets. This could be of interest to some practitioners, but is definitely not sufficient to warrant publication.
Decision: Reject