ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
Authors: Rishabh Tiwari
Udbhav Bamba
Arnav Chavan
Deepak Gupta
Venue: ICLR 2021 Poster (2021)
Keywords: Structured Pruning
Budget-Aware Pruning
Budget constraints
Sparsity Learning

Abstract: Structured pruning methods are among the effective strategies for extracting small resource-efficient convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network. Our choice of continuous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible and expected as final outcomes. Our approach's flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.

Published as a conference paper at ICLR 2021
CHIPNET: BUDGET-AWARE PRUNING WITH HEAVISIDE CONTINUOUS APPROXIMATIONS
Rishabh Towari Udbhav Bambalt, Arnav Chavan and Deepak K. Gupta
TTransmute AI Research, The Netherlands Indian Institute Technology, ISM Dhanbad, India
"Informatics Institute, University of Amsterdam, The Netherlands
ABSTRACT
Structured pruning methods are among the effective strategies for extracting small essource-efficien convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requir- ing model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and novel crispness loss to identify highly sparse network out of an existing dense network. Our choice of continu- ous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or are practically feasible and expected as final outcomes. Our approach's 's flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms sate-offtheeart structured pruning meth- ods by remarkable margins of up to 16. % in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on feature- rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.
INTRODUCTION
Convolution Neural Networks (CNNs) have resulted in several breakthroughs across various disci- plines of deep learning, especially for their effectiveness in extracting complex features. However, these models demand significantly high computational power, making it hard to use them on low- memory hardware platforms that require high-inference speed. Morcover, most of the existing deep networks are heavily pherpparaeeterize resulting in high memory footprint (Denil et al., 2013) Frankle & Carbin] 2018). Several strategies have been proposed to tackle this issue, that include network pruning (Liu et al. 2018), neural architecture search using methods such as reinforcement learning (Jaafra et al. 2019 and vector quantization (Gong et al. 2014), among others.
Among the methods outlined above, network pruning has proved to be very effective in designing small resource-efficient architectures that perform at par with their dense counterparts. Network pruning refers to removal of unnecessary weights or filters from given architecture without com- promising its accuracy. It can broadly be classified into two categories: unstructured pruning and structured pruning. Unstructured pruning involves removal of neurons the corresponding con- nection weights from the network to make it sparse. While this strategy reduces the number of parameters in the model, computational requirements are still the same (Li et al. 2017). Structured pruning methods on the other hand remove the entire channels from the network. This strategy pre-
All authors contributed equally.
Code is publicly available http:///ghhhhh.commmaasmmseeettt

---

Published as a conference paper at ICLR 2021
serves the regular structure, thereby taking advantage of the high degree of parallelism provided by modern hardware (Liu et al. 2017 Gordon et al. (2018).
Several structured pruning approaches have been proposed in the recent literature. A general con- sensus is that variational approaches using sparsity prior loss and learnable dropout parameters out- perform the deterministic methods (Lemaire et al. 2019). Some of these methods learn sparsity as a part of pretraining. and have proved to perform better than the three stage cettrain-runnee finetune methods. However, since such approaches need train the model from scratch with pruning-related variables embedded into the network, they cannot benefit from off-the-shelf pretrained weights (Liu al. 2017; Alvarez & Salzmann| 2017). Others require choosing hyperparameters based on the choice of the network, and cannot be easily adapted for new models (Gordon et al. 2018). Fur- ther, with most of these methods, controlled pruning cannot be performed, and a resource-usage constraint can only be satisfied through trial-and-error approach. Recently, [emairre et al. (2019) presented budget-aware pruning method that includes the budget constraint as part of the train- ing process. A major drawback of this approach and other recent methods is that they are unstable for very low resource budgets, and require additional tricks to work. Overall, a robust budget-aware pruning approach that can be coupled with different budget constraints as well as maintains stability for very low target budgets, is still missing in the existing literature.
In this paper, we present ChipNer, a deterministic strategy for structured pruning that employs con- tinuous Heaviside function and crispness loss to identify a highly sparse network out of an existing pretrained dense network. The abbreviation 'ChipNet' stands for Continuous Heaviside Pruning of Networks. Our pruning strategy draws inspiration from the field of design optimization, where the material distribution task is posed as continuous optimization problem, but only discrete values (0 or 1) are practically feasible. Thus, only such values are produced as final outcomes through con- tinuous Heaviside projections. We use a similar strategy to obtain the masks in our sparsity learning approach. The flexible design of ChipNet facilitates its use with different choices of budget con- straints, such as restriction on the maximum number of parameters, FLOOPs, channels or the volume of activations in the network. Through experiments, we show that ChipNet consistently outperforms statc-of-ther oo-tth---tt pruning methods for different choices of budget constraints.
ChipNet is stable for even very low resource budgets, and we demonstrate this through experiments where network is pruned to as low as I 1% of its parameters. We show that for such extreme cases, ChipNet outperforms the respective baselines by remarkable margins, with difference in accuracy of slightly beyond 16% observed for one of the experiments. The masks learnt by ChipNet are transferable across datasets. We show that for certain cases, masks transferred from model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning the student data itself.
2 RELATED WORK
As has been stated in the hypothesis by Frankle & Carbin (2018), most neural networks are over- parameterized with a large portion (as much as 90%) of the weights being of little significance to the output of the model. Clearly, there exists enormous scope reduce the size of these networks. Several works have explored the efficiency of network pruning strategies for reducing storage re- quirements of these networks and accelerating inference speed (LeCun et al. 1990; Dong et al.) 2017). Some early works by Han et al. (2015a|c); Zhu & Guptal (2017) involve removal of indi- vidual neurons from a network make sparse. This reduces the storage requirements of these networks, however, no improvement in inference speed is observed. Recently, several works have focused on structured network pruning, as involves pruning the entire channel/filters or even layers to maintain the regular structure (Luo et al. 2017: Li et al. 2017 Alvarez & Salzmann 2016).
The focus of this paper is on structured network pruning, thus, we briefly discuss here the recent works related to this approach. The recent work by Li et (2017) identifies less important channels based on Ll-norm. [uo et al. (0177); He et al./ (2017) perform channel selection based on their influence on the activation values of the next layer. Liu et al. (2017) perform channel-level pruning by imposing LASSO regularization on the scaling terms in the batchnorm layers, and prune the model based each layer with reinforcement learning [ouizos et al, (2017): Alvarez & Salzmann (2017) 2016) train and prune the network in single stage strategy.
global threshold. He et al. (2018b) automatically learn the compression ratio of
2

---

Published as a conference paper at ICLR 2021
(c)
08-
poojattions
0,3
0.4
04
0.4
02
0.2
0.2
Figure I: Representation of the different functions used in ChipNet for various choices of 3 and 7- The plots show (a) logistic curves, (b) continuous Heaviside functions, (c) the outputs of logistic and Heaviside functions shown together for | = 2.0 and 4.0, and (d) the crispness loss function.
The above mentioned approaches cannot optimize networks for pre-defined budget constraint. Adding budget constraint to the pruning process can provide a direct control on the size of the pruned network. For example, Morphnet imposes this budget by iteratively shrinking and expanding a network through a sparsifying regularizer and uniform layer wise width multiplier, respectively, and is adaptable to specific resource constraints (Gordon al. 2018). However, it requires model- specific hy hyperparameter grid search for choosing the regularization factor. Another approach is BAR (Lemaire et al.) 2019) that uses budget-constrained pruning approach based on variational method. A limitation of this approach is that for low resource budgets, it needs to explicitly ensure that at least one channel is active the downsample layer to avoid fatal pruning. The approach proposed in this paper does not require any such tweaking, and is stable for even very low resource budgets.
3 PROPOSED APPROACH
3.1 LEARNING SPARSITY MASKS
Sparsity learning forms the core of our approach. It refers to learning set of sparsity masks for dense convolutional neural network (parent). When designing the smaller pruned network (child), these masks identify the parts of the parent that describe here the general idea of learning these masks in the context of our method.
be included in the child network. We first
The proposed approach falls in the category of structured pruning where masks are designed for channels and not individual neurons. Let TRd - R² denote a convolutional neural network with weights W Rm and set of hidden channels H RR. We define IRP set of sparsity masks, where 2 E z refers to the mask associated with the feature map h, E H. To apply the mask, z4 is multiplied with all the entries of h; The optimization problem can further be stated
min L(f(z @ H(W);x),)) s.t. V(z) Vo. W.
(I)
where @ denotes elementwise multiplication and y E DD are data samples used to train the net- work f. The desired sparsity of the network is defined in terms of the equality constraint, where V(-)
denotes the budget function and Vo is the maximum permissible budget. Our proposed formulation of pruming is independent from the choice of the budget function. We later show this through exper- iments with volume budget as in Lemaire et al. (2019), channel budget similar to and budget defined in terms of parameters and FLOPs as well.
Liu et al. (017))
Originally, E would be defined such that E (0, 1). and a discrete optimization problem is to be solved. For the sake of using gradient-bant based methods, we convert it to a continuous optimization problem, such that E [0, 1]. Such reformulation would lead to intermediate values of occurring in the final optimized solution. Any intermediate value of 2, for example = 0.4, would imply that fraction of the respective channel is to be used, and clearly such solution is practically infeasible. We propose to overcome this challenge through the use of simple nonlinear projections and a novel loss term, and these discussed in detail in the next section.
3

---

Published as a conference paper at ICLR 2021
3.2 CONTINUOUS HEAVISIDE APPROXIMATION AND LOGISTIC CURVES
At the backbone of our pruning strategy lies three important functions: the commonly used logistic curves, continuous Heaviside function and crispness loss term. Figure presents Figure[[reseetss a graphical
representation of these functions. We further provide below a brief motivation for the choice of these functions as well as their significance in our pruning approach.
Logistic curves. A commonly used function for adding nonlinearity to a neural network (LeCun al., 1998), logistic curve projects an input from the real space to a range of 0-1 (Figure a and can be mathematicllly stated as
(2)
where d denotes the optimization parameter corresponding to the mask vo is the midpoint of the curve, and ă denotes the resultant intermediate projection. The additional parameter 3 is used to control the growth rate of the curve, and forms an important ingredient of our approach. While low values of 3 can produce an approximately linear curve between -1 and 1, higher values turn it into step function. During the initial stages of training, we propose keep | very low, and increase it to higher values at later stages of the optimization process. With increased values of B, the values further from 0.5 are made more favorable for ž.
In our experience, the logistic curve alone cannot be used to obtain approximately discrete (0-1) solutions for cannot sufficiently penalize the intermediate values between and I, and optimization algorithm can easily identify values of v for which the projected values are far from both. An example experiment demonstrating this issue is presented in Appendix C.2 To circumvent this issue, we add another nonlinear projection using a continuous approximation of the Heaviside function.
continuous optimization scheme. The nonlinearity introduced by this function
Continuous Heaviside function. A continuous approximation to the Heaviside step function, re- ferred as continuous Heaviside function this paper, a commonly used projection strategy to solve continuous versions of binary optimization (0-1) problems in the domain of design optimiza- tion Guest et al. 2004 1) The generalized form of this function can be stated as:
== e - + 2e=
where, the parameter y dictates the curvature of the regularization. Figure [igureIb]showe [b]shows the continuous Heaviside function for several values of y. We see that is linear for ? and approaches the Heaviside step function for very large values of
The advantages of our projection function are twofold. First, during projection, the values close to O and are not affected irrespective of the choice of This implies that the masks identified with most confidence in the early stage of training are not directly impacted by the continuation applied on the value of 7, thus helping in the convergence of the training process. Here, 'continuation' refers to slowly adapting the value of 7 during the course of training. Second, even the values of which are slightly greater than 0 are also nonlinearly projected close 1, and this effect is more prominent for larger values of 7. The projection adds higher penalty on values between 0 and 1, and makes them extremely unfavorable when higher values of 7 are chosen.
While the continuous Heaviside function helps to obtain approximately discrete masks, there is still no explicit constraint or penalty function that can regulate this. To overcome this problem, we tie the outputs of logistic and continuous Heaviside functions to define a novel loss term, referred as crispness loss.
Crispness Loss. This novel loss term explicitly penalizes the model performance for intermediate values of z, and drives the convergence towards crisp (0-1) masks. It is defined the squared L2 norm of the difference between ž and z. stated as L. 2| - z|12. and from Figure Ic we see that L. achieves its minima when either ž = with respect to z for different values of ,3 and r is shown in Figure Id] It can be seen that for lower values of 3 and 7, the loss value is low, and the crispness function plays little to no role in driving the pruning process. When the value of y slowly increases, the peak of the graph shifts upwards as well as towards the left, thereby increasing the penalty associated with values of tb. This drives the values of &e to move farther from the origin. The left shift in the graph adds higher penalty on the negative values, forcing them to become even more negative, thus forcing the respective 2 to move closer to O.
Further, the trend of this loss function
4

---

Published as a conference paper at ICLR 2021
The additional loss function associated with the model generally favors values towards 1. For ample, cross-entropy loss used for classification would prefer to set all values z to to be able to maximize the classification accuracy. With increasing values of y forcing the masks towards O, a balance between the two is identified during the training process. The term / acts as a regular- izer that to some extent counteracts the too abrupt impact of y and regulates the convergence of the training process.
3.3 IMPOSING BUDGET CONSTRAINT
The simplicity of our pruning strategy decouples it from the choice of budget constraint. In this paper, we show the working with four different choices of budget constraints: chunnel, activation volume, parameters and FLOPs. These choices are inspired from some of the recent state-of-the-a
methods from the existing literature (Liu et al. 2017 Lemaire et al. 2019).
For budget calculation, values of the masks z should be close to 0 or 1. However, during the initial iterations of training, masks would contain intermediate values well. This makes it difficult to accurately calculate the budget for the constraint specified in Eq. [T Thus, rather than computing directly over the masks these are computed on where 2; E Z obtained by applying logistic projection on z with 100 = 0.5 (Eq. 2). Further discussion related to it is provided in Appendix CC.3
The budget constraint is imposed using loss term La, referred as budget loss. We define the budget loss as La = (v(z)) - Vo) where (-)) can be one of the budget functions described below.
Channel budget. It refers to the maximum number of hidden channels h that can be used across all convolutional layers of the network. Mathematically, it can be stated as y(e) = =>>> 2.) where p denotes the number of hidden channels in the network. Constraint on the channel budget limits the number of channels, and thus the number of weights in the network.
Volume budget. This budget controls the size of the activations, thereby imposing an upper limit the memory requirement for the inference step. We define volume budget y(v) (E, E A, ( A5. Pi), where (h) denotes the number of convolutional layers, and A₁ and pj denote area of the feature maps and their count, respectively, in the 5th layer.
Parameter budget. This budget directly controls the total number parameters in the network, and can thus be used to impose an upper limit on the size of the model. For details, see Appendix A.
FLOPs budget. This budget can be directly used to control the computational requirement of the model. Mathematical formulae to calculate it is stated in Appendix A.1
3.4 SOFT AND HARD PRUNING
The pruning stage in our approach comprises two steps: soft pruning and hard pruning. After a deep dense network has been pretrained, masks added to the network and soft prun- ing is performed. The steps involved in soft pruning are stated in Algorithm [ During this stage, the network is optimized with the joint loss L= Lee + 02 C. + azZss, where an and a2 are the weights for the crispness and budget loss terms, respectively.
Algorithm 1: ChipNet Pruning Approach
Input pretrained network weights W;
budget constraint function V(-); budget value Vo: training data D;
pruning iterations N
Output: learnt sparsity masks z
wh E V/ 4- random initialization
for k = 1...N do
(x,y) + sample(D)
<- (OGISTIC())
After every epoch of soft pruning, the perfor- mance of the network is evaluated in hard pruned manner. For this purpose, masks are used, and a cutoff is chosen using binary search such that the budget constraint is exactly satisfied. Values above this cutoff are converted to and the ones lower turned to O. Finally, the model with best performance on the validation set is chosen for fine tuning.
Z <- CUNTINUOUSHEVVISDEEEE
ý <- Forward(x, W.z) 2) VZ))
L. <- CHIPNETLOSS(V, Vo. 2, 2.2. (VW, Ve) Backward(())
(W, -
Sttimizeetep(VVW Vv)
end
5

---

Published as a conference paper at ICLR 2021
CIFAR-10
CIEAR-100
80
71
84
Jost Accuracy (4)
70
Accurzy (1g
Accorney
60
812
80
50
Oupliin
60
Chaplin
Cliplin
MerehNe
1.21
Pruning Factor
Pruning Factor
Pruning Factor
Figure 2: Performance comparison of ChipNet with different structured pruning baselines for var- ious choices of volume constraint. Here, volume pruning factor refers to the factor by which the volume budget being reduced.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We test the efficacy of our pruning strategy on several network architectures for four different choices of budget constraint functions. The architectures chosen in this study include WideResNet-26-11 (WRN-26-12) (Zagoruyko & Komodakis 2016), PreResNet- 164 (He et al. 2016b). ResNet-50 50 and ResNet-101 (He et al. 2016a). For datasets, we have chosen CIFAR-
(rizhevsky 2009) and Tiny ImageNet (Wu et al.). For the combined loss L in Eq. | weights oi and a22 are set to 10 and 30, respectively, across all experiments. Implementation details related to the pretraining, pruning and finetuming steps, as well details of the hardware are described in Appendix
4.2 RESULTS
Performance of pruned sparse networks. We present here results obtained using ChipNet for WRN-26-12 12 and MeRReNNet1164 pruned with volume and channel constraints, respectively. For the two other constraints, parameters and FLOPs budget, we perform comparative study later this paper.
Volume budget. Figure shows the comparison of performance values for WRN-26-12 for CIFAR-10, CIFAR-100 and TinylmageNet datasets, when pruned using ChipNet. We compare our results with BAR (Lemaire et al. 2019), MorphNet (Gordon et al.] 2018) and L.RR (Louizos) (et al.,|2200 2017) for volume pruning factors of 2, 4, 8 and 16. Details related to the three baselines presented in Appendix D.I] ChipNet consistently outperforms all the basclines across all datasets and for all choices of the budget. For the case of extreme pruning of 16 folds CIFAR-100, the performance of BAR is close to ours, while the other two baselines significantly underperform.
Channel budget We study here the pruning efficacy of ChipNet coupled with channel constraint on PreResNet- 164 architecture for CIFAR-10 10 and CIFAR-100 datasets. Results are compared with the network slimming approach (Liu et al. 2017), implementation details related which can be found in Appendix D.I As constraints, we use channel budgets of 60%, 40%, 20% and 10%.
Table presents the results for different choices of channel budgets. We also report the number of parameters in the pruned network as well as the associated FLOPs. It is seen that ChipNet outperforms the baseline method for all the experimental settings. For CIFAR-IO in particular, we see that for even very low channel budget of 10%, accuracy of the pruned network drops by only 3.1%. For 10% channel budget, our method outperforms the network slimming strategy on CIFAR- 10 and CIFAR-100 by remarkable margins of 8.5% and 16.1%, respectively.
Note that lower channel usage does not necessarily imply lower number of parameters or reduced FLOPS in the pruned network, and we analyze this for the various cases of pruning considered in Table We see that ChipNet performs selection channels in a more optimized way, such that better accuracy is achieved with fewer parameters. In terms of FLOPS, both methods perform at par. Although, the FLOPS for ChipNet are slightly higher for the channel budget 10%, this overhead insignificant compared to the gain in accuracy and reduction of parameters. Overall, we infer that ChipNet couples well with the channel constraint, and stable for even extreme pruning cases of as low 10% channel budget.

---

Published as a conference paper at ICLR 2021
Table 1: Performance scores for pruning PreResNet-164 architecture on CIFAR-IO and CIFAR- 100 datasets for Network Slimming and ChipNet (ours). The number of parameters and FLOPs for the unpruned networks are 1.72 million and 5.03 10 respectively Here budget refers the percentage of total channels remaining. Abbreviations 'Acc.' and 'Params." refer to accuracy and number parameters, all scores are reported %, and parameters FLOPs are reported relative to those in the unpruned network.
CIFAR-10 Params. &
CIFAR-100
Method
Budget(% %
Acc.
FLOPs
Acc. f
Params. +
FLOPs +
Unpruned
94.9
100.0
100.0
77.1
100.0
100.0
Net-Slim
95.3
85.1
79.0
77.5
85.9
75.1
60
ChipNet
95.3
79.3
77.9
77.8
85.0
75.2
Net-Slim
94.9
65.4
58.9
76.6
71.9
55.4
40
ChipNet
95.0
51.7
54.7
77.3
65.8
53.1
Net-Slim
93.0
33.3
29.9
70.1
44.7
25.0
20
ChipNet
94.2
24.0
28.4
72.3
31.8
23.9
Net-Slim
87.1
19.0
15.3
51.2
19.2
11.1
10
ChipNet
91.8
13.8
16.4
67.3
14.6
12.6
Table 2: Performance scores for pruning ResNet-50 architecture on CIFAR-100 and CIFAR-10 for
BAR and ChipNet (ours) with volume budget (V) and channel budget (C). The number of parameters and FLOPS for the unpruned networks are 23.7 million and 2.45 10 respectively. Here budget refers to the percentage of total channcls/volume remaining. Abbreviations 'Ace.' and 'Param." refer accuracy and number of parameters, all scores reported in %, and parameters and FLOPs reported relative to those the unpruned network.
CIFAR-10 10
CIFAR- 100
Method
Budget (%)
Acc. t
Param,
FLOPs
Acc.
Param. 4
FLOPs 4
Unpruned
93.3
100
100
73.0
100
100
ChipNet (C)
92.8
4.5
17.7
71.1
7.3
10.9
ChipNet (V)
12.5
91.0
2.8
5.1
65.5
22.5
9.0
BAR (V)
88.4
1.8
3.8
63.8
5.2
4.2
ChipNet (C)
92.1
1.6
8.8
67.0
1.8
4.8
ChipNet (V)
6.25
83.6
1.3
2.0
54.7
14.5
5.1
BAR (V)
84.0
0.9
1.3
42.9
3.7
2.0
Effect of the choice of budget. Here, we analyze the impact of one budget type over another to under- stand whether the choice of budget really matters when pruning network. As first experiment, study side-by-side the results for channel and volume constraints when used to prune ResNet-50 on CIFAR-10 10 and CIFAR-ION 100 datasets. Results of this experiment are shown in Table 2 Note that do not intend to identify& a winner among the two, since both are meant to optimize different aspects of the network. For baseline comparison, the network is also pruned using the BAR method. The volume budget variant of ChipNet outperforms BAR by significant margin. Moreover, we see that for the same amount of volume constraint, the number of parameters used by BAR are lower than our method for most cases. A reason for significant drop in performance of BAR could be that the optimization algorithm does not fully exploit the choice channels be dropped, thereby choosing sub-optimal set and losing too many parameters from the network.
Between the results volume and channel constraints for ChipNet, at the first look, it seems that channel constraint is better throughout. However, stated above, direct comparison between the two is unfair. For example, volume constraints are meant to reduce the number of activations, and in turn would also reduce the FLOPs. This is evident from the results FLOPS reported for volume constraint are always lower than the respective channel constraint. For a better understanding of the effects of these budgets, perform another experiment for pairwise analysis of these constraints.

---

Published as a conference paper at ICLR 2021
50
60
80
Tex Ascumay (5)
73
Tox Accumay (8)
76
73
75
70
Tex
70
Carachepacd
7LA05-9400
16
16
53
125
022
64
255
Volume pranol (in folds)
Channel prined (in folds)
Pirametorn geuned (in folds)
FLOPP proned (in foldd)
Figure 3: Test accuracy versus the remaining budget for networks pruned using ChipNet with dif- ferent budget constraints.
Figure shows the pairwise plots of the budgets used prune WRN-26-12 CIFAR-100. From the first two plots, we see that the scores reported are higher for any volume budget when the net- work is optimized with volume constraint, and similarly higher for certain channel budget when the network is optimized for it. Similar observations can also be made between the number of pa- rameters and FLOPs. In nutshell, we observe that the pruned network performs best with respect to the constraint for which the masks are trained. Thus, the choice of constraint type should not be arbitrary but based on the practical applications, such as reducing FLOPs, among others.
Stability and robustmess. Our pruning strategy is also very stable, and this has already been demon- strated earlier for channel and volume pruning at low resource budgets. Compared to the baselines, networks obtained with ChipNet are found to perform significantly better even without the need for any additional tweaking such as explicitly opening certain channels to ensure network connectiv- ity (Liu et al. 2017 Lemaire et al.) 2019). Another example demonstrating the stability is volume pruning (6.25%) of ResNet-50 on CIFAR- 100, where ChipNet performs 1 11.8% better than BAR.
To account for robustness, have extensively performed hyperparameter grid search budget of 6.25% for WRN-26 12 to identify the suitable values for d1 and a:2. It has been observed that values in the neighborhood of this point do not affect the performance. Details related this grid search are further provided in Appendix C.5) Further, the same hyperparameter setting has been used for all the experiments. The consistent results across all datasets shows that ChipNet is robust.
channel
Transfer learning of masks. Inspired by knowledge distillation (Hinton al. 2015). where refined informa- tion obtained from deeper teacher network is transferred shallow student network, we study here the transfer of sparsity masks across datasets. For teacher and student, we use Tiny ImageNet and CIFAR-ION datasets, respec- tively, and ResNet-101 101 is pruned for different choices of channels budgets. Table g reports the performance scores for the pruned network are learnt on CIFAR-100 as well as when they are learnt on Tiny ImageNet and transferred. Interestingly, for mod- erate channel budgets of 40% and 60%, we see that the models using masks transferred from Tiny ImageNet per- form better than those obtained directly on CIFAR-100. This gain in performance from mask transfer could be attributed to the feature-richness of the chosen teacher dataset. We also see that for the very low budget case of 20%, masks from the student dataset outperform that from the teacher. For such low budgets, the expressive power of the model is too low to fully exploit the knowledge from the transferred masks.
Table 3: Accuracy values (%) on CIFAR-100 dataset for ResNet-101 pruned with different choices of channel budget (%) on CIFAR-100 (Base) and with masks from Tiny ImageNet (Trans- fer).
CIFAR-100 when the masks
Budget
Base Acc
Transfer Acc
20
71.3
68.3
40
71.6
72.0
60
71.8
72.1
100
73.6
CONCLUSION
We have presented ChipNet, a deterministic strategy for structured pruning of CNNs based on con- tinuous Heaviside function and crispness loss. Our approach provides the flexibility of using it with different budget constraints. Through several experiments, it has been demonstrated that ChipNet outperforms the other methods representative benchmark datasets. We have also shown that
8

---

Published as a conference paper at ICLR 2021
ChipNet generate well-peeforning pruned architectures for very low resource budgets well. To conclude, with the strongly effective pruning capability that ChipNet exhibits, can be used by the machine learning community to design efficient neural networks for variety of applications.
LIMITATIONS AND FUTURE WORK
In this paper, have explored the stability and robustness of ChipNet from various perspectives. Through experiments, we have shown that ChipNet consistently performs well across several CNN architectures and datasets. We analyzed it with respect to different choices of budget constraints, performed stability tests for even extreme scenarios of as low as 1% parameters, analyzed how the masks get distributed across the network, and even studied the transferability of masks. For all these experiments, ChipNet has proved to work well. However, before ChipNet can be considered full- proof solution for pruning, additional experiments might be needed. For example, the applicability of ChipNet is not yet explored on large datasets such as ImageNet, and we would like to explore it in our future work.
Further, it would be of interest explore how the pruned architectures obtained using ChipNet per- form for tasks beyond classification, such as segmentation and object tracking. Improving inference speed is an important aspect of object tracking, and it has not been explored from the point of view of network pruning. We would like to see if the recent object tracking algorithms that use backbones such ResNet-50 and ResNet- 101 can be made faster through our pruning method.
REFERENCES
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pp. 2270-2278, 2016.
Jose M Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. In Advances in Neural Information Processing Systems, pp. 356-867, 2017.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Pre- dicting parameters in deep learning. Advances neural information processing systems, pp- 2013.
Emily L. Denton, W. Zaremba, Joan Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. ArXiv, 4bb//1444.036, 2014.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4943-4953, 2019.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer- optimal brain surgeon. In Advances in Neural Information Processing Systems, pp. 4857-4867, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 18X3:08803.3365, 2018.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv:1412.6111 2014.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang. and Edward Choi. Mor- phnet: Fast simple resource-constrainee structure learning of deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1586-1595, 2018.
K. Guest, J. H. Prévost, and T. Belytschko. Achieving minimum length scale in topology optimiza- tion using nodal design variables and projection functions. International Journal for Numerical Methods in Engineering, 61:238-22 254, 2004.
K. Guest, A. Asadpoure, and Ha. Eliminating beta-continuation from heaviside projection and density filter algorithms. Structural and Multidisciplinary Optimization, 44:443 453, 2011.

---

Published as a conference paper at ICLR 2021
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 111.v.1510.00149, 20151
Song Han, J. Pool, John Tran, and W. Dally, Learning both weights and connections for efficient neural network. In NIPS, 2015b.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances neural information processing systems, pp. 1135-1143, 2015c.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645, Springer. 2016b.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808. 06866, 2018a.
Yihui He, X. Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1398-1406, 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration mobile devices. In Proceedings the European Conference Computer Vision (ECCV), pp. 784-800, 2018b.
Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. Distilling the knowledge in neural network. ArXiv, 0bbs/550.00553 2015.
Yesmina Jaafra, Jean Luc Laurent, Aline Deruyver, and Mohamed Saber Naceur. Reinforcement learning for neural architecture search: A review. Image and Vision Computing, 89:57-66, 2019.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11 86(11):2278-2222 1998.
C. Lemaire, A. Achkar, and P. Jodoin. Structured pruning of neural networks with budget-aware reg- ularization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9108-9116, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1rXiv:1608.08710 2016.
Hao Li, Asim Kadav, Igor Durdanovic, Samet, and Graf. Pruning filters for efficient convnets. ArXiv, abs/ 05//6688.88 00s..07110 10, 2017.
Tuanhui Li, Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convolu- tional neural networks via factorized convolutional filters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3977-3986, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference Computer Vision, 2017.
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05277 2018.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICIRR 2019.
10

---

Published as a conference paper at ICLR 2021
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv:1712.01122,22 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. 2017 IEEE International Conference Computer Vision (ICCV), pp-
5068-5076, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint 1rXiv:6611.0644 2016.
Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differen-
tiable scale-invariant sparsity measures. arXiv preprini arXiv: (at:::998..9999 2019.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07116, 2016.
Wide residual networks.
arXiv preprint
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.

---

Published as a conference paper at ICLR 2021
APPENDICES
A EXTENSION: PROPOSED APPROACH
A.1 BUDGET CONSTRAINTS
Additional details related to the 4 budget constraints discussed in this paper follow below.
Channel budget. refers the maximum number of hidden channels h that can be used across all convolutional layers of the network. Mathematically, it can be stated as
(4)
where p denotes the number of hidden channels in the network.
Volume budget. This budget controls the size of the activations, thereby imposing upper limit on the memory requirement for the inference step. We define volume budget as
Σv Σ Azz:
(5)
EN A5-P3 Pj
where (h) denotes the number of convolutional layers in the network, and Aj and Pj denote area of the feature maps and their count, respectively, in the jth layer.
Parameter budget. This budget term directly controls the total number of parameters in the network, and can thus be used to impose upper limit the size of the model parameters. It is defined as
VG) IN (Kr E 2
2 + EP
(6)
E Pj P 2- P.)
where K, denotes area of the kernel. The two terms the numerator account for the number of parameters in the convolutional layer and the batchnorm layer.
FLOPs budget. This budget can be directly used to control the computational requirement of the model. Assuming that sliding window is used to achieve convolution and the nonlinear computa- tional overhead is ignored, the FLOPs budget of the convolution neural network can be defined as in Molchanov al. (2016):
Vin END (K,
Aj
(7)
En (Ky Pp A
B TRAINING PROCEDURE
Details regarding the pretraining, pruning and finetuning steps are discussed below:
B. PRE-TRAINING
WRN-26-12, MobileNetV2, ResNet-50, ResNet-101, ResNet-110 were trained with batch size of 128 at initial learning rate of 10-2 using SGD optimizer with momentum 0.9 and weight decay 10 We use step learning rate strategy to decay learning rate by 0.5 after every 30 epochs. For CIFAR-10 and CIFAR-100, models were trained for 300 epochs whereas for Tiny ImageNet the reduced half to have maintain same number of iterations.
number of epochs
Preresnet-164 was trained with batch size of 64 at initial learning rate of x10 using SGD op- timizer with momentum 0.9 and weight decay 10-4, We Multi Step Learning rate strategy to decay learning rate by 0.1 after 80th and 120 epoch. The model was trained for 160 epochs for all datasets. This strategy is adopted from Liu al. (2017)
12

---

Published as a conference paper at ICLR 2021
B.2 PRUNING
A common pruning strategy was applied for all models irrespective of budget type or dataset. AdamW (Loshchilov Hutter 2019) with constant learning rate 10 and weight decay of 10-3 was used as optimizer. Weight decay for v was kept O. Weight for budget loss and crispness loss kept constant to 30 and respectively. 3 increases by 2 10-2 after every epoch starting from and doubles after every 2 epochs starting from 2.
B.3 FINE- TUNING
The finetuning of pruned model is done exactly similar to the pre-training step.
B.4 HARDWARE
All experiments were run on Google Cloud Platform instance with a NVIDIA V100 GPU (16GB), 16 GB RAM and 4 core processor.
C ADDITIONAL EXPERIMENTS
C.I PRUNING WITH VOLUME AND CHANNEL. BUDGET
This section shows results of ChipNet along with different baselines pruned with with channel and volume budget. Table is an extension to Table 2 presented in section 4.2 Table shows numerical
4
values corresponding figure discussed in section 4.2
Table 4: Performance scores for pruning ResNer-50 architecture on CIFAR-IOO/CIFAR-100 for BAR and ChipNet (ours) with volume budget (V) and channel budget (C). The number of parameters and FLOPS for the unpruned networks are 23.7 million and 2.45 x 10", respectively. Abbreviations 'Ace.' and 'Param.' refer accuracy and number of parameters, all scores are reported in %, and parameters and FLOPs are reported relative to those in the unpruned network.
CIFAR-10
CIFAR-100
Method
Budget (%)
Acc. f
Param.
FLOPs 1
Acc.
Param. |
FLOPs
Unpruned
93.3
100
100
73.0
100
100
ChipNet (C)
93.1
36.5
58.8
72.7
44.1
40.9
ChipNet (V)
50
93.4
18.6
29.0
72.1
58.0
38.4
BAR (V)
91.4
9.5
21.3
71.5
22.5
24.9
ChipNet (C)
93.0
12.3
30.0
72.6
18.7
20.7
ChipNet (V)
25
92.9
4.9
12.3
69.9
32.1
17.2
BAR (V)
91.5
2.3
7.4
71.8
5.4
7.3
ChipNet (C)
92.8
4.5
17.7
71.1
7.3
10.9
ChipNet (V)
12.5
91.0
2.8
5.1
65.5
22.5
9.0
BAR (V)
88.4
1.8
3.8
63.8
5.2
4.2
ChipNet (C)
92.1
1.6
8.8
67.0
1.8
4.8
ChipNet (V)
6.25
83.6
1.3
2.0
54.7
14.5
5.1
BAR (V)
84.0
0.9
1.3
42.9
3.7
2.0
C.2 PRUNING WITH ONLY LOGISTIC CURVES
continuous Heavyside approximation helps to penalize intermediate values of to attain values closer to 0-1. Only with logistic curves the distribution of soft masks gets concentrated at point shown in Figure 4b Although, the budget constraint will be satisfied, this kind of distribution hinders effective channel selection as the relative importance of z cannot be determined concretely. Contrary to this, using heaviside function with crispness loss models in
As discussed in section 3.2
13

---

Published as a conference paper at ICLR 2021
Table 5: Performance scores for pruning WideResNet architecture on CIFAR-IO. 10, CIFAR-100 and
Tiny ImageNet datasets for BAR (Lemaire et al. 2019), MorphNet (Gordon et al. 2018). ID (Denton al. (2014). WM (Han al. 2015b), Random Pruning and ChipNet (ours). All results are reported
in % accuracy
Method
Budget (%)
CIFAR-10 1
Tiny ImageNet 1
CIFAR-100) f
50
92.7
52.4
74.1
92.8
52
73.6
BAR
12.5
92.8
51.4
72.6
6,25
91.6
52.0
70.5
50
93.3
58.2
73.6
25
92.9
55.8
70.4
MorphNet
12.5
90.7
51.7
69.9
6.25
86.4
39.2
55.5
91.09
49.96
69.29
25
91.44
49.55
69.75
ID
12.5
90.37
45.77
66.03
6.25
86.92
39.72
59.13
50
91.11
49.01
68.98
25
91.20
49.67
69.10
WM
12.5
89.68
47.72
65.42
6.25
86.33
40.19
58.99
50
89.63
48.25
67.51
88.02
46.08
63.64
Random
12.5
84.62
39.41
59.22
6.25
81.36
29.53
48.88
50
94.7
61.6
77.3
94.4
61.5
77.1
ChipNet
12.5
93.9
59.6
75.8
6.25
92.7
56.7
71.4
Dainbbsion
Darribatonn
Dacttbution
9ranco
Sraffe
2006
J006
0.5
0.8
04
(a) with proposed approach
without Crispness Loss
(c) without logistic round function
Figure 4: Distribution of zetas obtained on pruning WRN-26-12 with CIFAR-100 for 16x channel pruning factor.
terms of their relative importance shown in Figure and hence results in more effective pruning of less important channels.
C.3 ROLE OF ROUNDIING IN BUDGET CALCULATION
As discussed section 3.3 budget calculation is done on rather than computing it directly over the masks z where E Z obtained by applying logistic projection z with yo 0.5 (Eq. 2 Importance of this projection can be seen through figure 4c The distribution of soft masks obtained with the proposed approach (Figure is clearly much more distinct than the one calculated without logistic round projection (Figure 44c.. Thus better threshold can selected to choose the active sparsity mask that satisfies the budget constraint.
14

---

Published as a conference paper at ICLR 2021
Numter
Numbee
Numose
Convo Lyye
Lyye'
CoCovoitiinaa Liye'
conoiiitooo Lyve'
(a) Volume Budget
(b) Channel Budget
(c) Parameter Budget
(d) FLOPs Budget
Figure 5: Visualization of the number of channels remaining per convolutional layer in the architec-
tures obtained from ChipNet with different choices of budget constraint and various pruning factors.
C.4 EFFECT THE CHOICE BUDGET
We further visualize how ChipNet performs pruning across the various convolutional layers of network for different choices budget. Figure shows the number active channels per convo- lutional layer for several pruning factors for the budget types. These results have been obtained on WRN-26-12 We see that the pruned networks with low resource budgets are aligned with those with the higher budgets in terms of distribution of active channels across layers. This could mean that the networks pruned for low resource budgets should be achievable hierarchically from those pruned on larger budgets. Further, we also see that there layers with almost channels left. The performance of our model is still not affected since these dead layers correspond to the skip connections in the network. ChipNet identifies such extra connections, and eliminates them the performance of the model is not affected significantly.
C.5 HYPERPARAMETER GRID SEARCH
An extensive grid search done search hyperparameters, Or,, 022, bine and Jine for pruning WRN- 26-12 12 on CIFAR-100 100 at 6.25% volume budget. Here a₁ and are the weightage values given to crispness loss and budget loss repectively in the joint loss as shown section 3.4 Aine and Time refers the the number of epochs after which value of beta increases by 0.1 and value of gamma doubles, effect these hyperparameters discussed in section 3.2 We found a cloud values for which the pruning accuracy is comparable. This cloud can be seen in table 6 We choose the best values from these for all our other experiments as we concluded that model pruning less senstive to these hyperparameters.
Table 6: Grid search on WRN-CI00 for 16x volume pruning factor. Here Acc refers to the validation accuracy of hard pruned model during pruning.
3
Binc
Vine
Ace( %
10
5.5
10
45
5
1
5.4
15
1
5.3
15
30
5
5.2
10
5
4.8
10
60
5
1
4.7
15
20
1
1
4.7
5
60
2
2
4.6
15
60
5
2
4.5
1
45
4.5
15
30
5
2
4.3
5
45
4.2
15
60
2
1
3.9
5
45
3.9
10
20
1
1
3.8
15

---

Published as a conference paper at ICLR 2021
C.6 SENSTIVITY ANALYSIS
In this section show the senstivity analysis for WRN-26612 12 Cifar-100 at 16x Volume budget constraint. We ran 5 experiments of pruning where value of all four hyperparameters 01, 02 Sines Tine were sampled from the uniform distribution with +110% perturbations from the tuned values.
Table 7: Senstivity analysis on WRN-CI00 for 16x volume pruning factor. Here Accuracy refers to the test accuracy of hard pruned model after finetuning.
Q1
Q2
iine
Vic
Accuracy
10.19
28.23
5.12
1.98
0.7143
10.34
29.17
4.67
2.05
0.7232
9.28
27.34
4.8
1.91
0.72
9.24
32,25
5.13
1.89
0.7132
9.24
30.47
5,48
2.07
0.7168
Mean
0.7175
Std dev.
0.00412
C.7 TRANSFERABILITY OF MASK
Here we show the complete results of Table to depict the transfcrability of mask proposed in section 4.2
Table Accuracy values (%) on CIFAR- 100 dataset for ResNet-100 101 pruned with different choices of channel budget (%) CIFAR- 100 (Base) and with masks from Tiny ImageNet (Host).
Budget(%)
Tiny ImageNet (Host Acc)
C100 (Base Acc)
C100 (Transfer Acc)
20
51.6
71.3
68.3
40
55.2
71.6
72.0
60
56.0
71.8
72.1
100
63.3
73.6
D IMPLEMENTATION DETAILS
D.1 BASELINE METHODS
BAR, LZR, MorphNet, WM, ID WRN-26-12: All results taken from Lemaire al. (2019) We reproduced few results to cross-check and ensure that there are no big deviations. We found that our reproduced results were very close to the one reported in the paper.
BAR on Resnet-50: Results are produced from the code given by (Lemaire et al. 2019). Pruning strategy adopted from Lemaire et al. (2019) and the number of iterations are adjusted to match ours for fair comparison.
Network Slimming on PreResNet-164: We have reproduced the results by using the same pre- training, pruning and finetuning strategy is used by Liu et al. (2017) and the same pretraining and finetuning strategy used for our results in order to fair comparison pruning algorithms.
E
PSEUDO CODE
Here present the explanations and pseudo-codes for the various functions used in 1
16

---

Published as a conference paper at ICLR 2021
E.1
LOGISTIC FUNCTION
Algorithm 2: LOGISTIC
Input
Optimization parameter corresponding to every mask v Growth rate control
parameter B
Output: Resultant intermediate projection ž
Z 1+e-PP
E.2 CONTINUOUS HEAVISIDE FUNCTION
Algorithm 3: CONTINUOUS HEAVISIDE
Input Intermediate projection ă; Curvature regularization parameter Output: Resultant final projection
- 77
E.3 CHIPNET LOss FUNCTION
Algorithm 4: CHIPNET LOSS
Input Target budget Vo: Current model budget V; Intermediate projection ž; Final projection 2; Predicted output y; Ground truth y; Crispness loss weight a;; Budget loss weight
012 Output: Loss L.
Lee Eylog(})
Le + 7 -
Lb (V- (V-VV)2
L + Lee + anc + 022 Lb
E.4 FORWARD FUNCTION
Forward function takes three inputs network weights (W), input data batch (x) and sparsity masks (z). The forward function is the forward pass of regular CNN; with one change that the respective sparsity masks are multiplied to the activation obtained after every batch normalization layer.
E.5 BACKWARD FUNCTION
The backward function is the back propagation pass of regular CNN to obtain the gradients of the loss with respect to the model parameters (W and v/
F ADDITIONAL RESULTS
Table 9: Performance scores for pruning MobileNetV2 architecture Cifar-10 10 for ChipNet with channel budget.
Budget(%)
Acc.
Unpruned
93.55
80
92.58
60
92.44
40
91.98
20
90.65
17

---

Published as a conference paper at ICLR 2021
Table 10: Performance scores for pruning ResNet-50 architecture on Inn-eme for ChipNet with volume budget (V) and channel budget (C).
Method
Budget(%)
Acc.
Unpruned
61.38
ChipNet (C)
50
56.65
25
54.72
12.5
52.73
6.25
47.49
ChipNet (V)
50
54.23
25
53
12.5
50.03
6.25
45.51
Table 11: Accuracy values (%) on CIFAR-10 dataset for ResNet- 10 pruned using volume budget.
Model
Base acc. f
Prune
t
FLOPs reduction
Pruning-A A (Li et al.
2016)
93.53%
93.51%
1.19x
Pruning-B B [L et al.
2016)
93.53%
93.30%
1.62x
SFP (He al.
20 (a
93.68%
93.86%
1.69x
C-SGD-578 (Din el al.
2019
94.38%
94.41% 1%
2.56x
CNN-FCF-A (Li al.
2019)
93.58%
93.67%
1.76x
CNN-FCF-B (Li al. 2019
93.58%
92.96%
3.42x
Group-HS 7e-5
Yang et
2019
93.62%
94.06%
2.30x
Gr lc-4
Yang et al.
2019
93.62%
93.80%
3.09x
Group-HS 1.5e-4
Yang et al.
2019
93.62%
93.54%
4.38x
Group-HS 2e-4
Yang et al.
2019
93.62%
92.97%
5.84x
Chip Net (Volume-2x)
93.98%
93.78%
2,66x
Chip Net (Volume-4x)
93.98%
92.38%
7.54x
Chip Net (Volume-8x)
93.98%
91.36%
12.53x
18


Reviews:
[{"review_id": "qX-o0YGpZTl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a way to prune neural networks using a continuous penalty function. \n\n== Pros ==\n\nThe experiments include some interesting illustrations beyond the basic accuracy comparisons. This includes illustrations of how weights are distributed between the layers (Fig 5), ablations on different components of the penalty function, and experiments on the transferability of the selected sparsity pattern.\n\nThere is a very complete description of the training procedure. Writing overall is clear. Code is included with submission. The code is readable and easy to get running.\n\n== Cons ==\n\nMany previous works propose a new sparsity penalty function, and associated optimization, for neural network training. As well as (Lemaire et al) and (Louizos et al) cited in the submission, there's is works such as:\n\n  * Srinivas et al. \"Training Sparse Neural Networks.\" CVPR 2017.\n  * Manessi et al. \"Automated Pruning for Deep Neural Network Compression.\" ICPR 2018.\n  * Zhu et. al. \"Improving Deep Neural Network Sparsity through Decorrelation Regularization.\" IJCAI 2018.\n  * Yang et. al. \"DeepHoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures.\" ICLR 2020\n\nWithin this general class of approaches, though, the specifics of the penalty function and optimization in this paper are novel. Whether that is a valuable novelty depends entirely on the empirical results: since the higher-level idea is well-explored this adds to the literature iff the submission has made the best design choices among these other methods.\n\nDespite their overall heft, the experiments have some missing pieces that make it hard to evaluate this. To start, the evaluation is on a less common (within the sparse neural network literature) choice of backbones for each dataset. See:\n\nBlalock et. al. \"What is the State of Neural Network Pruning?\" Sections 3.3 & 4.2\n\nWider-ResNet would be expected to have greater redundancy than ordinary ResNet, perhaps making an easier problem for sparsifying. However, this choice is inherited from BAR.\n\nOrdinary ResNet-50 is a more common choice, but less so on CIFAR-10. This choice differs from the corresponding experiments in the (Liu et al) work that is compared against, that uses VGGNet, DenseNet-40, and ResNet-164.\n\nThe code appears to have partial support for a much wider variety of models (see line 19 of pruning.py), though maybe not all are actually supported: for instance r32 and r152 don't seem to be included in the branches in models/resnet.py lines 421-428.\n\nAnd generally there are few comparisons made. This is possibly unavoidable due to the overall fragmentation of metrics. Choices of comparisons are the more closely similar papers in the literature (this is good), but lacks more some of the simple baselines (that can often do surprisingly well):\n\nGale et. al. \"The State of Sparsity in Deep Neural Networks\"\n\nI don't see experiments that show proposed optimization has consistent convergence across runs. Experiments dealing with \"stability\" are either highlighted single comparisons, or look at stability w.r.t. hyperparameters. Some of the reported accuracy differences in comparison are quite small, especially at higher levels of sparsity, so seeing the standard deviation of multiple runs of each method compared might be especially important.\n\n== After Rebuttal ==\n\nI had previously missed that Lemaire et. al. had some of the key comparisons (simple baselines done for the same architecture/dataset pairs) that are necessary to judge this method. I thank the authors for pointing this out. It is also helpful that these baselines are now included for completeness.\n\nC.6 still seems like an indirect measure of optimization stability, but it is reasonable to conclude from it that the differences from BAR are real, given the very low standard deviation in accuracy.\n\nI remain on the side that this is still a relatively incremental addition to the pruning literature, but I am now satisfied that it is well-validated.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fG9hsUk20k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new deterministic pruning strategy that employs continuous Heaviside function and crispness loss to identify a sparse network out of an existing dense network. Experiments show its effectiveness and robustness. Generally, it is well-written and easy to follow. Some minor issues are shown below.\n\n1.\tPseudo codes for the functions mentioned in Algorithm 1 should be provided to show them clearly. \n2.\tThe proposed algorithm is a deterministic algorithm and may fail in complex network pruning problems. An in-depth analysis of its limitation is needed.\n3.\tIt is recommended to use different labels for different algorithms in the figures.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P2_BNQHzgrh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper presents a new method for structure pruning called ChipNet. The ChipNet employs continuous Heaviside function with commonly used logistic curve and crispness loss to estimate sparsity masks. A combination of above three components is helpful to obtain approximately discrete solutions for a continuous optimization scheme.  As a result, it is possible to get a highly sparse network out of an existing pre-trained dense network. Experimentally, ChipNet outperforms other previous structured pruning methods by a large margin. \n\nOverall,  I don't think this paper has any critical drawbacks, and there are only a few comments. Therefore, I recommend accepting this paper.\n\nStrength\n\nThe motivation for this paper is clear and quite interesting. Also, the performance of the proposed pruning method is much better than previous baseline methods, and experiments were performed in various ways to show the effectiveness of it. \n\n\n\nComments and Weakness\n\nI have not much to criticize this paper, but I have some questions and comments as follows.\n\nIs it possible to get a similar result by applying the proposed method to the originally lightweight network such as Mobile Net?\n\nAre there comparisons with NAS-based methods?\n\nIs the proposed method applicable to networks for other tasks such as detection and segmentation?\n\n\nThere is a submitted paper (to this ICLR) that includes continuous relaxation of discrete network structure optimization for network growing (not pruning). \n\n“Growing Efficient Deep Networks by Structured Continuous Sparsification”\n\nThe direction of the paper is totally different, but there seems to be a similar part in terms of continuous relaxation. It would be good to mention the difference with this paper w.r.t. continuous relaxation.\n\nAlso, I would like that more related works are mentioned in recent techniques using continuous relaxation(approximation).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Vh050D0u5U", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\n\nSummary:\n\nThe paper provides a budget-aware regularizer method to train the network to be pruned. Existing methods based-on the regularizer method suffer from satisfying the user-specified constraints and resort to trial-and-error approach. The authors leverage logistic function and continuous heaviside function for continuous relaxation of the discrete variable to select the channels to be pruned.  There are four types of budget constraints (channel, volume, parameter, FLOPs).  \nThe experiment results on several different budget constraints (even in extreme condition) are quite promising. Also, these methods can be applied in transfer learning (transferrable network structure).\n\n##########################################################################\n\n\nReasons for score:\n\nOverall, I vote for marginal acceptance. I like the budget-aware constraints concept and its tightness which is shown in figure 3.\nHowever, I have several concerns about the papers (see cons) and authors need to consider these concerns in the rebuttal period. \n\n\n##########################################################################\n\n1. The budget-aware constraint is explicitly introduced and easy to implement.\n \n2. Ablation study in figure 5 provides insights into the network channel pruning.\n\n##########################################################################\n\n\nCons: \n \n(1) Experiment results about the other several pruning experiments are omitted. Therefore, I concern that the other readers are confused about the performance of the proposed method over the other pruning methods. Particularly, I wish to check the performance with [1].  Also, Experiments on the ImageNet (large dataset experiments) should be added for future research.\n\n(2) Details on the forward steps are omitted. (See 'Forward(x,W,z)' in Algorithm 1) Readers should have prior knowledge to guess the details on the forwarding steps.\n\n(3) Justification of the usage of two functions (logistic and heaviside function) is not enough. Two separate functions for continuous approximations are quite confusing. For example, '(Logistic(\\psi)+1)/2' can be used for continuous relaxation or 'Gumbel trick' can be used. \n \n(4) This might be the most critical one. Since the budget constraint is given based on the l2 loss with the target budget, '\\alpha2' should be tuned like the other regularizer based method. From a critical perspective,  there is no difference with the other regularized-based method in the sense that the trial-and-error on the tuning '\\alpha2' is stilled required and very large '\\alpha2' could degrade the performance of the network.\n\n\n=======================================================================================================\n\nAfter rebuttal : \n\nSorry for the delay. The authors address my concerns and reflect them on the revised version. I'm keeping my rating.\n\n\n[1] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures, ICLR 2020.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review:  This paper proposed a new method to prune neural networks using a continuous penalty function. All reviewers suggest acceptance (some are on borderline though) as the authors did a good job in the rebuttal phase. AC also could not find any particular reason to reject the paper (in particular, the overall writing is clear) and thinks that this paper is a meaningful addition to ICLR 2021. 
Decision: Accept (Poster)