CHIPNET: BUDGET-AWARE PRUNING WITH HEAVISIDE CONTINUOUS APPROXIMATIONS
Year: 2021

Abstract:
Structured pruning methods are among the effective strategies for extracting small resource-efficient convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network. Our choice of continuous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible and expected as final outcomes. Our approach’s flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on featurerich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself. 1

heading: 1 INTRODUCTION
text: Convolution Neural Networks (CNNs) have resulted in several breakthroughs across various disciplines of deep learning, especially for their effectiveness in extracting complex features. However, these models demand significantly high computational power, making it hard to use them on lowmemory hardware platforms that require high-inference speed. Moreover, most of the existing deep networks are heavily over-parameterized resulting in high memory footprint (Denil et al., 2013; Frankle & Carbin, 2018). Several strategies have been proposed to tackle this issue, that include network pruning (Liu et al., 2018), neural architecture search using methods such as reinforcement learning (Jaafra et al., 2019) and vector quantization (Gong et al., 2014), among others.
Among the methods outlined above, network pruning has proved to be very effective in designing small resource-efficient architectures that perform at par with their dense counterparts. Network pruning refers to removal of unnecessary weights or filters from a given architecture without compromising its accuracy. It can broadly be classified into two categories: unstructured pruning and structured pruning. Unstructured pruning involves removal of neurons or the corresponding connection weights from the network to make it sparse. While this strategy reduces the number of parameters in the model, computational requirements are still the same (Li et al., 2017). Structured pruning methods on the other hand remove the entire channels from the network. This strategy pre-
∗All authors contributed equally. 1Code is publicly available at https://github.com/transmuteAI/ChipNet
serves the regular structure, thereby taking advantage of the high degree of parallelism provided by modern hardware (Liu et al., 2017; Gordon et al., 2018).
Several structured pruning approaches have been proposed in the recent literature. A general consensus is that variational approaches using sparsity prior loss and learnable dropout parameters outperform the deterministic methods (Lemaire et al., 2019). Some of these methods learn sparsity as a part of pretraining, and have proved to perform better than the three stage pretrain-prune-finetune methods. However, since such approaches need to train the model from scratch with pruning-related variables embedded into the network, they cannot benefit from off-the-shelf pretrained weights (Liu et al., 2017; Alvarez & Salzmann, 2017). Others require choosing hyperparameters based on the choice of the network, and cannot be easily adapted for new models (Gordon et al., 2018). Further, with most of these methods, controlled pruning cannot be performed, and a resource-usage constraint can only be satisfied through trial-and-error approach. Recently, Lemaire et al. (2019) presented a budget-aware pruning method that includes the budget constraint as a part of the training process. A major drawback of this approach and other recent methods is that they are unstable for very low resource budgets, and require additional tricks to work. Overall, a robust budget-aware pruning approach that can be coupled with different budget constraints as well as maintains stability for very low target budgets, is still missing in the existing literature.
In this paper, we present ChipNet, a deterministic strategy for structured pruning that employs continuous Heaviside function and crispness loss to identify a highly sparse network out of an existing pretrained dense network. The abbreviation ‘ChipNet’ stands for Continuous Heaviside Pruning of Networks. Our pruning strategy draws inspiration from the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible. Thus, only such values are produced as final outcomes through continuous Heaviside projections. We use a similar strategy to obtain the masks in our sparsity learning approach. The flexible design of ChipNet facilitates its use with different choices of budget constraints, such as restriction on the maximum number of parameters, FLOPs, channels or the volume of activations in the network. Through experiments, we show that ChipNet consistently outperforms state-of-the-art pruning methods for different choices of budget constraints.
ChipNet is stable for even very low resource budgets, and we demonstrate this through experiments where network is pruned to as low as 1% of its parameters. We show that for such extreme cases, ChipNet outperforms the respective baselines by remarkable margins, with a difference in accuracy of slightly beyond 16% observed for one of the experiments. The masks learnt by ChipNet are transferable across datasets. We show that for certain cases, masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.

heading: 2 RELATED WORK
text: As has been stated in the hypothesis by Frankle & Carbin (2018), most neural networks are overparameterized with a large portion (as much as 90%) of the weights being of little significance to the output of the model. Clearly, there exists enormous scope to reduce the size of these networks. Several works have explored the efficiency of network pruning strategies for reducing storage requirements of these networks and accelerating inference speed (LeCun et al., 1990; Dong et al., 2017). Some early works by Han et al. (2015a;c); Zhu & Gupta (2017) involve removal of individual neurons from a network to make it sparse. This reduces the storage requirements of these networks, however, no improvement in inference speed is observed. Recently, several works have focused on structured network pruning, as it involves pruning the entire channel/filters or even layers to maintain the regular structure (Luo et al., 2017; Li et al., 2017; Alvarez & Salzmann, 2016).
The focus of this paper is on structured network pruning, thus, we briefly discuss here the recent works related to this approach. The recent work by Li et al. (2017) identifies less important channels based on L1-norm. Luo et al. (2017); He et al. (2017) perform channel selection based on their influence on the activation values of the next layer. Liu et al. (2017) perform channel-level pruning by imposing LASSO regularization on the scaling terms in the batchnorm layers, and prune the model based on a global threshold. He et al. (2018b) automatically learn the compression ratio of each layer with reinforcement learning. Louizos et al. (2017); Alvarez & Salzmann (2017; 2016) train and prune the network in a single stage strategy.
The above mentioned approaches cannot optimize networks for a pre-defined budget constraint. Adding budget constraint to the pruning process can provide a direct control on the size of the pruned network. For example, Morphnet imposes this budget by iteratively shrinking and expanding a network through a sparsifying regularizer and uniform layer wise width multiplier, respectively, and is adaptable to specific resource constraints (Gordon et al., 2018). However, it requires a modelspecific hyperparameter grid search for choosing the regularization factor. Another approach is BAR (Lemaire et al., 2019) that uses a budget-constrained pruning approach based on variational method. A limitation of this approach is that for low resource budgets, it needs to explicitly ensure that at least one channel is active in the downsample layer to avoid fatal pruning. The approach proposed in this paper does not require any such tweaking, and is stable for even very low resource budgets.

heading: 3 PROPOSED APPROACH

heading: 3.1 LEARNING SPARSITY MASKS
text: Sparsity learning forms the core of our approach. It refers to learning a set of sparsity masks for a dense convolutional neural network (parent). When designing the smaller pruned network (child), these masks identify the parts of the parent that are to be included in the child network. We first describe here the general idea of learning these masks in the context of our method.
The proposed approach falls in the category of structured pruning where masks are designed for channels and not individual neurons. Let f : Rd → Rk denote a convolutional neural network with weights W ∈ Rm and a set of hidden channels H ∈ Rp. We define z ∈ Rp as a set of sparsity masks, where zi ∈ z refers to the mask associated with the feature map hi ∈ H. To apply the mask, zi is multiplied with all the entries of hi. The optimization problem can further be stated as
min W,z L(f(z H(W);x),y) s.t. V(z) = V0, (1)
where denotes elementwise multiplication and {x,y} ∈ D are data samples used to train the network f . The desired sparsity of the network is defined in terms of the equality constraint, where V(·) denotes the budget function and V0 is the maximum permissible budget. Our proposed formulation of pruning is independent from the choice of the budget function. We later show this through experiments with volume budget as in Lemaire et al. (2019), channel budget similar to Liu et al. (2017), and budget defined in terms of parameters and FLOPs as well.
Originally, zi ∈ z would be defined such that zi ∈ {0, 1}, and a discrete optimization problem is to be solved. For the sake of using gradient-based methods, we convert it to a continuous optimization problem, such that zi ∈ [0, 1]. Such reformulation would lead to intermediate values of z occurring in the final optimized solution. Any intermediate value of z, for example z = 0.4, would imply that a fraction of the respective channel is to be used, and clearly such a solution is practically infeasible. We propose to overcome this challenge through the use of simple nonlinear projections and a novel loss term, and these are discussed in detail in the next section.

heading: 3.2 CONTINUOUS HEAVISIDE APPROXIMATION AND LOGISTIC CURVES
text: At the backbone of our pruning strategy lies three important functions: the commonly used logistic curves, continuous Heaviside function and crispness loss term. Figure 1 presents a graphical representation of these functions. We further provide below a brief motivation for the choice of these functions as well as their significance in our pruning approach.
Logistic curves. A commonly used function for adding nonlinearity to a neural network (LeCun et al., 1998), logistic curve projects an input from the real space to a range of 0-1 (Figure 1a), and can be mathematically stated as
z̃ = 1
1 + e−β(ψ−ψ0) , (2)
where ψ denotes the optimization parameter corresponding to the mask z, ψ0 is the midpoint of the curve, and z̃ denotes the resultant intermediate projection. The additional parameter β is used to control the growth rate of the curve, and forms an important ingredient of our approach. While low values of β can produce an approximately linear curve between -1 and 1, higher values turn it into a step function. During the initial stages of training, we propose to keep β very low, and increase it to higher values at later stages of the optimization process. With increased values of β, the values further from 0.5 are made more favorable for z̃.
In our experience, the logistic curve alone cannot be used to obtain approximately discrete (0-1) solutions for z in a continuous optimization scheme. The nonlinearity introduced by this function cannot sufficiently penalize the intermediate values between 0 and 1, and optimization algorithm can easily identify values of ψ for which the projected values are far from both. An example experiment demonstrating this issue is presented in Appendix C.2. To circumvent this issue, we add another nonlinear projection using a continuous approximation of the Heaviside function.
Continuous Heaviside function. A continuous approximation to the Heaviside step function, referred as continuous Heaviside function in this paper, is a commonly used projection strategy to solve continuous versions of binary optimization (0-1) problems in the domain of design optimization (Guest et al., 2004; 2011). The generalized form of this function can be stated as:
z = 1− e−γz̃ + z̃e−γ , (3) where, the parameter γ dictates the curvature of the regularization. Figure 1b shows the continuous Heaviside function for several values of γ. We see that z is linear for γ = 0 and approaches the Heaviside step function for very large values of γ.
The advantages of our projection function are twofold. First, during projection, the values close to 0 and 1 are not affected irrespective of the choice of γ. This implies that the masks identified with most confidence in the early stage of training are not directly impacted by the continuation applied on the value of γ, thus helping in the convergence of the training process. Here, ‘continuation’ refers to slowly adapting the value of γ during the course of training. Second, even the values of z̃ which are slightly greater than 0 are also nonlinearly projected to close to 1, and this effect is more prominent for larger values of γ. The projection adds higher penalty on values between 0 and 1, and makes them extremely unfavorable when higher values of γ are chosen.
While the continuous Heaviside function helps to obtain approximately discrete masks, there is still no explicit constraint or penalty function that can regulate this. To overcome this problem, we tie the outputs of logistic and continuous Heaviside functions to define a novel loss term, referred as crispness loss.
Crispness Loss. This novel loss term explicitly penalizes the model performance for intermediate values of z, and drives the convergence towards crisp (0-1) masks. It is defined as the squared L2 norm of the difference between z̃ and z, stated as Lc = ‖z̃ − z‖22, and from Figure 1c, we see that Lc achieves its minima when either z̃ = z = 0 or z̃ = z = 1. Further, the trend of this loss function with respect to ψ for different values of β and γ is shown in Figure 1d. It can be seen that for lower values of β and γ, the loss value is low, and the crispness function plays little to no role in driving the pruning process. When the value of γ slowly increases, the peak of the graph shifts upwards as well as towards the left, thereby increasing the penalty associated with values of ψ. This drives the values of ψ to move farther from the origin. The left shift in the graph adds higher penalty on the negative values, forcing them to become even more negative, thus forcing the respective z to move closer to 0.
The additional loss function associated with the model generally favors values towards 1. For example, cross-entropy loss used for classification would prefer to set all values in z to 1 to be able to maximize the classification accuracy. With increasing values of γ forcing the masks towards 0, a balance between the two is identified during the training process. The term β acts as a regularizer that to some extent counteracts the too abrupt impact of γ and regulates the convergence of the training process.

heading: 3.3 IMPOSING BUDGET CONSTRAINT
text: The simplicity of our pruning strategy decouples it from the choice of budget constraint. In this paper, we show the working with four different choices of budget constraints: channel, activation volume, parameters and FLOPs. These choices are inspired from some of the recent state-of-the-art methods from the existing literature (Liu et al., 2017; Lemaire et al., 2019).
For budget calculation, values of the masks z should be close to 0 or 1. However, during the initial iterations of training, masks would contain intermediate values as well. This makes it difficult to accurately calculate the budget for the constraint specified in Eq. 1. Thus, rather than computing it directly over the masks z, these are computed on z̄, where z̄i ∈ z̄ is obtained by applying a logistic projection on z with ψ0 = 0.5 (Eq. 2). Further discussion related to it is provided in Appendix C.3.
The budget constraint is imposed using a loss term Lb, referred as budget loss. We define the budget loss as Lb = (V(z)− V0)2, where V(·) can be one of the 4 budget functions described below. Channel budget. It refers to the maximum number of hidden channels h that can be used across all convolutional layers of the network. Mathematically, it can be stated as V(c) = ( ∑p i=1 z̄i)/p, where p denotes the number of hidden channels in the network. Constraint on the channel budget limits the number of channels, and thus the number of weights in the network.
Volume budget. This budget controls the size of the activations, thereby imposing an upper limit on the memory requirement for the inference step. We define volume budget V(v) = ( ∑N (h) j=1 ∑pj i=1Aj z̄i)/( ∑N (h) j=1 Aj · pj), where N (h) denotes the number of convolutional layers, and Aj and pj denote area of the feature maps and their count, respectively, in the jth layer.
Parameter budget. This budget directly controls the total number of parameters in the network, and can thus be used to impose an upper limit on the size of the model. For details, see Appendix A.1.
FLOPs budget. This budget can be directly used to control the computational requirement of the model. Mathematical formulae to calculate it is stated in Appendix A.1.
3.4 SOFT AND HARD PRUNING
Algorithm 1: ChipNet Pruning Approach Input : pretrained network weights W;
budget constraint function V(·); budget value V0; training data D; pruning iterations N
Output: learnt sparsity masks z ψi ∈ Ψ← random initialization for k = 1 . . . N do
(x,y)← sample(D) z̃← LOGISTIC(ψ) z← CONTINUOUSHEAVISIDE(z̃) ŷ← Forward(x,W, z) V ← V(z) L ← CHIPNETLOSS(V,V0, z̃, z, ŷ,y) (∇W,∇ψ)← Backward(L) (W, ψ)← OptimizeStep(∇W,∇ψ)
end
The pruning stage in our approach comprises two steps: soft pruning and hard pruning. After a deep dense network has been pretrained, masks are added to the network and soft pruning is performed. The steps involved in soft pruning are stated in Algorithm 1. During this stage, the network is optimized with the joint loss L = Lce + α1Lc + α2Lb, where α1 and α2 are the weights for the crispness and budget loss terms, respectively.
After every epoch of soft pruning, the performance of the network is evaluated in a hard pruned manner. For this purpose, masks z are used, and a cutoff is chosen using binary search such that the budget constraint is exactly satisfied. Values above this cutoff are converted to 1 and the ones lower turned to 0. Finally, the model with best performance on the validation set is chosen for fine tuning.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: We test the efficacy of our pruning strategy on several network architectures for four different choices of budget constraint functions. The architectures chosen in this study include WideResNet-26-12 (WRN-26-12) (Zagoruyko & Komodakis, 2016), PreResNet-164 (He et al., 2016b), ResNet-50 and ResNet-101 (He et al., 2016a). For datasets, we have chosen CIFAR-10/100 (Krizhevsky, 2009) and Tiny ImageNet (Wu et al.). For the combined loss L in Eq. 1, weights α1 and α2 are set to 10 and 30, respectively, across all experiments. Implementation details related to the pretraining, pruning and finetuning steps, as well details of the hardware are described in Appendix B.

heading: 4.2 RESULTS
text: Performance of pruned sparse networks. We present here results obtained using ChipNet for WRN-26-12 and PreResNet-164 pruned with volume and channel constraints, respectively. For the two other constraints, parameters and FLOPs budget, we perform a comparative study later in this paper.
Volume budget. Figure 2 shows the comparison of performance values for WRN-26-12 for CIFAR-10, CIFAR-100 and TinyImageNet datasets, when pruned using ChipNet. We compare our results with BAR (Lemaire et al., 2019), MorphNet (Gordon et al., 2018) and LZR (Louizos et al., 2017) for volume pruning factors of 2, 4, 8 and 16. Details related to the three baselines are presented in Appendix D.1. ChipNet consistently outperforms all the baselines across all datasets and for all choices of the budget. For the case of extreme pruning of 16 folds on CIFAR-100, the performance of BAR is close to ours, while the other two baselines significantly underperform.
Channel budget We study here the pruning efficacy of ChipNet coupled with channel constraint on PreResNet-164 architecture for CIFAR-10 and CIFAR-100 datasets. Results are compared with the network slimming approach (Liu et al., 2017), implementation details related to which can be found in Appendix D.1. As constraints, we use channel budgets of 60%, 40%, 20% and 10%.
Table 1 presents the results for different choices of channel budgets. We also report the number of parameters in the pruned network as well as the associated FLOPs. It is seen that ChipNet outperforms the baseline method for all the experimental settings. For CIFAR-10 in particular, we see that for even very low channel budget of 10%, accuracy of the pruned network drops by only 3.1%. For 10% channel budget, our method outperforms the network slimming strategy on CIFAR10 and CIFAR-100 by remarkable margins of 8.5% and 16.1%, respectively.
Note that lower channel usage does not necessarily imply lower number of parameters or reduced FLOPS in the pruned network, and we analyze this for the various cases of pruning considered in Table 1. We see that ChipNet performs selection of channels in a more optimized way, such that better accuracy is achieved with fewer parameters. In terms of FLOPS, both methods perform at par. Although, the FLOPS for ChipNet are slightly higher for the channel budget of 10%, this overhead is insignificant compared to the gain in accuracy and reduction of parameters. Overall, we infer that ChipNet couples well with the channel constraint, and is stable for even extreme pruning cases of as low as 10% channel budget.
Effect of the choice of budget. Here, we analyze the impact of one budget type over another to understand whether the choice of budget really matters when pruning a network. As a first experiment, we study side-by-side the results for channel and volume constraints when used to prune ResNet-50 on CIFAR-10 and CIFAR-100 datasets. Results of this experiment are shown in Table 2. Note that we do not intend to identify a winner among the two, since both are meant to optimize different aspects of the network. For baseline comparison, the network is also pruned using the BAR method. The volume budget variant of ChipNet outperforms BAR by a significant margin. Moreover, we see that for the same amount of volume constraint, the number of parameters used by BAR are lower than our method for most cases. A reason for significant drop in performance of BAR could be that the optimization algorithm does not fully exploit the choice of channels to be dropped, thereby choosing a sub-optimal set and losing too many parameters from the network.
Between the results of volume and channel constraints for ChipNet, at the first look, it seems that channel constraint is better throughout. However, as stated above, a direct comparison between the two is unfair. For example, volume constraints are meant to reduce the number of activations, and in turn would also reduce the FLOPs. This is evident from the results as FLOPS reported for volume constraint are always lower than the respective channel constraint. For a better understanding of the effects of these budgets, we perform another experiment for pairwise analysis of these constraints.
Figure 3 shows the pairwise plots of the budgets used to prune WRN-26-12 on CIFAR-100. From the first two plots, we see that the scores reported are higher for any volume budget when the network is optimized with volume constraint, and similarly higher for a certain channel budget when the network is optimized for it. Similar observations can also be made between the number of parameters and FLOPs. In a nutshell, we observe that the pruned network performs best with respect to the constraint for which the masks are trained. Thus, the choice of constraint type should not be arbitrary but based on the practical applications, such as reducing FLOPs, among others.
Stability and robustness. Our pruning strategy is also very stable, and this has already been demonstrated earlier for channel and volume pruning at low resource budgets. Compared to the baselines, networks obtained with ChipNet are found to perform significantly better even without the need for any additional tweaking such as explicitly opening certain channels to ensure network connectivity (Liu et al., 2017; Lemaire et al., 2019). Another example demonstrating the stability is volume pruning (6.25%) of ResNet-50 on CIFAR-100, where ChipNet performs 11.8% better than BAR.
To account for robustness, we have extensively performed hyperparameter grid search on a channel budget of 6.25% for WRN-26-12 to identify the suitable values for α1 and α2. It has been observed that values in the neighborhood of this point do not affect the performance. Details related to this grid search are further provided in Appendix C.5. Further, the same hyperparameter setting has been used for all the experiments. The consistent results across all datasets shows that ChipNet is robust.
Transfer learning of masks. Inspired by knowledge distillation (Hinton et al., 2015), where refined information obtained from a deeper teacher network is transferred to a shallow student network, we study here the transfer of sparsity masks across datasets. For teacher and student, we use Tiny ImageNet and CIFAR-100 datasets, respectively, and ResNet-101 is pruned for different choices of channels budgets. Table 8 reports the performance scores for the pruned network on CIFAR-100 when the masks are learnt on CIFAR-100 as well as when they are learnt on Tiny ImageNet and transferred. Interestingly, for moderate channel budgets of 40% and 60%, we see that the models using masks transferred from Tiny ImageNet perform better than those obtained directly on CIFAR-100. This gain in performance from mask transfer could be attributed to the feature-richness of the chosen teacher
dataset. We also see that for the very low budget case of 20%, masks from the student dataset outperform that from the teacher. For such low budgets, the expressive power of the model is too low to fully exploit the knowledge from the transferred masks.

heading: 5 CONCLUSION
text: We have presented ChipNet, a deterministic strategy for structured pruning of CNNs based on continuous Heaviside function and crispness loss. Our approach provides the flexibility of using it with different budget constraints. Through several experiments, it has been demonstrated that ChipNet outperforms the other methods on representative benchmark datasets. We have also shown that
ChipNet can generate well-performing pruned architectures for very low resource budgets as well. To conclude, with the strongly effective pruning capability that ChipNet exhibits, it can be used by the machine learning community to design efficient neural networks for a variety of applications.

heading: 6 LIMITATIONS AND FUTURE WORK
text: In this paper, we have explored the stability and robustness of ChipNet from various perspectives. Through experiments, we have shown that ChipNet consistently performs well across several CNN architectures and datasets. We analyzed it with respect to different choices of budget constraints, performed stability tests for even extreme scenarios of as low as 1% parameters, analyzed how the masks get distributed across the network, and even studied the transferability of masks. For all these experiments, ChipNet has proved to work well. However, before ChipNet can be considered a fullproof solution for pruning, additional experiments might be needed. For example, the applicability of ChipNet is not yet explored on large datasets such as ImageNet, and we would like to explore it in our future work.
Further, it would be of interest to explore how the pruned architectures obtained using ChipNet perform for tasks beyond classification, such as segmentation and object tracking. Improving inference speed is an important aspect of object tracking, and it has not been explored from the point of view of network pruning. We would like to see if the recent object tracking algorithms that use backbones such as ResNet-50 and ResNet-101 can be made faster through our pruning method.

heading: APPENDICES

heading: A EXTENSION: PROPOSED APPROACH

heading: A.1 BUDGET CONSTRAINTS
text: Additional details related to the 4 budget constraints discussed in this paper follow below.
Channel budget. It refers to the maximum number of hidden channels h that can be used across all convolutional layers of the network. Mathematically, it can be stated as
V(c) = ∑p i=1 z̄i p , (4)
where p denotes the number of hidden channels in the network.
Volume budget. This budget controls the size of the activations, thereby imposing an upper limit on the memory requirement for the inference step. We define volume budget as
V(v) = ∑N (h) j=1 ∑pj i=1Aj z̄i∑N (h)
j=1 Aj · pj , (5)
where N (h) denotes the number of convolutional layers in the network, and Aj and pj denote area of the feature maps and their count, respectively, in the jth layer.
Parameter budget. This budget term directly controls the total number of parameters in the network, and can thus be used to impose an upper limit on the size of the model parameters. It is defined as
V(p) = ∑N (h) j=1 (Kj · ∑pj i=1 z̄ j i · ∑pj−1 i=1 z̄ j−1 i + 2 · ∑pj i=1 z̄ j i )∑N (h)
j=1 (Kj · pj · pj−1 + 2 · pj) , (6)
where Kj denotes area of the kernel. The two terms in the numerator account for the number of parameters in the convolutional layer and the batchnorm layer.
FLOPs budget. This budget can be directly used to control the computational requirement of the model. Assuming that a sliding window is used to achieve convolution and the nonlinear computational overhead is ignored, the FLOPs budget of the convolution neural network can be defined as in Molchanov et al. (2016):
V(f) = ∑N (h) j=1 (Kj · ∑pj−1 i=1 z̄ j−1 i + 1) · ∑pj i=1 z̄ j i ·Aj∑N (h)
j=1 (Kj · pj−1 + 1) · pj ·Aj . (7)

heading: B TRAINING PROCEDURE
text: Details regarding the pretraining, pruning and finetuning steps are discussed below:

heading: B.1 PRE-TRAINING
text: WRN-26-12, MobileNetV2, ResNet-50, ResNet-101, ResNet-110 were trained with batch size of 128 at initial learning rate of 5× 10−2 using SGD optimizer with momentum 0.9 and weight decay 10−3. We use step learning rate strategy to decay learning rate by 0.5 after every 30 epochs. For CIFAR-10 and CIFAR-100, models were trained for 300 epochs whereas for Tiny ImageNet the number of epochs were reduced to half to have maintain same number of iterations.
Preresnet-164 was trained with batch size of 64 at initial learning rate of ×10−1 using SGD optimizer with momentum 0.9 and weight decay 10−4. We use Multi Step Learning rate strategy to decay learning rate by 0.1 after 80th and 120th epoch. The model was trained for 160 epochs for all datasets. This strategy is adopted from Liu et al. (2017)

heading: B.2 PRUNING
text: A common pruning strategy was applied for all models irrespective of budget type or dataset. AdamW (Loshchilov & Hutter, 2019) with constant learning rate of 10−3 and weight decay of 10−3 was used as optimizer. Weight decay for ψ was kept 0. Weight for budget loss and crispness loss is kept constant to 30 and 10 respectively. β increases by 2 × 10−2 after every epoch starting from 1 and γ doubles after every 2 epochs starting from 2.

heading: B.3 FINE-TUNING
text: The finetuning of pruned model is done exactly similar to the pre-training step.

heading: B.4 HARDWARE
text: All experiments were run on a Google Cloud Platform instance with a NVIDIA V100 GPU (16GB), 16 GB RAM and 4 core processor.

heading: C ADDITIONAL EXPERIMENTS

heading: C.1 PRUNING WITH VOLUME AND CHANNEL BUDGET
text: This section shows results of ChipNet along with different baselines pruned with with channel and volume budget. Table 4 is an extension to Table 2 presented in section 4.2. Table 5 shows numerical values corresponding to figure 2 discussed in section 4.2.

heading: C.2 PRUNING WITH ONLY LOGISTIC CURVES
text: As discussed in section 3.2, continuous Heavyside approximation helps to penalize intermediate values of z to attain values closer to 0-1. Only with logistic curves the distribution of soft masks gets concentrated at one point as shown in Figure 4b. Although, the budget constraint will be satisfied, this kind of distribution hinders effective channel selection as the relative importance of z cannot be determined concretely. Contrary to this, using heaviside function with crispness loss models z in
terms of their relative importance as shown in Figure 4a and hence results in more effective pruning of less important channels.

heading: C.3 ROLE OF LOGISTIC-ROUNDING IN BUDGET CALCULATION
text: As discussed in section 3.3 budget calculation is done on z̄ rather than computing it directly over the masks z where z̄i ∈ z̄ is obtained by applying a logistic projection on z with ψ0 = 0.5 (Eq. 2). Importance of this projection can be seen through figure 4c. The distribution of soft masks obtained with the proposed approach (Figure 4a) is clearly much more distinct than the one calculated without logistic round projection (Figure 4c). Thus a better threshold can be selected to choose the active sparsity mask that satisfies the budget constraint.

heading: C.4 EFFECT OF THE CHOICE OF BUDGET
text: We further visualize how ChipNet performs pruning across the various convolutional layers of a network for different choices of budget. Figure 5 shows the number of active channels per convolutional layer for several pruning factors for the 4 budget types. These results have been obtained on WRN-26-12. We see that the pruned networks with low resource budgets are aligned with those with the higher budgets in terms of distribution of active channels across layers. This could mean that the networks pruned for low resource budgets should be achievable hierarchically from those pruned on larger budgets. Further, we also see that there are layers with almost no channels left. The performance of our model is still not affected since these dead layers correspond to the skip connections in the network. ChipNet identifies such extra connections, and eliminates them if the performance of the model is not affected significantly.

heading: C.5 HYPERPARAMETER GRID SEARCH
text: An extensive grid search is done to search hyperparameters, α1, α2, binc and ginc for pruning WRN26-12 on CIFAR-100 at 6.25% volume budget.Here α1 and α2 are the weightage values given to crispness loss and budget loss repectively in the joint loss as shown in section 3.4. βinc and γinc refers the the number of epochs after which value of beta increases by 0.1 and value of gamma doubles, effect of these hyperparameters is discussed in section 3.2. We found a cloud of values for which the pruning accuracy is comparable. This cloud can be seen in table 6. We choose the best values from these for all our other experiments as we concluded that model pruning is less senstive to these hyperparameters.

heading: C.6 SENSTIVITY ANALYSIS
text: In this section we show the senstivity analysis for WRN-26-12 on Cifar-100 at 16x Volume budget constraint. We ran 5 experiments of pruning where value of all four hyperparameters α1, α2, βinc, γinc were sampled from the uniform distribution with -+10% perturbations from the tuned values.

heading: C.7 TRANSFERABILITY OF MASK
text: Here we show the complete results of Table 3 to depict the transferability of mask proposed in section 4.2
D IMPLEMENTATION DETAILS

heading: D.1 BASELINE METHODS
text: BAR, LZR, MorphNet, WM, ID on WRN-26-12: All results are taken from Lemaire et al. (2019). We reproduced a few results to cross-check and ensure that there are no big deviations. We found that our reproduced results were very close to the one reported in the paper.
BAR on Resnet-50: Results are produced from the code given by (Lemaire et al., 2019). Pruning strategy is adopted from Lemaire et al. (2019) and the number of iterations are adjusted to match ours for fair comparison.
Network Slimming on PreResNet-164: We have reproduced the results by using the same pretraining, pruning and finetuning strategy is used by Liu et al. (2017) and the same pretraining and finetuning strategy is used for our results in order to do fair comparison of pruning algorithms.

heading: E PSEUDO CODE
text: Here we present the explanations and pseudo-codes for the various functions used in 1.

heading: E.1 LOGISTIC FUNCTION
text: Algorithm 2: LOGISTIC Input : Optimization parameter corresponding to every mask ψ; Growth rate control parameter β Output: Resultant intermediate projection z̃ z̃← 1
1+e−βψ

heading: E.2 CONTINUOUS HEAVISIDE FUNCTION
text: Algorithm 3: CONTINUOUS HEAVISIDE Input : Intermediate projection z̃; Curvature regularization parameter γ Output: Resultant final projection z z← 1− e−γz̃ + z̃e−γ

heading: E.3 CHIPNET LOSS FUNCTION
text: Algorithm 4: CHIPNET LOSS Input : Target budget V0; Current model budget V; Intermediate projection z̃; Final projection
z; Predicted output ŷ; Ground truth y; Crispness loss weight α1; Budget loss weight α2
Output: Loss L Lce ← − ∑ y log(ŷ) Lc ← ‖z̃− z‖22 Lb ← (V − V0)2 L ← Lce + α1Lc + α2Lb

heading: E.4 FORWARD FUNCTION
text: Forward function takes three inputs - network weights (W), input data batch (x) and sparsity masks (z). The forward function is the forward pass of regular CNN; with one change that the respective sparsity masks are multiplied to the activation obtained after every batch normalization layer.

heading: E.5 BACKWARD FUNCTION
text: The backward function is the back propagation pass of a regular CNN to obtain the gradients of the loss with respect to the model parameters (W and ψ ).

heading: F ADDITIONAL RESULTS

Review summaries:
The paper provides a budget-aware regularizer method to train the network to be pruned. Existing methods based-on the regularizer method suffer from satisfying the user-specified constraints and resort to trial-and-error approach. The authors leverage logistic function and continuous heaviside function for continuous relaxation of the discrete variable to select the channels to be pruned.  There are four types of budget constraints (channel, volume, parameter, FLOPs).  

This paper presents a new method for structure pruning called ChipNet. The ChipNet employs continuous Heaviside function with commonly used logistic curve and crispness loss to estimate sparsity masks. A combination of above three components is helpful to obtain approximately discrete solutions for a continuous optimization scheme.  As a result, it is possible to get a highly sparse network out of an existing pre-trained dense network. Experimentally, ChipNet outperforms other previous structured pruning methods by a large margin.