BWCP: PROBABILISTIC LEARNING-TO-PRUNE CHAN-
Year: 2021

Abstract:
This work presents a probabilistic channel pruning method to accelerate Convolutional Neural Networks (CNNs). Previous pruning methods often zero out unimportant channels in training in a deterministic manner, which reduces CNN’s learning capacity and results in suboptimal performance. To address this problem, we develop a probability-based pruning algorithm, called batch whitening channel pruning (BWCP), which can stochastically discard unimportant channels by modeling the probability of a channel being activated. BWCP has several merits. (1) It simultaneously trains and prunes CNNs from scratch in a probabilistic way, exploring larger network space than deterministic methods. (2) BWCP is empowered by the proposed batch whitening tool, which is able to empirically and theoretically increase the activation probability of useful channels while keeping unimportant channels unchanged without adding any extra parameters and computational cost in inference. (3) Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet with various network architectures show that BWCP outperforms its counterparts by achieving better accuracy given limited computational budgets. For example, ResNet50 pruned by BWCP has only 0.58% Top-1 accuracy drop on ImageNet, while reducing 42.9% FLOPs of the plain ResNet50.

text: This work presents a probabilistic channel pruning method to accelerate Convolutional Neural Networks (CNNs). Previous pruning methods often zero out unimportant channels in training in a deterministic manner, which reduces CNN’s learning capacity and results in suboptimal performance. To address this problem, we develop a probability-based pruning algorithm, called batch whitening channel pruning (BWCP), which can stochastically discard unimportant channels by modeling the probability of a channel being activated. BWCP has several merits. (1) It simultaneously trains and prunes CNNs from scratch in a probabilistic way, exploring larger network space than deterministic methods. (2) BWCP is empowered by the proposed batch whitening tool, which is able to empirically and theoretically increase the activation probability of useful channels while keeping unimportant channels unchanged without adding any extra parameters and computational cost in inference. (3) Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet with various network architectures show that BWCP outperforms its counterparts by achieving better accuracy given limited computational budgets. For example, ResNet50 pruned by BWCP has only 0.58% Top-1 accuracy drop on ImageNet, while reducing 42.9% FLOPs of the plain ResNet50.

heading: 1 INTRODUCTION
text: Deep convolutional neural networks (CNNs) have achieved superior performance in a variety of computer vision tasks such as image recognition (He et al., 2016), object detection (Ren et al., 2017), and semantic segmentation (Chen et al., 2018). However, despite their great success, deep CNN models often have massive demand on storage, memory bandwidth, and computational power (Han & Dally, 2018), making them difficult to be plugged onto resource-limited platforms, such as portable and mobile devices (Deng et al., 2020). Therefore, proposing efficient and effective model compression methods has become a hot research topic in the deep learning community.
Model pruning, as one of the vital model compression techniques, has been extensively investigated. It reduces model size and computational cost by removing unnecessary or unimportant weights or channels in a CNN (Han et al., 2016). For example, many recent works (Wen et al., 2016; Guo et al., 2016) prune fine-grained weights of filters. Han et al. (2015) proposes to discard the weights that have magnitude less than a predefined threshold. Guo et al. (2016) further utilizes a sparse mask on a weight basis to achieve pruning. Although these unstructured pruning methods achieve optimal pruning schedule, they do not take the structure of CNNs into account, preventing them from being accelerated on hardware such as GPU for parallel computations (Liu et al., 2018).
To achieve efficient model storage and computations, we focus on structured channel pruning (Wen et al., 2016; Yang et al., 2019a; Liu et al., 2017), which removes entire structures in a CNN such as filter or channel. A typical structured channel pruning approach commonly contains three stages, including pre-training a full model, pruning unimportant channels by the predefined criteria such as `p norm, and fine-tuning the pruned model (Liu et al., 2017; Luo et al., 2017), as shown in Fig.1 (a). However, it is usually hard to find a global pruning threshold to select unimportant channels, because the norm deviation between channels is often too small (He et al., 2019). More importantly, as some channels are permanently zeroed out in the pruning stage, such a multi-stage procedure usually not only relies on hand-crafted heuristics but also limits the learning capacity (He et al., 2018a; 2019).
To tackle the above issues, we propose a simple but effective probability-based channel pruning framework, named batch-whitening channel pruning (BWCP), where unimportant channels are pruned in a stochastic manner, thus preserving the channel space of CNNs in training (i.e. the diversity of CNN architectures is preserved). To be specific, as shown in Fig.1 (b), we assign each channel with an activation probability (i.e. the probability of a channel being activated), by exploring the properties of the batch normalization layer (Ioffe & Szegedy, 2015; Arpit et al., 2016). A larger activation probability indicates that the corresponding channel is more likely to be preserved.
We also introduce a capable tool, termed batch whitening (BW), which can increase the activation probability of useful channels, while keeping the unnecessary channels unchanged. By doing so, the deviation of the activation probability between channels is explicitly enlarged, enabling BWCP to identify unimportant channels during training easily. Such an appealing property is justified by theoretical analysis and experiments. Furthermore, we exploit activation probability adjusted by BW to generate a set of differentiable masks by a soft sampling procedure with Gumbel-Softmax technique, allowing us to train BWCP in an online “pruning-from-scratch” fashion stably. After training, we obtain the final compact model by directly discarding the channels with zero masks.
The main contributions of this work are three-fold. (1) We propose a probability-based channel pruning framework BWCP, which explores a larger network space than deterministic methods. (2) BWCP can easily identify unimportant channels by adjusting their activation probabilities without adding any extra model parameters and computational cost in inference. (3) Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet datasets with various network architectures show that BWCP can achieve better recognition performance given the comparable amount of resources compared to existing approaches. For example, BWCP can reduce 68.08% Flops by compressing 93.12% parameters of VGG-16 with merely accuracy drop and ResNet-50 pruned by BWCP has only 0.58% top-1 accuracy drop on ImageNet while reducing 42.9% FLOPs.

heading: 2 RELATED WORK
text: Weight Pruning. Early network pruning methods mainly remove the unimportant weights in the network. For instance, Optimal Brain Damage (LeCun et al., 1990) measures the importance of weights by evaluating the impact of weight on the loss function and prunes less important ones. However, it is not applicable in modern network structure due to the heavy computation of the Hessian matrix. Recent work assesses the importance of the weights through the magnitude of the weights itself. Specifically, (Guo et al., 2016) prune the network by encouraging weights to become exactly zero. The computation involves weights with zero can be discarded. However, a major drawback of weight pruning techniques is that they do not take the structure of CNNs into account, thus failing to help scale pruned models on commodity hardware such as GPUs (Liu et al., 2018; Wen et al., 2016).
Channel Pruning. Channel pruning approaches directly prune feature maps or filters of CNNs, making it easy for hardware-friendly implementation. For instance, relaxed `0 regularization (Louizos
et al., 2017) and group regularizer (Yang et al., 2019a) impose channel-level sparsity, and filters with small value are selected to be pruned. Some recent work also propose to rank the importance of filters by different criteria including `1 norm (Liu et al., 2017; Li et al., 2017), `2 norm (Frankle & Carbin, 2018) and High Rank channels (Lin et al., 2020). For example, (Liu et al., 2017) explores the importance of filters through scale parameter γ in batch normalization. Although these approaches introduce minimum overhead to the training process, they are not trained in an end-to-end manner and usually either apply on a pre-trained model or require an extra fine-tuning procedure.
Recent works tackle this issue by pruning CNNs from scratch. For example, FPGM (He et al., 2019) zeros in unimportant channels and continues training them after each training epoch. Furthermore, both SSS and DSA learn a differentiable binary mask that is generated by channel importance and does not require any additional fine-tuning. Our proposed BWCP is most related to variational pruning (Zhao et al., 2019) and SCP (Kang & Han, 2020) as they also employ the property of normalization layer and associate the importance of channel with probability. The main difference is that our method adopts the idea of whitening to perform channel pruning. We will show that the proposed batch whitening (BW) technique can adjusts the activation probability of different channels according to their importance, making it easy to identify unimportant channels. Although previous work SPP(Wang et al., 2017) and DynamicCP (Gao et al., 2018) also attempt to boost salient channels and skip unimportant ones, they fail to consider the natural property inside normalization layer and deign the activation probability empirically .

heading: 3 PRELIMINARY
text: Notation. We use regular letters, bold letters, and capital letters to denote scalars such as ‘x’, and vectors (e.g.vector, matrix, and tensor) such as ‘x’ and random variables such as ‘X’, respectively.
We begin with introducing a building layer in recent deep neural nets which typically consists of a convolution layer, a batch normalization (BN) layer, and a rectified linear unit (ReLU) (Ioffe & Szegedy, 2015; He et al., 2016). Formally, it can be written by
xc = wc ∗ z, x̃c = γcx̄c + βc, yc = max{0, x̃c} (1) where c ∈ [C] denotes channel index and C is channel size. In Eqn.(1), ‘∗’ indicates convolution operation and wc is filter weight corresponding to the c-th output channel, i.e. xc ∈ RN×H×W . To perform normalization, xc is firstly standardized to x̄c through x̄c = (xc − E[xc])/ √ D[xc] where E[·] and D[·] indicate calculating mean and variance over a batch of samples, and then is re-scaled to x̃c by scale parameter γc and bias βc. Moreover, the output feature yc is obtained by ReLU activation that discards the negative part of x̃c.
Criterion-based channel pruning. For channel pruning, previous methods usually employ a ‘smallnorm-less-important’ criterion to measure the importance of channels. For example, BN layer can be applied in channel pruning (Liu et al., 2017), where a channel with a small value of γc would be removed. The reason is that the c-th output channel x̃c contributes little to the learned feature representation when γc is small. Hence, the convolution in Eqn.(1) can be discarded safely, and filter wc can thus be pruned. Unlike these criterion-based methods that deterministically prune unimportant filters and rely on a heuristic pruning procedure as shown in Fig.1(a), we explore a probability-based channel pruning framework where less important channels are pruned in a stochastic manner.
Activation probability. To this end, we define an activation probability of a channel by exploring the property of the BN layer. Those channels with a larger activation probability could be preserved with a higher probability. To be specific, since x̄c is acquired by subtracting the sample mean and being divided by the sample variance, we can treat each channel feature as a random variable following standard Normal distribution (Arpit et al., 2016), denoted as X̄c. Note that only positive parts can be activated by ReLU function. Proposition 1 gives the activation probability of the c-th channel, i.e.P (X̃c) > 0.
Proposition 1 Let a random variable X̄c ∼ N (0, 1) and Yc = max{0, γcX̄c + βc}. Then we have (1) P (Yc > 0) = P (X̃c > 0) = (1 + Erf(βc/( √ 2|γc|))/2 where Erf(x) = ∫ x 0 2/ √ π · exp−t2dt, and (2) P (X̃c > 0) = 0⇔ βc ≤ 0 and γc → 0. Note that a pruned channel can be modelled by P (X̃c > 0) = 0. With Proposition 1 (see proof in Appendix A.2), we know that the unnecessary channels satisfy that γc approaches 0 and βc is
negative. To achieve channel pruning, previous compression techniques (Li et al., 2017; Zhao et al., 2019) merely impose a regularization on γc, which would deteriorate the representation power of unpruned channels (Perez et al., 2018; Wang et al., 2020). Instead, we adopt the idea of whitening to build a probabilistic channel pruning framework where unnecessary channels are stochastically disgarded with a small activation probability while important channels are preserved with a large activation probability.

heading: 4 BATCH WHITENING CHANNEL PRUNING
text: This section introduces the proposed batch whitening channel pruning (BWCP) algorithm, which contains a batch whitening module that can adjust the activation probability of channels, and a soft sampling module that stochastically prunes channels with the activation probability adjusted by BW. The whole pipeline of BWCP is illustrated in Fig.2.
By modifying the BN layer in Eqn.(1), we have the formulation of BWCP, xoutc = x̂c︸︷︷︸
batch whitening
mc(P (X̂c > 0))︸ ︷︷ ︸ soft sampling
(2)
where xoutc , x̂c ∈ RN×H×W denote the output of proposed BWCP algorithm and BW module, respectively. ‘ ’ denotes broadcast multiplication. mc ∈ [0, 1] denotes a soft sampling that takes the activation probability of output features of BW (i.e.P (X̂c > 0)) and returns a soft mask. The closer the activation probability is to 0 or 1, the more likely the mask is to be hard. To distinguish important channels from unimportant ones, BW is proposed to increase the activation probability of useful channels while keeping the probability of unimportant channels unchanged during training. Since Eqn.(2) always retain all channels in the network, our BWCP can preserve the learning capacity of the original network during training (He et al., 2018a). The following sections present BW and soft sampling module in detail.

heading: 4.1 BATCH WHITENING
text: Unlike previous works (Zhao et al., 2019; Kang & Han, 2020) that simply measure the importance of channels by parameters in BN layer, we attempt to whiten features after BN layer by the proposed BW module. We show that BW can change the activation probability of channels according to their importances without adding additional parameters and computational overhead in inference.
As shown in Fig.2, BW acts after the BN layer. By rewriting Eqn.(1) into a vector form, we have the formulation of BW,
x̂nij = Σ − 12 (γ x̄nij + β) (3)
where x̂nij ∈ RC×1 is a vector of C elements that denote the output of BW for the n-th sample at location (i, j) for all channels. Σ− 1 2 is a whitening operator and Σ ∈ RC×C is the covariance matrix of channel features {x̃c}Cc=1. Moreover, γ ∈ RC×1 and β ∈ RC×1 are two vectors by stacking γc and βc of all the channels respectively. x̄nij ∈ RC×1 is a vector by stacking elements from all channels of x̄ncij into a column vector.
Training and inference. Note that BW in Eqn.(3) requires computing a root inverse of a covariance matrix of channel features after the BN layer. Towards this end, we calculate the covariance matrix Σ within a batch of samples during each training step as given by
Σ = 1
NHW N,H,W∑ n,i,j=1 (γ x̄nij)(γ x̄nij)T = (γγT) ρ (4)
where ρ is a C-by-C correlation matrix of channel features {x̄c}Cc=1 (see details in Appendix A.1). The Newton Iteration is further employed to calculate its root inverse, Σ− 1 2 , as given by the following iterations Σk = 1
2 (3Σk−1 −Σ3k−1Σ), k = 1, 2, · · · , T. (5)
where k and T are the iteration index and iteration number respectively and Σ0 = I is a identity matrix. Note that when ‖I−Σ‖2 < 1, Eqn.(5) converges to Σ− 1 2 (Bini et al., 2005). To satisfy this condition, Σ can be normalized by Σ/tr(Σ) following (Huang et al., 2019), where tr(·) is the trace operator. In this way, the normalized covariance matrix can be written as ΣN = γγT ρ/ ‖γ‖22.
During inference, we use the moving average to calculate the population estimate of Σ̂− 1 2
N by
following the updating rules, Σ̂− 1 2 N = (1− g)Σ̂ − 12 N + gΣ − 12 N . Here ΣN is the covariance calculated within each mini-batch at each training step, and g denotes the momentum of moving average. Note that Σ̂− 1 2
N is fixed during inference, the proposed BW does not introduce extra costs in memory or
computation since Σ̂− 1 2
N can be viewed as a convolution kernel with size of 1, which can be absorbed into previous convolutional layer. For completeness, we also analyze the training overhead of BWCP in Appendix Sec.A.3 where we see BWCP introduces a little extra training overhead.

heading: 4.2 ANALYSIS OF BWCP
text: In this section, we show that BWCP can easily identify unimportant channels by increasing the difference of activation between important and unimportant channels.
Proposition 2 Let a random variable X̄ ∼ N (0, 1) and Yc = max{0, [Σ̂ − 12 N (γ X̄ + β)]c}.
Then we have P (Yc > δ) = P (X̂c > δ) = (1 + Erf((β̂c − δ)/( √
2|γ̂c|))/2, where δ is a small positive constant, γ̂c and β̂c are two equivalent scale parameter and bias defined by BW module. Take T = 1 in Eqn.(5) as an example, we have γ̂c = 12 (3γc − ∑C d=1 γ 2 dγcρdc/ ‖γ‖ 2 2), and β̂c = 1 2 (3βc − ∑C d=1 βdγdγcρdc/ ‖γ‖ 2 2) where ρdc is the Pearson’s correlation between channel features x̄c and x̄d.
By Proposition .2, BWCP can adjust activation probability by changing the values of γc and βc in Proposition 1 through BW module (see detail in Appendix A.4). Here we introduce a small positive constant δ to avoid the small activation feature value. To see how BW changes the activation probability of different channels, we consider two cases as shown in Proposition 3.
Case 1: βc ≤ 0 and γc → 0. In this case, the c-th channel of the BN layer would be activated with a small activation probability as it sufficiently approaches zero. We can see from Proposition 3, the activation probability of c-th channel still approaches zero after BW is applied, showing that the proposed BW module can keep the unimportant channels unchanged in this case. Case 2: |γc| > 0. For this case, the c-th channel of the BN layer would be activated with a high activation probability. From Proposition 3, the activation probability of c-th channel is enlarged after BW is applied. Therefore, our proposed BW module can increase the activation probability of important channels. Detailed proof of Proposition 3 can be found in Appendix A.5. We also empirically verify Proposition 3 in Sec. 5.3. Notice that we neglect a trivial case in which the channel can be also activated (i.e.βc > 0 and |γc| → 0). In fact, the channels can be removed in this case because the channel feature is always constant which can be deemed as a bias.

heading: 4.3 SOFT SAMPLING MODULE
text: The soft sampling procedure samples the output of BW through a set of differentiable masks. To be specific, as shown in Fig.2, we leverage the Gumbel-Softmax sampling (Jang et al., 2017) that takes the activation probability generated by BW and produces a soft mask as given by
mc = GumbelSoftmax(P (X̂c > 0); τ) (6)
where τ is the temperature. By Eqn.(2) and Eqn.(6), BWCP stochastically prunes unimportant channels with activation probability. A smaller activation probability makes mc more likely to be
close to 0. Hence, our proposed BW can help identify less important channels by enlarging the activation probability of important channels, as mentioned in Sec.4.2. Note that mc can converge to 0-1 mask when τ approaches to 0. In the experiment, we find that setting τ = 0.5 is enough for BWCP to achieve hard pruning at test time.
Proposition 3 Let δ = ‖γ‖2 √∑C j=1(γjβc − γcβj)2ρ2cj/(‖γ‖ 2 2 − ∑C j=1 γ 2 j ρcj). With γ̂ and β̂
defined in Proposition 2, we have (1) P (X̂c > δ) = 0 if |γc| → 0 and βc ≤ 0, and (2) P (X̂c > δ) ≥ P (X̃c ≥ δ) if |γc| > 0. Solution to residual issue. Note that the number of channels in the last convolution layer must be the same as previous blocks due to the element-wise summation in the recent advanced CNN architectures (He et al., 2016; Huang et al., 2017). We solve this problem by letting BW layer in the last convolution layer and shortcut share the same mask as discussed in Appendix A.6.

heading: 4.4 TRAINING OF BWCP
text: This section introduces a sparsity regularization, which makes the model compact, and then describes the training algorithm of BWCP.
Sparse Regularization. With Proposition.1, we see a main characteristic of pruned channels in BN layer is that γc sufficiently approaches 0, and βc is negative. By Proposition 3, we find that it is also a necessary condition that a channel can be pruned after BW module is applied. Hence, we obtain unnecessary channels by directly imposing a regularization on γc and βc as given by
Lsparse = ∑C
c=1 λ1|γc|+ λ2βc (7)
where the first term makes γc small, and the second term encourages βc to be negative. The above sparse regularizer is imposed on all BN layers of the network. By changing the strength of regularization (i.e.λ1 and λ2), we can achieve different pruning ratios. In fact, βc and |γc| represent the mean and standard deviation of a Normal distribution, respectively. Following the empirical rule of Normal distribution, setting λ1 as triple or double λ2 would be a good choice to encourage sparse channels in implementation. Moreover, we observe that 42.2% and 41.3% channels with βc ≤ 0 , while 0.47% and 5.36% channels with |γc| < 0.05 on trained plain ResNet-34 and ResNet-50. Hence, changing the strength of regularization on γc will affect FLOPs more than that of βc. If one wants to pursue a more compact model, increasing λ1 is more effective than λ2.
Training Algorithm. BWCP can be easily plugged into a CNN by modifying the traditional BN operations. Hence, the training of BWCP can be simply implemented in existing software platforms such as PyTorch and TensorFlow. In other words, the forward propagation of BWCP can be represented by Eqn.(2-3) and Eqn.(6), all of which define differentiable transformations. Therefore, our proposed BWCP can train and prune deep models in an end-to-end manner. Appendix A.7 also provides the explicit gradient back-propagation of BWCP. On the other hand, we do not introduce extra parameters to learn the pruning mask mc. Instead, mc in Eqn.(6) is totally determined by the parameters in BN layers including γ,β and Σ. Hence, we can perform joint training of pruning mask mc and model parameters. The BWCP framework is provided in Algorithm 1 of Appendix Sec A.6
Final architecture. The final architecture is fixed at the end of training. During training, we use the Gumbel-Softmax procedure by Eqn.(6) to produce a soft mask. At test time, we instead use a hard 0-1 mask achieved by a sign function (i.e. sign(P (X̂c > 0) > 0.5)) to obtain the network’s output. To make the inference stage stable, we use a sigmoid-alike transformation to make the activation probability approach 0 or 1 in training. By this strategy, we find that both the training and inference stage are stable and obtain a fixed compact model. After training, we obtain the final compact model by directly pruning channels with a mask value of 0. Therefore, our proposed BWCP does not need an extra fine-tuning procedure.

heading: 5 EXPERIMENTS
text: In this section, we extensively experiment with the proposed BWCP on CIFAR-10/100 and ImageNet. We show the advantages of BWCP in both recognition performance and FLOPs reduction comparing with existing channel pruning methods. We also provide an ablation study to analyze the proposed framework. The details of datasets and training configurations are provided in Appendix B.

heading: 5.1 RESULTS ON CIFAR-10
text: For CIFAR-10 dataset, we evaluate our BWCP on ResNet-56, DenseNet-40 and VGG-16 and compare our approach with Slimming (Liu et al., 2017), Variational Pruning (Zhao et al., 2019) and SCP (Kang & Han, 2020). These methods prune redundant channels using BN layers like our algorithm. We also compare BWCP with previous strong baselines such as AMC (He et al., 2018b) and DCP (Zhuang et al., 2018). The results of slimming are obtained from SCP (Kang & Han, 2020). As mentioned in Sec.4.2, our BWCP adjusts their activation probability of different channels. Therefore, it would present better recognition accuracy with comparable computation consumption by entirely exploiting important channels. As shown in Table 1, our BWCP achieves the lowest accuracy drops and comparable FLOPs reduction compared with existing channel pruning methods in all tested base networks. For example, although our model is not fine-tuned, the accuracy drop of the pruned network given by BWCP based on DenseNet-40 and VGG-16 outperforms Slimming with fine-tuning by 1.41% and 0.91% points, respectively. And ResNet-56 pruned by BWCP attains better classification accuracy than previous strong baseline DCP AMC (He et al., 2018b) and DCP (Zhuang et al., 2018) without an extra fine-tuning stage. Besides, our method achieves superior accuracy compared to the Variational Pruning even with significantly smaller model sizes on DensNet-40 and VGGNet-16, demonstrating its effectiveness. We also test BWCP with MobileNet-V2 on the CIFAR10 dataset. From Table 1, we see that BWCP achieves better classification accuracy while reducing more FLOPs We also report results of BWCP on CIFAR100 in Appendix B.3.

heading: 5.2 RESULTS ON IMAGENET
text: For ImageNet dataset, we test our proposed BWCP on two representative base models ResNet-34 and ResNet-50. The proposed BWCP is compared with SFP (He et al., 2018a)), FPGM (He et al., 2019), SSS (Huang & Wang, 2018)), SCP (Kang & Han, 2020) HRank (Lin et al., 2020) and DSA (Ning et al., 2020) since they prune channels without an extra fine-tuning stage. As shown in Table 2, we see that BWCP consistently outperforms its counterparts in recognition accuracy under comparable FLOPs. For ResNet-34, FPGM (He et al., 2019) and SFP (He et al., 2018a) without fine-tuning accelerates ResNet-34 by 41.1% speedup ratio with 2.13% and 2.09% accuracy drop respectively, but our BWCP without finetuning achieve almost the same speedup ratio with only 1.16% top-1 accuracy drop. On the other hand, BWCP also significantly outperforms FPGM (He et al., 2019) by 1.07% top-1 accuracy after going through a fine-tuning stage. For ResNet-50, BWCP still achieves better performance compared with other approaches. For instance, at the level of 40% FLOPs reduction, the top-1 accuracy of BWCP exceeds SSS (Huang & Wang, 2018) by 3.72%. Moreover, BWCP outperforms DSA (Ning et al., 2020) by top-1 accuracy of 0.34% and 0.21% at level of 40% and 50% FLOPs respectively. However, BWCP has slightly lower top-5 accuracy than DSA (Ning et al., 2020).
Inference Acceleration. We analyze the realistic hardware acceleration in terms of GPU and CPU running time during inference. The CPU type is Intel Xeon CPU E5-2682 v4, and the GPU is
NVIDIA GTX1080Ti. We evaluate the inference time using ResNet-50 with a mini-batch of 32 (1) on GPU (CPU). GPU inference batch size is larger than CPU to emphasize our method’s acceleration on the highly parallel platform as a structured pruning method. We see that BWCP has 29.2% inference time reduction on GPU, from 48.7ms for base ResNet-50 to 34.5ms for pruned ResNet-50, and 21.2% inference time reduction on CPU, from 127.1ms for base ResNet-50 to 100.2ms for pruned ResNet-50.

heading: 5.3 ABLATION STUDY
text: Effect of BWCP on activation probability. From the analysis in Sec. 4.2, we have shown that BWCP can increase the activation probability of useful channels while keeping the activation probability of unimportant channels unchanged through BW technique. Here we demonstrate this using Resnet-34 and Resnet-50 trained on ImageNet dataset. We calculate the activation probability of channels of BN and BW layer. It can be seen from Fig.3 (a-d) that (1) BW increases the activation probability of important channels when |γc| > 0; (2) BW keeps the the activation probability of unimportant channels unchanged when βc ≤ 0 and γc → 0. Therefore, BW indeed works by making useful channels more important and unnecessary channels less important, respectively. In this way, BWCP can identify unimportant channels reliably.
Effect of BW, Gumbel-Softmax (GS) and sparse Regularization (Reg). The proposed BWCP consists of three components including BW module (i.e. Eqn. (3)) and Soft Sampling module with Gumbel-Softmax (i.e. Eqn. (6)) and a spare regularization (i.e. Eqn. (7)). Here we investigate the effect of each component. To this end, five variants of BWCP are considered: (1) only BW module is used; (2) only sparse regularization is imposed; (3)BWCP w/o BW module; (4) BWCP w/o sparse regularization; and (5) BWCP with Gumbel-Softmax replaced by Straight Through Estimator (STE) (Bengio et al., 2013). For case (5), we select channels by hard 0-1 mask generated with mc = sign(P (X̂c > 0) − 0.5) 1. The gradient is back-propagated through STE. From results on Table 3, we can make the following conclusions: (a) BW improves the recognition performance, implying that it can enhance the representation of channels; (b) sparse regularization on γ and β slightly harm the classification accuracy of original model but it encourages channels to be sparse as also shown in Proposition 3; (c) BWCP with Gumbel-Softmax achieves higher accuracy than STE, showing that a soft sampling technique is better than the deterministic ones as reported in (Jang et al., 2017).
1y = sign(x) = 1 if x ≥ 0 and 0 if x < 0.
Impact of regularization strength λ1 and λ2. We analyze the effect of regularization strength λ1 and λ2 for sparsity loss on CIFAR-100. The trade-off between accuracy and FLOPs reduction is investigated using VGG-16. Table 4 illustrates that the network becomes more compact as λ1 and λ2 increase, implying that both terms in Eqn.(7) can make channel features sparse. Moreover, the flops metric is more sensitive to the regularization on γ, which validates our analysis in Sec.4.2). Besides, we should search for proper values for λ1 and λ2 to trade off between accuracy and FLOPs reduction, which is a drawback for our method.
Effect of the number of BW. Here the effect of the number of BW modules of BWCP is investigated trained on CIFAR-10 using Resnet-56 consisting of a series of bottleneck structures. Note that there are three BN layers in each bottleneck. We study four variants of BWCP:
(a) we use BW to modify the last BN in each bottleneck module; hence there are a total of 18 BW layers in Resnet-56; (b) the last two BN layers are modified by our BW technique (36 BW layers) (c) All BN layers in bottlenecks are replaced by BW (54 BW layers), which is our proposed method. The results are reported in Table 5. We can see that BWCP achieves the best top-1 accuracy when BW acts on all BN layers, given the comparable FLOPs and model
size. This indicates that the proposed BWCP more benefits from more BW layers in the network.
BWCP selects representative channel features. It is worth noting that BWCP can whiten channel features after BN through BW as shwon in Eqn.(3). Therfore, BW can learn diverse channel features by reducing the correlations among channels(Yang et al., 2019b). We investigate this using VGGNet16 with BN and the proposed BWCP trained on CIFAR-10. The correlation score can be calculated by taking the average over the absolute value of the correlation matrix of channel features. A larger value indicates that there is redundancy in the encoded features. We plot the correlation score among channels at different depths of the network. As shown in Fig.3 (e & f), channel features after BW block have significantly smaller correlations, implying that channels selected by BWCP are representative. This also accounts for the effectiveness of the proposed scheme.

heading: 6 DISCUSSION AND CONCLUSION
text: This paper presented an effective and efficient pruning technique, termed Batch Whitening Channel Pruning (BWCP). We show BWCP increases the activation probability of useful channels while keeping unimportant channels unchanged, making it appealing to pursue a compact model. Particularly, BWCP can be easily applied to prune various CNN architectures by modifying the batch normalization layer. However, to achieve different levels of FLOPs reduction, the proposed BWCP needs to search for the strength of sparse regularization. With probabilistic formulation in BWCP, the expected FLOPs can be modeled. The multiplier method can be used to encourage the model to attain target FLOPs. For future work, an advanced Pareto optimization algorithm can be designed to tackle such multi-objective joint minimization. We hope that the analyses of BWCP could bring a new perspective for future work in channel pruning.
Ethics Statement. We aim at compressing neural nets by the proposed BWCP framework. It could improve the energy efficiency of neural network models and reduce the emission of carbon dioxide. We notice that deep neural networks trained with BWCP can be plugged into portable or edge devices such as mobile phones. Hence, our work and AI in edge devices would have the same negative impact on ethics. Moreover, network pruning may have different effects on different classes, thus producing unfair models as a result. We will carefully investigate the results of our method on the fairness of the model output in the future.
Reproducibility Statement. For theoretical results, clear explanations of assumptions and a complete proof of propostion 1-3 are included in Appendix. To reproduce the experimental results, we provide training details and hyper-parameters in Appendix Sec.B. Moreover, we will also make our code available by a link to an anonymous repository during the discussion stage.

heading: A MORE DETAILS ABOUT APPROACH

heading: A.1 CALCULATION OF COVARIANCE MATRIX Σ
text: By Eqn.(1) in main text, the output of BN is x̃ncij = γcx̄ncij + βc. Hence, we have E[x̃c] = 1 NHW ∑N,H,W n,i,j (γcx̄ncij + βc) = βc. Then the entry in c-th row and d-th column of covariance matrix Σ of x̃c is calculated as follows:
Σcd = 1
NHW N,H,W∑ n,i,j (γcx̄ncij + βc − E[x̃c])(γdx̄ndij + βd − E[x̃d]) = γcγdρcd (8)
where ρcd is the element in c-th row and j-th column of correlation matrix of x̄. Hence, we have ρcd ∈ [−1, 1]. Furthermore, we can write Σ into the vector form: Σ = γγT 1NHW ∑N,H,W n,i,j x̄nijx̄nij T = γγT ρ.

heading: A.2 PROOF OF PROPOSITION 1
text: For (1), we notice that we can define γc = −γc and X̄c = −X̄c ∼ N (0, 1) if γc < 0. Hence, we can assume γc > 0 without loss of generality. Then, we have
P (Yc > 0) = P (X̃c > 0) = P (X̄c > − βc γc )
= ∫ +∞ − βcγc 1√ 2π exp− t2 2 dt
= ∫ 0 − βcγc 1√ 2π exp− t2 2 dt+ ∫ +∞ 0 1√ 2π exp− t2 2 dt
=
∫ βc γc
0
1√ 2π exp− t2 2 dt+ ∫ +∞ 0 1√ 2π exp− t2 2 dt
= Erf( βc√ 2γc ) + 1
2
(9)
When γc < 0, we can set γc = −γc. Hence, we arrive at
P (Yc > 0) = P (X̃c > 0) = Erf( βc√ 2|γc| ) + 1
2 (10)
For (2), let us denote X̄c ∼ N (0, 1), and X̃c = γcX̄c + βc and Yc = max{0, X̃c} where Yc represents a random variables corresponding to output feature yc in Eqn.(1) in main text. Firstly, it is easy to see that P (X̃c > 0) = 0⇔ EX̄c [Yc] = 0 and EX̄c [Y 2 c ] = 0. In the following we show that EX̄c [Yc] = 0 and EX̄c [Y 2 c ] = 0⇔ βc ≤ 0 and γc = 0. Similar with (1), we assume γc > 0 without loss of generality.
For the sufficiency, we have EX̄c [Yc] = ∫ − βcγc −∞ 0 · 1√ 2π exp− x̄2c 2 dx̄c + ∫ +∞ − βcγc (γcx̄c + βc) · 1√ 2π exp− x̄2c 2 dx̄c,
= γcexp
− β 2 c
2γ2c
√ 2π
+ βc 2 (1 + Erf[ βc√ 2γc ]),
(11)
where Erf[x] = 2√ π ∫ x 0 exp−t 2 dt is the error function. From Eqn.(11), we have
lim γc→0+ EX̄c [Yc] = lim γc→0+
γcexp − β
2 c 2γ2c
√ 2π
+ lim γc→0+ βc 2 (1 + Erf[ βc√ 2γc ]) = 0 (12)
In the same way, we can calculate
Ex̄c [Y 2c ] = ∫ − βcγc −∞ 0 · 1√ 2π exp− x̄2c 2 dx̄c + ∫ +∞ − βcγc (γcx̄c + βc) 2 · 1√ 2π exp− x̄2c 2 dx̄c,
= γcβcexp
− β 2 c
2γ2c
√ 2π
+ γ2c + β 2 c
2 (1 + Erf[ βc√ 2γc ]),
(13)
From Eqn.(13), we have
lim γc→0+ Ex̄c [Y 2c ] = lim γc→0+
γcβcexp − β
2 c 2γ2c
√ 2π
+ lim γc→0+
γ2c + β 2 c
2 (1 + Erf[ βc√ 2γc ]) = 0 (14)
For necessity, we show that if Ex̄c [Yc] = 0 and Ex̄c [Yc 2] = 0, then γc → 0 and βc ≤ 0. In essence, It can be acquired by solving Eqn Eqn.(11) and Eqn.(13). To be specific, βc∗Eqn.(.11)−Eqn.(13) gives us γc = 0+. Substituting it into Eqn.(.11), we can obtain βc ≤ 0. This completes the proof.

heading: A.3 TRAINING OVERHEAD OF BWCP
text: The proposed BWCP introduces a little extra computational cost during training. To see this, we evaluate the computational complexity of SCP and BWCP for ResNet50 on ImageNet with an input image size of 224× 224. We can see from the table below that the training BWCP is slightly slower on both CPU and GPU than the plain ResNet and SCP. In fact, the computational burden mainly comes from calculating the covariance matrix and its root inverse. In our paper, we calculate the root inverse of the covariance matrix by Newton’s iteration, which is fast and efficient. Although BWCP brings extra training overhead, it achieves better top-1 accuracy drop under the same FLOPs consumption.

heading: A.4 PROOF OF PROPOSITION 2
text: First, we can derive that X̂c = Σ − 12 N (γ X̄+β) = Σ − 12 N (γ X̄)+Σ − 12 N β = (Σ − 12 N γ) X̄+Σ − 12 N β.
Hence, the newly defined scale and bias parameters are γ̂ = Σ− 1 2 N γ and β̂ = Σ − 12 N β. When T = 1, we have Σ− 1 2
N = 1 2 (3I−ΣN ) by Eqn.(5) in main text. Hence we obtain,
γ̂ = 1
2 (3I−ΣN )γ =
1 2 (3I− γγ
T
‖γ‖22 ρ)γ
= 1 2 (3γ − 1
‖γ‖22  C∑ j=1 γ1γjρ1jγj , · · · , C∑ j=1 γCγjρCjγj T) = 1
2 (3− C∑ j=1 γ2j ρ1j ‖γ‖22 )γ1, · · · , (3− C∑ j=1 γ2j ρCj ‖γ‖22 )γC T (15)
Similarly, β̂ can be given by
β̂ = 1 2 (3I−Σ)β = 1 2 (3I− γγ
T
‖γ‖22 ρ)β
= 1 2 (3β − 1
‖γ‖22  C∑ j=1 γ1γjρ1jβj , · · · , C∑ j=1 γCγjρCjβj T) = 1
2 3β1 − ( C∑ j=1 γjβjρ1j ‖γ‖22 )γ1, · · · , 3βC − ( C∑ j=1 γjβjρCj ‖γ‖22 )γC T (16)
Taking each component of vector Eqn.(15-16) gives us the expression of γ̂c and β̂c in Proposition 2.

heading: A.5 PROOF OF PROPOSITION 3
text: For (1), through Eqn.(15), we acquire |γ̂c| = 12 |3− ∑C j=1 γ2j ρcj
‖γ‖22 ||γc|. Therefore, |γ̂c| → 0 if |γc| → 0.
On the other hand, by Eqn.(16), we have β̂c ≈ 12 (3 − γ2c ‖γ‖22
)βc < βc ≤ 0. Here we assume that ρcd = 1 if c = d and 0 otherwise. Note that the assumption is plausible by Fig.3 (e & f) in main text from which we see that the correlation among channel features will gradually decrease during training. We also empirically verify these two conclusions by Fig.4. From Fig.4 we can see that |γ̂c| ≥ |γc| where the equality holds iff |γc| = 0, and β̂c is larger than βc if βc is positive, and vice versa. By Proposition 1, we arrive at
P (X̂c > δ) < P (X̂c > 0) = 0 (17)
where the first ‘>’ holds since δ is a small positive constant and ‘=’ follows from |γ̂c| → 0 and β̂c ≤ 0. For (2), to show P (X̂c > δ) > P (X̃c > δ), we only need to prove P (X̄c > δ−β̂c|γ̂c| ) > P (X̄c > δ−βc |γc| ), which is equivalent to δ−β̂c |γ̂c| < δ−βc |γc| . To this end, we calculate
|γc|β̂c − |γ̂c|βc |γ̂c| − |γc|
= |γc| 12 (3βc − (
∑C j=1
γjβjρcj ‖γ‖22 )γc)− 12 (3− ∑C j=1 γ2j ρcj ‖γ‖22 )|γc|βc
1 2 (3− ∑C j=1 γ2j ρcj
‖γ‖22 )|γc| − |γc|
=
∑C j=1
γjβjγcρcj ‖γ‖22
− ∑C j=1 γ2j βcρcj
‖γ‖22 1− ∑C j=1 γ2j ρcj
‖γ‖22
=
∑C j=1
γj(βjγc−γjβc)ρcj ‖γ‖22
1− ∑C j=1 γ2j ρcj
‖γ‖22
≤ 1 ‖γ‖2
√∑C j=1 (βjγc − γjβc)2ρ2cj
1− ∑C j=1 γ2j ρcj
‖γ‖22
= ‖γ‖2
√∑C j=1 (βjγc − γjβc)2ρ2cj
‖γ‖22 − ∑C j=1 γ 2 j ρcj
= δ
(18)
where the ‘≤’ holds due to the Cauchy–Schwarz inequality. By Eqn.(18), we derive that |γc|(δ − β̂c) ≤ |γ̂c|(δ − βc) which is exactly what we want. Lastly, we empirically verify that δ defined in Proposition 3 is a small positive constant. In fact, δ represents the minimal activation feature value (i.e. X̂c = γ̂cX̄c + β̂c ≥ δ by definition). We visualize the value of δ in shallow and deep layers in ResNet-34 during the whole training stage and value of δ of each layer in trained ResNet-34 on ImageNet dataset in Fig.5. As we can see, δ is always a small positive number in the training process. We thus empirically set δ as 0.05 in all experiments.
To conclude, by Eqn.(17), BWCP can keep the activation probability of unimportant channel unchanged; by Eqn.(18), BWCP can increase the activation probability of important channel. In this way, the proposed BWCP can pursuit a compact deep model with good performance.
Algorithm 1 Forward Propagation of the proposed BWCP. 1: Input: mini-batch inputs x ∈ RN×C×H×W . 2: Hyperparameters: momentum g for calculating root inverse of covariance matrix, iteration number T . 3: Output: the activations xout obtained by BWCP. 4: calculate standardized activation: {x̄c}Cc=1 in Eqn.(1). 5: calculate the output of BN layer: x̃c = γcx̄c + βc. 6: calculate normalized covariance matrix: ΣN = γγ T
‖γ‖22 1 NHW
∑N,H,W n,i,j=1 x̄nij x̄nij T
7: Σ0 = I. 8: for k = 1 to T do 9: Σk = 12 (3Σk−1 −Σ 3 k−1ΣN )
10: end for 11: calculate whitening matrix for training: Σ − 1 2
N = ΣT .
12: calculate whitening matrix for inference: Σ̂ − 1
2 N ← (1− g)Σ̂
− 1 2 N + gΣ − 1 2 N .
13: calculate whitened output: x̂nij = Σ − 1 2 N x̃nij . 14: calculate equivalent scale and bias defined by BW: γ̂ = Σ− 1 2 γ and β̂ = Σ− 1 2 β. 15: calculate the activation probability by Proposition 2 with γ̂ and β̂, obtain soft masks {mc}Cc=1 by Eqn.(6). 16: calculate the output of BWCP: xoutc = x̂c mc.
BN BW
Act. Prob. Gumbel-Softmax Soft Mask ReLU
Soft Gating Module
Cov. Newton Iter.
BN BW
Act. Prob. Gumbel-SoftMax Soft Mask
ReLU Soft Gating Module
Cov.

heading: A.6 SOLUTION TO RESIDUAL ISSUE
text: The recent advanced CNN architectures usually have residual blocks with shortcut connections (He et al., 2016; Huang et al., 2017). As shown in Fig.6, the number of channels in th last convolution layer must be the same as in previous blocks due to the element-wise summation. Basically, there are two types of residual connections, i.e. shortcut with downsampling layer consisting of ConvBN modules, and shortcut with identity. For shortcut with Conv-BN modules, the proposed BW technique is utilized in downsampling layer to generate pruning mask msc. Furthermore, we use a simple strategy that lets BW layer in the last convolution layer and shortcut share the same mask as given by mc = msc ·mlastc where mlastc and msc denote masks of the last convolution layer. For shortcut with identity mappings, we use the mask in the previous layer. In doing so, their activated output channels must be the same.

heading: A.7 BACK-PROPAGATION OF BWCP
text: The forward propagation of BWCP can be represented by Eqn.(3-4) and Eqn.(9) in the main text (see detail in Table 1), all of which define differentiable transformations. Here we provide the back-propagation of BWCP. By comparing the forward representation of BN and BWCP in Fig.7, we need to back-propagate the gradient ∂L
∂xoutnij to ∂L∂x̄nij for backward propagation of BWCP. For
simplicity, we neglect the subscript ‘nij’.
By chain rules, we have
∂L ∂x̄ = γ̂ m ∂L ∂xout + ∂L ∂x̄ (Σ − 12 N ) (19)
where ∂L∂x̄ (Σ − 12 N ) denotes the gradient w.r.t. x̄ back-propagated through Σ − 12 N . To calculate it, we first obtain the gradient w.r.t. Σ− 1 2
N as given by
∂L
∂Σ − 12 N
= γ ∂L ∂γ̂ T + β ∂L ∂β̂ T (20)
where ∂L ∂γ̂ = x̄ m ∂L ∂xout + ∂m ∂γ̂ (x̂ ∂L ∂xout ) (21)
and ∂L ∂β̂ = m + ∂m ∂β̂ (x̂ ∂L ∂xout ) (22)
The remaining thing is to calculate ∂m∂γ̂ and ∂m ∂β̂ . Based on the Gumbel-Softmax transformation, we arrive at
∂mc ∂γ̂d =
{ −mc(1−mc)f(γ̂c,β̂c)
τP (X̂C>0)(1−P (X̂C>0)) βcγc |γc|2 if d = c
0, otherwise (23)
∂mc ∂β̂d =
{ mc(1−mc)f(γ̂c,β̂c)
τP (X̂C>0)(1−P (X̂C>0)) , if d = c
0, otherwise (24)
where f(γ̂c, β̂c) is the probability density function of R.V. X̂c as written in Eqn.(2) of main text.
To proceed, we deliver the gradient w.r.t. Σ− 1 2
N in Eqn.(20) to Σ by Newton Iteration in Eqn.(6) of
main text. Note that Σ− 1 2
N = ΣT , we have
∂L ∂ΣN = −1 2 T∑ k=1 (Σ3k−1) T ∂L ∂Σk
(25)
where ∂L∂Σk can be calculated by following iterations:
∂L ∂Σk−1 = 3 2 ∂L ∂Σk − 1 2 ∂L ∂Σk (Σ2k−1Σ) T − 1 2 (Σ2k−1) T ∂L ∂Σk ΣT
− 1 2 (Σk−1) T ∂L ∂Σk (Σk−1Σ) T k = T, · · · , 1.
Given the gradient w.r.t. ΣN in Eqn.(25), we can calculate the gradient w.r.t. x̄ back-propagated through Σ− 1 2
N in Eqn.(19) as follows
∂L ∂x̄ (Σ − 12 N ) = ( γγT ‖γ‖2 ( ∂L ∂ΣN + ∂L ∂ΣN T))x̄ (26)
Based on Eqn.(19-26), we obtain the back-propagation of BWCP.

heading: B MORE DETAILS ABOUT EXPERIMENT

heading: B.1 DATASET AND METRICS
text: We evaluate the performance of our proposed BWCP on various image classification benchmarks, including CIFAR10/100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015). The CIFAR-10 and CIFAR-100 datasets have 10 and 100 categories, respectively, while both contain 60k color images with a size of 32× 32, in which 50k training images and 10K test images are included. Moreover, the ImageNet dataset consists of 1.28M training images and 50k validation images. Top-1 accuracy are used to evaluate the recognition performance of models on CIFAR 10/100. Top-1 and Top-5 accuracies are reported on ImageNet. We utilize the common protocols, i.e. number of parameters and Float Points Operations (FLOPs) to obtain model size and computational consumption.
For CIFAR-10/100, we use ResNet (He et al., 2016), DenseNet (Huang et al., 2017), and VGGNet (Simonyan & Zisserman, 2014) as our base model. For ImageNet, we use ResNet-34 and ResNet-50. We compare our algorithm with other channel pruning methods without a fine-tuning procedure. Note that a extra fine-tuning process would lead to remarkable improvement of performace (Ye et al., 2020). For fair comparison, we also fine-tune our BWCP to compare with those pruning methods. The training configurations are provided in Appendix B.2. The base networks and BWCP are trained together from scratch for all of our models.

heading: B.2 TRAINING CONFIGURATION
text: Training Setting on ImageNet. All networks are trained using 8 GPUs with a mini-batch of 32 per GPU. We train all the architectures from scratch for 120 epochs using stochastic gradient descent (SGD) with momentum 0.9 and weight decay 1e-4. We perform normal training without sparse regularization in Eqn.(7) on the original networks for first 20 epochs by following (Ning et al., 2020). The base learning rate is set to 0.1 and is multiplied by 0.1 after 50, 80 and 110 epochs. During fine-tuning, we use the standard SGD optimizer with Nesterov momentum 0.9 and weight decay 0.00005 to fine-tune pruned network for 150 epochs. We decay learning rate using cosine schedule with initial learning rate 0.01. The coefficient of sparse regularization λ1 and λ2 are set to 7e-5 and 3.5e-5 to achieve Flops Reduction at a level of 40%, while λ1 and λ2 are set to 9e-5 and 3.5e-5 respectively to achieve FLOPs reduction at a level of 40%. Besides, the covariance matrix in the proposed BW technique is calculated within each GPU. Like (Huang et al., 2019), we also use group-wise decorrelation with group size 16 across the network to improve the efficiency of BW.
Training setting on CIFAR-10 and CIFAR-100. We train all models on CIFAR-10 and CIFAR-100 with a batch size of 64 on a single GPU for 160 epochs with momentum 0.9 and weight decay 1e-4. The initial learning rate is 0.1 and is decreased by 10 times at 80 and 120 epoch. The coefficient of sparse regularization λ1 and λ2 are set to 4e-5 and 8e-5 for CIFAR-10 dataset and 7e-6 and 1.4e-5 for CIFAR-100 dataset.

heading: B.3 MORE RESULTS OF BWCP
text: The results of BWCP on CIFAR-100 dataset is reported in Table 7. As we can see, our approach BWCP achieves the lowest accuracy drops and comparable FLOPs reduction compared with existing channel pruning methods in all tested base models.

Review summaries:
This paper presents a probabilistic channel pruning method (BWCP) for accelerating CNNs. The key newly proposed technique is “batch whitening”. They evaluate their method on CIFAR-10/100 and ImageNet compared to other recent filter pruning methods.

In this paper, authors propose to compress convolutional layers via batch whitening channel pruning (BWCP). By apply batch whitening on incoming feature maps, unimportant channels are automatically identified and stochastically discarded during the training process. A sigmoid-like transformation is used to push activation probabilities to either 0 or 1, so that additional fine-tuning is not needed, unlike many other channel pruning methods.

This paper proposes a way to train compact neural networks. The feature channels are whitened, then combined with a sampling method. At the testing time, this sampling is discarded and replaced with a deterministic selection.   The benefits of whitening in improving the convergence of NNs is widely explored. This paper explores another useful property of whitening,  the sparsity in the whitened results. Since the representation is sparser,  one might use fewer weights to reach similar performance. This paper presents such a solution.