BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening
Authors: Wenqi Shao
Hang Yu
Zhaoyang Zhang
Hang Xu
Zhenguo Li
Ping Luo
Venue: ICLR 2022 Submitted (2022)
Keywords: Classification
Normalization

Abstract: This work presents a probabilistic channel pruning method to accelerate Convolutional Neural Networks (CNNs). Previous pruning methods often zero out unimportant channels in training in a deterministic manner, which reduces CNN's learning capacity and results in suboptimal performance. To address this problem, we develop a probability-based pruning algorithm, called batch whitening channel pruning (BWCP), which can stochastically discard unimportant channels by modeling the probability of a channel being activated. BWCP has several merits. (1) It simultaneously trains and prunes CNNs from scratch in a probabilistic way, exploring larger network space than deterministic methods. (2) BWCP is empowered by the proposed batch whitening tool, which is able to empirically and theoretically increase the activation probability of useful channels while reducing the probability of unimportant channels without adding any extra parameters and computational cost in inference. (3) Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet with various network architectures show that BWCP outperforms its counterparts by achieving better accuracy given limited computational budgets. For example, ResNet50 pruned by BWCP has only 0.58% Top-1 accuracy drop on ImageNet, while reducing 42.9% FLOPs of the plain ResNet50.

Under review as a conference paper at ICLR 2022
BWCP: PROBABILISTIC LEARNING-TO-PRUNT CHAN- NELS FOR CONVNETS VIA BATCH WHITENING
Anonymous authors Paper under double-blind review
ABSTRACT
This work presents a probabilistic channel pruning method to accelerate Con- volutional Neural Networks (CNNs). Previous pruning methods often zero out unimportant channels in training in deterministic manner, which reduces CNN's learning capacity and results in suboptimal performance. To address this problem, we develop a probability-basee pruning algorithm, called batch whitening channel pruning (B (CP), which can stochastically discard unimportant channels by mod- eling the probability of a channel being activated. BWCP has several merits. (1) It simultaneously trains and prunes CNNs from scratch in probabilistic way, explor- ing larger network space than deterministic methods. (2) BWCP is empowered by the proposed batch whitening tool, which is able to empirically and theoretically increase the activation probability of useful channels while keeping unimportant channels unchanged without adding any extra parameters and computational cost in inference. (3) Extensive experiments CIFAR-10, CIFAR- 1000 and ImageNet with various network architectures show that BWCP outperforms its counterparts by achieving better accuracy given limited computational budgets. For example, ResNet50 pruned by BWCP has only 0.58% Top-1 accuracy drop on ImageNet, while reducing 42.9% FLOPs of the plain ResNet50.
INTRODUCTION
Deep convolutional neural networks (CNNs) have achieved superior performance in a variety of computer vision tasks such as image recognition (He et al.) 2016), object detection (Ren al., 2017), and semantic segmentation (Chen et al.) 2018). However, despite their great success, deep CNN models often have massive demand on storage, memory bandwidth, and computational power (Han & Dally 2018), making them difficult to be plugged onto resource-limited platforms, such as portable and mobile devices (Deng et al. 2020 Therefore, proposing efficient and effective model compression methods has become a hot research topic in the deep learning community.
Model pruning, as one of the vital model compression techniques, has been extensively investigated. It reduces model size and computational cost by removing unnecessary unimportant weights or channels in a CNN (Han et al. 2016). For example, many recent works (Wen et al., 2016)] Cuo et al. 2016) prune fine-grained weights of filters. Han al. (2015) proposes to discard the weights that have magnitude less than a predefined threshold. Guo et al. (2016) further utilizes a sparse mask on weight basis achieve pruning. Although these unstructured pruning methods achieve optimal pruning schedule, they do not take the structure of CNNs into account, preventing them from being accelerated on hardware such GPU for parallel computations (Liu al. 2018).
To achieve efficient model storage and computations, we focus on structured channel pruning (Wen al. 2016) Yang et al., 2019a [iu et al. (017), which removes entire structures in a CNN such as filter or channel. A typical structured channel pruning approach commonly contains three stages, including pre-training a full model, pruning unimportant channels by the predefined criteria such as l, norm, and fine-tuning the pruned model (Liu et al.. 2017 Luo et al. 2017), shown in Fig (a). However, it is usually hard to find a global pruning threshold to select unimportant channels, because the norm deviation between channels is often too small (He et al. 2019). More importantly, some channels are permanently zeroed out in the pruning stage, such multi-staee procedure usually not only relies hand-crafted heuristics but also limits the learning capacity (He al. 2018a 2019).

---

Under review as a conference paper at ICLR 2022
Charees
Noom
Thimbedd
Prose
Has--uza
(a) based method
Chanzek
Saft Masios
Sall Maaka
Chosmals
Probshdit
Preze
Probakiitt
Our proposed BWCP
Figure 1: Illustration of our proposed BWCP. (a) Previous channel pruning methods utilize a hard criterion such the norm (Liu et al. 2017) of channels to deterministically remove unimportant channels, which deteriorates performance and needs a extra fine-tuning process(Frankle & Carbin 2018). (b) Our proposed BWCP is probability-base pruning framework where unimportant channels are stochastically pruned with activation probability, thus maintaining the learning capacity of original CNNs. In particular, our proposed batch whitening (BW) tool can increase the activation probability of useful channels while keeping the activation probability of unimportant channels unchanged, enabling BWCP to identify unimportant channels reliably.
To tackle the above issues, we propose simple but effective probability- -baee channel pruning framework, named batch-whitening channel pruning (BWCP), where unimportant channels are pruned in a stochastic manner, thus preserving the channel space of CNNs in training (i.e. the diversity of CNN architectures is preserved). To be specific, as shown in [ (b), we assign each channel with an activation probability (i.e. the probability of channel being activated), by exploring the properties of the batch normalization layer (loffe & Szegedy 2015 Arpit et al. 2016). A larger activation probability indicates that the corresponding channel more likely to be preserved.
We also introduce capable tool, termed batch whitening (BW), which can increase the activation probability of useful channels, while keeping the unnecessary channels unchanged. By doing so, the deviation of the activation probability between channels is explicitly enlarged, enabling BWCP to identify unimportant channels during training easily. Such an appealing property is justified by theoretical analysis and experiments. Furthermore, we exploit activation probability adjusted by BW to generate a set of differentiable masks by a soft sampling procedure with Gumbel-Softman technique, allowing us train BWCP in an online "pruning-froms from-scratchh" fashion stably. After training, we obtaín the final compact model by directly discarding the channels with zero masks.
The main contributions of this work are three-fold. (1) We propose probability basedd channel pruning framework BWCP, which explores a larger network space than deterministic methods. (2) BWCP can easily identify unimportant channels by adjusting their activation probabilities without adding any extra model parameters and computational cost in inference. (3) Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet datasets with various network architectures show that BWCP can achieve better recognition performance given the comparable amount of resources compared to existing approaches. For example, BWCP can reduce 68.08% Flops by compressing 93.12% parameters of VGG-16 16 with merely accuracy drop and ResNet-50 pruned by BWCP has only 0.58% top-1 accuracy drop on ImageNet while reducing 42.9% FL.PPs.
2 RELATED WORK
Weight Pruning. Early network pruning methods mainly remove the unimportant weights in the network. For instance, Optimal Brain Damage (LeCun et al. 1990) measures the importance of weights by evaluating the impact of weight on the loss function and prunes less important ones. However, it not applicable in modern network structure due the heavy computation of the Hessian matrix. Recent work assesses the importance of the weights through the magnitude of the weights itself. Specifically, (Guo et al. 2016) prune the network by encouraging weights to become exactly zero. The computation involves weights with zero can be discarded. However, major drawback of weight pruning techniques is that they do not take the structure of CNNs into account, thus failing to help scale pruned models on commodity hardware such GPUs (Liu al. 2018 Wen et al., 2016).
Channel Pruning, Channel pruning approaches directly prune feature maps or filters of CNNs. making easy for frrewlly implementation. For instance, relaxed to regularization (Louizos)
2

---

Under review as a conference paper at ICLR 2022
et al. 2017) and group regularizer (Yang et al.
2019a) impose channel-level sparsity, and filters with
small value are selected to be pruned. Some recent work also propose to rank the importance of filters by different criteria including & norm (Liu et al., 2017 Li ct al. (017), la norm (Frankle & Carbin 2018) and High Rank channels (Lin et al. 2020). For example, (Liu et al. 2017) explores the importance of filters through scale parameter ? batch normalization. Although these approaches introduce minimum overhead to the training process, they are not trained in an end-to-end manner and usually either apply on a pre-trained model require an extra fine-tuning procedure.
Recent works tackle this issue by pruning CNNs from scratch. For example, FPGM (He et al. 2019) zeros in unimportant channels and continues training them after each training epoch. Furthermore. both SSS and DSA learn a differentiable binary mask that is generated by channel importance and does not require any additional fine-tuning. Our proposed BWCP is most related to variational pruning (Zhao et al., 2019) and SCP (Kang & Han, 2020) as they also employ the property of normalization layer and associate the importance of channel with probability. The main difference is that our method adopts the idea of whitening to perform channel pruning. We will show that the proposed batch whitening (BW) technique can adjusts the activation probability of different channels according to their importance, making it easy identify unimportant channels. Although previous work SPP (Wan et al., (017)) and DynamicCP (Gao et (0118) also attempt to boost salient channels and skip unimportant ones, they fail to consider the natural property inside normalization layer and deign the activation probability empirically.
3 PRELIMINARY
Notation. We use regular letters, bold letters, and capital letters to denote scalars such as 'x', and vectors (e.g. vector, matrix, and tensor) such as 'x' and random variables such as 'X', respectively.
We begin with introducing a building layer in recent deep neural nets which typically consists of a convolution layer, a batch normalization (BN) layer, and a rectified linear unit (ReLU) (loffe & Szegedy 2015 He et al., 2016). Formally, it can be written by
X W _ Tc8e + Be, Yc = max x
where c E [C] denotes channel index and C' is channel size. In Eqn.(] '*" indicates convolution operation and w. is filter weight corresponding to the e-th output channel, i.e. x. E X To perform normalization, x, is firstly standardized to X through x, (x. - B(x,l)/ /Btl)//DDibe DDx. where E[|] and D|: indicate calculating mean and variance over batch of samples, and then is re-scaled to x. by scale parameter "ee and bias Sc. Moreover, the output feature y. is obtained by ReLU activation that discards the negative part of x
Criterion-based channel pruning. For channel pruning, previous methods usually employ 'small- norm-less-impen teaollssssimpoaantt criterion to measure the importance of channels. For example, BN layer can be applied in channel pruning (Liu et al. 2017). where a channel with a small value of ye would be removed. The reason is that the c-th output channel X. contributes little to the learned feature representation when % is small. Hence, the convolution in Eqn.(I) can be discarded safely, and filter w. can thus be pruned. Unlike these criterion hased methods that deterministically prune unimportant filters and rely on heuristic pruning procedure as shown in Fig a)), we explore probability-bane based channel pruning framework where less important channels are pruned in a stochastic manner.
Activation probability. To this end, define an activation probability of channel by exploring the property of the BN layer. Those channels with a larger activation probability could be preserved with higher probability. To be specific, since X, is acquired by subtracting the sample mean and being divided by the sample variance, we can treat each channel feature as a random variable following standard Normal distribution (Arpit et al. 2016), denoted as X.. Note that only positive parts can be activated by ReLU function. Proposition | gives the activation probability of the c-th channel, i.e. P(X)) > O.
Proposition 1 Let a random variable X, ~ (O 1) and Y. = max Xe + 1.Bc). Then we have (I) P(Y. > 0)- = P(X. > 0) = (1+ + (at(ae//////aa/a//e and (2) P(X. 0) = 0 + 3. < 0 and + 0.
where Erf(x)- = So 2/ exp -? dt,
Note that a pruned channel can be modelled by P(X. > 0) = 0. With Proposition (see proof in Appendix A.2), know that the unnecessary channels satisfy that %. approaches and 3, is
3

---

Under review as a conference paper at ICLR 2022
BN
BW
Nevion
R
Sell Gating
P(R 0)
Act. Prob.
G Gumbel-Bee Softmax
Solt Mank
ReLU
A schematic of the proposed Batch Whitening Channel Pruning (BWCP) algorithm that consists of a BW module and a soft sampling procedure. By modifying BN layer with a whitening operator, the proposed BW technique adjusts activation probabilities of different channels. These activation probabilities are then utilized by a soft sampling procedure.
Figure 2:
negative. To achieve channel pruning, previous compression techniques (Li al., 2017: Zhao et al. 2019) merely impose a regularization %co which would deteriorate the representation power of unpruned channels (Perez et al., 2018 Wang et al. 2020). Instead, we adopt the idea of whitening to build a probabilistic channel pruning framework where unnecessary channels are stochastically disgarded with small activation probability while important channels are preserved with a large activation probability.
4 BATCH WHITENING CHANNEL PRUNING
This section introduces the proposed batch whitening channel pruning (BWCP) algorithm, which contains batch whitening module that adjust the activation probability of channels, and a soft sampling module that stochastically prunes channels with the activation probability adjusted by BW. The whole pipeline of BWCP illustrated in Fig
By modifying the BN layer in Eqn.
we have the formulation of BWCP,
x
X
my (P(Pe > 0))
(2)
batch whitening
soft sampling
where x & E NxNxW denote the output of proposed BWCP algorithm and BW module, respectively. 'O' denotes broadcast multiplication. m. E [o, 1] denotes a soft sampling that takes the activation probability of output features of BW (i.e. P(X- > 0)) and returns a soft mask. The closer the activation probability is to 0 or 1, the more likely the mask is to be hard. To distinguish important channels from unimportant ones, BW is proposed to increase the activation probability of useful channels while keeping the probability of unimportant channels unchanged during training. Since Eqn. (2) always retain all channels in the network, our BWCP can preserve the learning capacity of the original network during training (He et al. 2018a). The following sections present BW and soft sampling module in detail.
4.1 BATCH WHITENING
Unlike previous works (Zhao al. 2019 Kang & Han 2020) that simply measure the importance of channels by parameters in BN layer, attempt to whiten features after BN layer by the proposed BW module. We show that BW can change the activation probability channels according their importances without adding additional parameters and computational overhead in inference.
As shown in Fig BW acts after the BN layer. By rewriting Eqn. into a vector form, we have the formulation of BW.
Enij E ( @ Xnij (3)
(3)
where Enij E RCx1 is a vector of C elements that denote the output of BW for the n-th sample at location (i.j) for all channels. Σー is a whitening operator and Σ E RCxC is the covariance matrix of channel features {x.}_]- Moreover, & E gxxx and / E Cx11 are two vectors by stacking Ye and Be of all the channels respectively. Rnij Cx1 is vector by stacking elements from all channels of Eneij into a column vector.
Training and inference. Note that BW in E) requires computing root inverse of covariance matrix of channel features after the BN layer. Towards this end, we calculate the covariance matrix Σ within batch of samples during each training step as given by
4

---

Under review as a conference paper at ICLR 2022
N.H.W Σ o xni))(7 O xnsy) = op
E
(4)
NHW
where p is a C-by-C correlation matrix of channel features {x.] (see details in Appendix A.1
The Newton lteration is further employed to calculate its root inverse, E 2, as given by the following iterations
m (3 (3
- k- - 1,2, ,
(5)
2
where & and T are the iteration index and iteration number respectively and Eg is a identity
? matrix. Note that when ||T ر|ل < 1, Eqn.d5) converges to Σー (Bini et al. 2005). ). To satisfy this
condition, E can be normalized by 2/tr(E) Tollowing (Huang al. 2019), where tx(-) ((- the trace
operator. In this way, the normalized covariance matrix can be written as ΣN -Y @ p/ || l2.
During inference, we use the moving average to calculate the population estimate of Σn² by following the updating rules, Σー (1 - g) E + yyΣN Here Σn is the covariance calculated within each mini-batch at cach training step, and g denotes the momentum of moving average. Note that Σ N is fixed during inference, the proposed BW does not introduce extra costs in memory or
computation since Σン² can be viewed as a convolution kernel with size of 1, which can be absorbed into previous convolutional layer. For completeness, we also analyze the training overhead of BWCP in Appendix Sec A.3 where we see BWCP introduces little extra training overhead.
4.2 ANALYSIS OF BWCP
In this section, we show that BWCP can easily identify unimportant channels by increasing the difference of activation between important and unimportant channels.
Proposition 2 Let random variable X (0,, 1) and Y. = max (0, ( o X 3)]c)- Then we have P(Y, > 8) P(X, > 5) (1 + Er[((3, - 6)/(v2((.)))/ where ō is small positive constant, No and B. are two equivalent scale parameter and bias defined by BW module. Take T = in (5 as example, have &e (37 -Ed= P (1(l)). and Be = 2 (38, - -d BaldrioPaol (12l))) where Pae is the Pearson's 's correlation between channel features & and Xd.
By Proposition BWCP can adjust activation probability by changing the values of Ta and B. in Proposition through BW module (see detail in Appendix A.4). Here we introduce small positive constant to avoid the small activation feature value. To see how BW changes the activation probability of different channels, consider two cases shown in Proposition 3
Case 3. < and % - O. In this case, the c-th channel of the BN layer would be activated with small activation probability it sufficiently approaches zero. We can see from Proposition B, the activation probability of c-th channel still approaches zero after BW is applied, showing that the proposed BW module can keep the unimportant channels unchanged in this case. Case 2: hel > O. For this case, the e-th channel of the BN layer would be activated with a high activation probability. From Proposition 3 the activation probability of e-th channel is enlarged after BW is applied. Therefore, our proposed BW module can increase the activation probability of important channels. Detailed proof of Proposition can be found in Appendix A.5 We also empirically verify Proposition 3 in Sec. 5.3 Notice that we neglect trivial case in which the channel can be also activated (i.e. 3g > 0 and hrl - 0). In fact, the channels can be removed in this case because the
channel feature is always constant which can be deemed as a bias.
4.3 SOFT SAMPLING MODULE
The soft sampling procedure samples the output of BW through a set of differentiable masks. To be specific, as shown in Fig we leverage the Gumbel-Softmas sampling (Jang et al. 2017) that takes the activation probability generated by BW and produces a soft mask as given by
(6) where 7 is the temperature. By ( and Eqn. 6). BWCP stochastically prunes unimportant channels with activation probability. A smaller activation probability makes m. more likely to be
m. GumbelSSftmax (P(.. > 0); 7)
5

---

Under review as a conference paper at ICLR 2022
close to O. Hence, our proposed BW can help identify less important channels by enlarging the activation probability of important channels, as mentioned in See 4.2 Note that my can converge to 0-1 mask when approaches to O. In the experiment, we find that setting T = 0.5 is enough for BWCP to achieve hard pruning at test time.
Proposition 3 Let 5
Ej= (.pe)) With 4 and B defined Proposition 2 we have (1) P(X. 6) = 0 hel + and Be I O, and (2) P(X. > 8) z P(X, 2 5) hie > O.
1 V EE== (7,8e
Solution to residual issue. Note that the number of channels in the last convolution layer must be the same previous blocks due to the element-wise summation in the recent advanced CNN architectures (He et al. 2016, Huang et al. 2017). We solve this problem by letting BW layer in the last convolution layer and shorteut share the same mask as discussed in Appendix A.6
4.4 TRAINING OF BWCP
This section introduces a sparsity regularization, which makes the model compact, and then describes the training algorithm of BWCP.
Sparse Regularization. With Proposition]] I we see a main characteristic of pruned channels in BN layer is that T. sufficiently approaches and 3e is negative. By Proposition 3] we find that it is also a necessary condition that a channel can be pruned after BW module is applied. Hence, we obtain unnecessary channels by directly imposing regularization on % and Be as given by
Exparse
Xilrrel + 22
where the first term makes %e small, and the second term encourages Be to be negative. The above sparse regularizer is imposed on all BN layers of the network. By changing the strength of regularization (Lee. a and A2), we can achieve different pruning ratios. In fact, Be and hye represent the mean and standard deviation of a Normal distribution, respectively. Following the empirical rule triple double 22 would be good choice encourage sparse
of Normal distribution, setting channels in implementation. Moreover, we observe that 42.2% and 41.3% channels with 3. I 0 while 0.47% and 5.36% channels with leel < 0.05 on trained plain ResNet-34 34 and ResNet-50. Hence, changing the strength of regularization on Yr will affect FLOPs more than that of 3r. If one wants to pursue a more compact model, increasing λ₁ more effective than A2.
Training Algorithm. BWCP can be easily plugged into a CNN by modifying the traditional BN operations. Hence, the training of BWCP can be simply implemented in existing software platforms such as PyTorch and TensorFlow. In other words, the forward propagation of BWCP can be represented by Eqn. 2 and (6n. . all of which define differentiable transformations. Therefore, our proposed BWCP
train and prune deep models in end-to-en manner. Appendix (A.7 also provides the explicit gradient back- pookppropagttion of BWCP On the other hand, we do not introduce extra parameters to learn the pruning mask me. Instead, m. in Eqn. is totally determined by the parameters in BN layers including Y, 3 and Σ. Hence, we can perform joint training of pruning mask m. and model parameters. The BWCP framework is provided in Algorithm of Appendix Sec A.6
Final architecture. The final architecture is fixed at the end of training. During training, we use the Gumbel-Softmax procedure by Eqn.(6 to produce a soft mask. At test time, instead use a hard 0-1 mask achieved by sign function (i.e. sign(P(X, > 0) 0.5)) to obtain the network's 's output. To make the inference stage stable, we use sigmoid-alike transformation to make the activation probability approach 0 or training. By this strategy, we find that both the training and inference stage are stable and obtain fixed compact model. After training, we obtain the final compact model by directly pruning channels with a mask value of Therefore, our proposed BWCP does not need an extra fine-tuning procedure.
5 EXPERIMENTS
In this section, we extensively experiment with the proposed BWCP on CIFAR-10/ and ImageNet. We show the advantages of BWCP in both recognition performance and FLOPs reduction comparing with existing channel pruming methods. We also provide an ablation study to analyze the proposed framework. The details of datasets and training configurations are provided in Appendix B

---

Under review as a conference paper at ICLR 2022
Table 1: Performance comparison between our proposed approach BWCP and other methods on CIFAR-10, 10. "Baseline Acc." and "Acc." " denote the accuracies of the original and pruned models, respectively. "Acc. Drop" means the accuracy of the base model minus that of pruned models (smaller is better). "Channels J", "Model Size 4", and "FLOPs 4"" denote the relative reductions in individual metries compared to the unpruned networks (larger is better). **' indicates the method needs extra fine-tuning to recover performance. The best-performing results are highlighted in bold.
Model
Method
Boseline Acc. (5)
Acc.
Acc. Drop
Channels L (%)
Model (%)
FLOPs (96
DCP
Ziuaag et 1i
2018
95.80
95.49
0.31
49.24
50.25
AMC
92.50
91.90
0.90
$0.000
RosNes-56
SFP
93.59
92:36
1.38
40
$2.60
FPCM
93.59
92.93
0.66
40
52,60
SCP
Kalg
45
46.47
51.20
50.35
Slimming" (Lag
Dene:Ner-40
Vasiational Priniin 24
SCP
King & Tell 20733
2019
94.11
94.55 50
95.16
91.77
0.95
0.62
81
60
75 41
59.76
44.7%
70.77
(WCrTOOrs)
94.21
93.82
0.39
82
76.03
71.72
Slimming"
[Layela 2017
93.55
9291
0.4
70
B7.97
4k.
VGGiNet-16
Variational Puunii
SCP
Kang
93.05
73.34
39.10
BWC
76
93.12
DCP
Zhuasg et 2018
94.47
94.09
4.22
23.5
27.0
MobileNet-V2
MDP Trim if mi 200
95.02
95.14
-0.12
28.7
DWC (Our))
91.56
94.90
-0.36
32.3
37.7
5.1 RESULTS ON CIFAR-10
For CIFAR- 10 dataset, we evaluate our BWCP on ResNet-56, DenseNet-40 and VGG-16 and compare approach with Slimming (Liu et al. 2017), Variational Pruming (Zhao et al. 2019) and SCP (Kang) & Han 2020). These methods prune redundant channels using BN layers like our algorithm. We also compare BWCP with previous strong baselines such as AMC (He al. 2018b) and DCP (Zhuang et al. 2018). The results of slimming are obtained from SCP (Kang & Han 2020). As
mentioned in See 4.2 our BWCP adjusts their activation probability of different channels. Therefore, it would present better recognition accuracy with comparable computation consumption by entirely exploiting important channels. As shown in Table I our BWCP achieves the lowest accuracy drops and comparable FLOPs reduction compared with existing channel pruning methods in all tested base networks. For example, although our model is not fune-tuned, the accuracy drop of the pruned network given by BWCP based on DenseNet-40 and VGG-16 outperforms Slimming with fine-tuning by 1.41% and 0.91% points, respectively. And ResNet-56 pruned by BWCP attains better classification accuracy than previous strong baseline DCP AMC (He et al. 2018b) and DCP (Zhuang al.) 2018) without an extra fine-tuning stage. Besides, our method achieves superior accuracy compared to the Variational Pruming even with significantly smaller model sizes DensNet-40 and VGGNet- 16, demonstrating its effectiveness. We also test BWCP with MobileNet- V2 on the CIFAR10 dataset. From Table D we that BWCP achieves better classification accuracy while reducing more FL.OPs We also report results of BWCP on CIFAR 100 in Appendix B.3
5.2 RESULTS ON IMAGENET
For ImageNet dataset, we test our proposed BWCP on two representative base models ResNet-34 and
ResNet-50. The proposed BWCP is compared with SFP (He et al., ).). FPGM (He et al. 2019). SSS Huang & Wang 2018). SCP (Kang & Han 2020) HRank (Lin et al. 2020) and DSA Ning ctt al. 2020) since they prune channels without an extra fine-tuning stage. As shown in Table 2 we see that BWCP consistently outperforms its counterparts in recognition accuracy under comparable
FLOPs. For ResNet-34, FPGM (He et al. 2019) and SFP (He et al. 2018a) without fine-tuning accelerates 34 by 41.1% speedup ratio with 2.13% and 2.09% accuracy drop respectively, but our BWCP without finetuning achieve almost the same speedup ratio with only 1.16% top-1 accuracy drop. On the other hand, BWCP also significantly outperforms FPGM (He et al., 2019) by 1.07% top-1 accuracy after going through a fine-tuning stage. For ResNet-50, BWCP still achieves better performance compared with other approaches. For instance, at the level of 40% FLOPs reduction. the top-1 accuracy of BWCP exceeds SSS (Huang & Wang 2018) by 3. 7.%.. Moreover, BWCP outperforms DSA (Ning al. 2020) by top-1 accuracy of 0.34% and 0.21% at level of 40% and 50% FLOPs respectively. However, BWCP has slightly lower top-5 accuracy than DSA (Ning et al. 2020).
Inference Acceleration. We analyze the realistic hardware acceleration in terms of GPU and CPU running time during inference. The CPU type is Intel Xeon CPU E5-2682 v4, and the GPU is
7

---

Under review as a conference paper at ICLR 2022
Table 2: Performance of our proposed BWCP and other pruning methods on ImageNet using base models ResNet-34 and ResNet-50. '*' indicates the pruned model is fine-tuned.
Model
Methed
Reseline
(9) Raseline Top-5
Top-1 Drop
Top-5 Drop
FL.PP
FPGM (H. 2019
1.35
BW pomu
73.72
91.64
0.31
0.34
41.0
RosNos-34
SFP (R et al. 2018a
73.02
91.62
2.09
1.29
41.1
EPU [16 ct
73,92
91.52
2.13
0.92
41.1
BWCF (CBIS)
73.72
91.64
1,16
0.83
41.0
FPGM (Ic at. 2011
2019
76.15
92.87
L.32
0.55
53.5
BWC y (U)in)
76.20
93.15
0.46
0.40
51.2
SSS Ruisig & Wane
201
'25.12"
3,30
207
150
DSA
0.92
0.41
40.0
H[Rank
76.15
92.87
L.17
0.61
ResNee-50 50
ThiNet*
76.20
93.15
0.58
(Hfe st
15
92.87
2.02
0.93
SCP
Kabar & 131n 2010
75.89
92.98
1.69
0.98
54.3
DSA [vig or 20227
1.33
0.80
50.0
BWCF(OWS)
76.20
93.15
1.022
0.60
51.2
Table 3: Effect of BW, Gumbel- (GS), and sparse Regularization BWCP. The results are obtained by training ResNet-56 on CIFAR-10 dataset. 'BL'' denotes baseline model.
Table 4: Effect of regularization strength A and
22 with magnitude le 4 for the sparsity loss in
Eqn. (7). The results are obtained using VGG-16
on
CIT-R-100 dataset.
Cases
Bw
GS Reg
Acc. (%)
Model Size
PLOP
A
Aa
Acc. (%)
Acc. Drop
FLOPs | (%)
BL
x
33.61l
94,12
1.2
0.6
13.85
-U.34
33.33
(2)
X
93,46
1.2
1.2
73.66
-0.15
35.92
(3)
x
/
V
92.81
46.37
51.16
1.2
2.4
73.33
0.18
54.19
/
94.10
7.78
6.25
0.6
1.2
74.27
-0.76
30.67
2.4
1.2
71.73
1.78
60.75
BWCP
/
93.37
44.42
50.35
NVIDIA GTX1080Ti. We evaluate the inference time using ResNet-50 with mini-batch of 32 (1) on GPU (CPU). GPU inference batch size is larger than CPU to emphasize our method's acceleration on the highly parallel platform
structured pruning method. We see that BWCP has 29.2% inference time reduction on GPU, from 48. 7ms for base ResNet-50 to 34.5ms for pruned ResNet-50, and 21. 21.2% inference time reduction on CPU, from 127. 1ms for base ResNet-50 to 100.2ms for pruned ResNet-50.
5.3 ABLATION STUDY
Effeet of BWCP on activation probability. From the analysis Sec. 4.2 we have shown that BWCP can increase the activation probability of useful channels while keeping the activation probability of unimportant channels unchanged through BW technique. Here we demonstrate this using Resnet-34 34 and Resnet-50 trained on ImageNet dataset. We calculate the activation probability of channels of BN and BW layer. It can be seen from Fig (a that (1) BW increases the activation probability of important channels when he| > O; (2) BW keeps the the activation probability of unimportant channels unchanged when 3, < 0 and % + O. Therefore, BW indeed works by making useful channels more important and unnecessary channels less important, respectively. In this way, BWCP can identify unimportant channels reliably.
Effect of BW, Gumbel-Softmax (GS) and sparse Regularization (Reg). The proposed BWCP consists of three components including BW module (i.e. Eqn. (3)) and Soft Sampling module with Gumbel-Softmal (i.e. Eqn. (6)) and a spare regularization (i.e. Eqn. (7). Here we investigate the effect of each component. To this end, five variants of BWCP are considered: (1) only BW module used; (2) only sparse regularization is imposed; (3)BWCP w/o BW module; (4) BWCP w/o sparse regularization; and (5) BWCP with Gumbel-Softmax replaced by Straight Through Estimator (STE) (Bengio et al. 2013). For case (5), we select channels by hard 0-1 mask generated with m. sign( ((x, > 0) - 0.5) The gradient is back-propagated through STE. From results on Table 3 we can make the following conclusions: (a) BW improves the recognition performance, implying that it can enhance the representation of channels; (b) sparse regularization on fy and B slightly harm the classification accuracy of original model but it encourages channels to be sparse as also shown in Proposition 3 (c) BWCP with Gumbel-Softmax achieves higher accuracy than STE, showing that soft sampling technique is better than the deterministic ones reported in (Jang et al. 2017).
y sign(r) 1 if x and of O.
8

---

Under review as a conference paper at ICLR 2022
34-ect Prok
B4-- Prec
orginido KAMPP AM
0rlp maAa waacc IW
Cervenation Scaw
0.4
0.4
911
-2
02-
6.2
0.3
20
40
60
20
20
40 60
20 60 60
40000 50000 110080
49000 20000 120000
Figure 3: ((a) & (b)) and ((c) & (d)) show the effect of BWCP on activation probability with trained ResNet-34 and ResNet-50 on ImageNet, respectively. The proposed batch whitening (BW) can increase the actívation probability of useful channels when 7ell > 0 while keeping the unimportant channels unchanged when when 3e S and The + O. (e) & (f) show the correlation score for the output response maps in shallow and deeper BWCP modules during the whole training period. BWCP has lower correlation score among feature channels than original BN baseline.
Impact of regularization strength X1 and A2-- We analyze the effect of regularization strength A and b for sparsity loss on CIFAR- 100. The trade-off between accuracy and FLOPs reduction is investigated using VGG-16. Table 4 illustrates that the network becomes more compact as a and X₂ increase, implying that both terms in Eqn.(7 can make channel features sparse. Moreover, the flops metric more sensitive to the regularization on 7, which validates our analysis in Sec[4.2j. Besides, we should search for proper values for a and X2 to trade off between accuracy and FLOPs reduction, which is a drawback for our method.
Effect of the number of BW. Here the effect of the number of BW modules of BWCP is in- vestigated trained on CIFAR-10 using Resnet-56 consisting of a series of bottleneck structures. Note that there are three BN layers in each bottleneck. We study four variants of BWCP: (a) we use BW to modify the last BN in each bottleneck module; hence there are total of 18
Table 5: Effect of the number of BW modules on CIFAR-I0 dataset trained with ResNet-56. '# BW' indicates the number of BW. More BW modules in the network would lead to lower recogmition accuracy drop with comparable computation con- sumption.
BW layers in Resnet-56; (b) the last two BN layers are modified by our BW technique (36
BW layers) (c) All BN layers in bottlenecks are replaced by BW (54 BW layers), which is our proposed method. The results are reported in Table 5 We can see that BWCP achieves the best top-1 accuracy when BW acts on all BN layers, given the comparable FLOPSs and model size. This indicates that the proposed BWCP more benefits from more BW layers in the network.
#BW
Acc. (%)
Acc. Drop
Model Size J (%)
FLOPs 1 (%)
93.01
0.63
44.70
50.77
36
93.14
0.50
45,29
50.45
54
93.37
0.27
44.42
50.35
BWCP selects representative channel features. It is worth noting that BWCP can whiten channel features after BN through BW as shwon in Eqn. Therfore, BW can learn diverse channel features by reducing the correlations among channels Yang et al. 2019b). We investigate this using VGGNet- 16 with BN and the proposed BWCP trained on CIFAR I0. The correlation score be calculated by taking the average over the absolute value of the correlation matrix of channel features. A larger value indicates that there is redundancy in the encoded features. We plot the correlation score among channels at different depths of the network. As shown in 3 & f), channel features after BW block have significantly smaller correlations, implying that channels selected by BWCP are representative. This also accounts for the effectiveness of the proposed scheme.
6 DISCUSSION AND CONCLUSION
This paper presented an effective and efficient pruning technique, termed Batch Whitening Channel Pruning (BWCP). We show BWCP increases the activation probability of useful channels while keeping unimportant channels unchanged, making it appealing to pursue a compact model. Partic- ularly, BWCP can be easily applied to prune various CNN architectures by modifying the batch normalization layer. However, to achieve different levels of FLOPs reduction, the proposed BWCP needs to search for the strength of sparse regularization. With probabilistic formulation in BWCP, the expected FLOPs can be modeled. The multiplier method can be used to encourage the model to attain target FLOPs. For future work, advanced Pareto optimization algorithm can be designed to tackle such multi mujti--bbecctve joint minimization. We hope that the analyses of BWCP could bring a new perspective for future work in channel pruning.
9

---

Under review as a conference paper at ICLR 2022
Ethics Statement. We aim compressing neural nets by the proposed BWCP framework. It could improve the energy efficiency of neural network models and reduce the emission of carbon dioxide. We notice that deep neural networks trained with BWCP can be plugged into portable edge devices such as mobile phones. Hence, our work and Al in edge devices would have the same negative impact on ethics. Moreover, network pruning may have different effects on different classes, thus producing unfair models as a result. We will carefully investigate the results of our method on the fairness of the model output in the future.
Reproducibility Statement. For theoretical results, clear explanations of assumptions and complete proof of propostion 1-3 are included Appendix. To reproduce the experimental results, we provide training details and hyyee-parameter in Appendix Sec[B] Moreover, we will also make our code available by link to an anonymous repository during the discussion stage.
REFERENCES
Devansh Arpit, Yingbo Zhou, Bhargava U Kota, and Venu Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift in deep networks. International Conference in Machine Learning, 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL
http::/arxx: org/oaggo....... wwwpwbs.//////ooooo.ott...p..w.e 1308 3432
Dario Bini, Nicholas J Higham, and Beatrice Meini. Algorithms for the matrix pth root. Numerical Algorithms, 39(4):349-33) 0774999-7,200 2005.
Liang-Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L., Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-848, 2018.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXi::1803.03633 2018.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv: 2018.
Jinyang Guo, Wanli Ouyang, and Dong Xu. Multi-dimensional pruning: A unified framework for model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1508-1517, 2020.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dans. In Advances in neural information processing systems, pp. 1379-1387, 2016.
Song Han and William J. Dally. Bandwidth-effiiiee deep learning. In Proceedings of the 55th Annual Design Automation Conference on, pp. 147, 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR 2016 International Conference on Learning Representations 2016, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2022
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint 1808. 06866, 2018a
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4349 2019.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784-800, 2018b.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4700 47000-470.1100 2017.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardization towards efficient whitening. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4874-4883, 2019.
Zehao Huang and Naiyan Wang. Data-Driven Sparse Structure Selection for Deep Neural Networks. In ECCV, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. 2017.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. arXiv preprint arXiv:2007.03938 2020.
Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, John S Denker, and Sara A Solla. Optimal Brain Damage. In NIPS, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning Filters for Efficient ConvNets. In ICLR, 2017.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529-1538, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Lear- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference Computer Vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 1rXiv:1810.05271 2018.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. International Conference on Learning Representation, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. arXiv preprint arXiv:2004.02214 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
11

---

Under review as a conference paper at ICLR 2022
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-11
2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint wrXiv:1409.1556. 2014.
Huan Wang, Qiming Zhang, Yuehai Wang. and Haoji Hu. Structured probabilistic pruning for convolutional neural network acceleration. arXiv preprint arXiv:1709.0000 06994, 2017.
Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong. and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in Neural Information Processing Systems, 33, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 2074 2082, 2016.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. arXiv preprint arXiv: 1rXiv:1908..9900 2019a.
Jianwei Yang, Zhile Ren, Chuang Gan, Hongyuan Zhu, and Devi Parikh. Cross-channel com- munication networks. In Advances in Neural Information Processing Systems, pp. 1295-1304, 2019b.
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. International Conference on Machine Learning, pp.
PMLR, 2020.
Chenglong Zhao, Binghing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2780-2789, 2019.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-ave aware channel pruning for deep neural networks. arXiv preprint arXiv: 1809, 2018
12

---

Under review as a conference paper at ICLR 2022
The appendix provides more details about approach and experiments of our proposed batch whitening channel pruning (BWCP) framework. The broader impact of this work is also discussed.
A MORE DETAILS ABOUT APPROACH
A.1 CALCULATION OF COVARIANCE MATRIX E
By Eqn. T in main text, the output of BN is Enoij = YoEneij + 9e. Hence, we have E|x.] NHW En.ie (YeFnciiy 3e) Pe- Then the entry in e-th row and d-th column of covariance matrix E X. is calculated follows:
N.H.W
Edd NHW
Σ
(TeTnnii + 8c (T + Ba 1E(x,)) YaPaPed
(8)
where Pod is the element in c-th row and j-th column of correlation matrix of σ. Hence, we have Ped [-1, 1]. Furthermore, can write E into the vector form: Σ -y² NHW Enis KniyXnij
H.W.
yy C
A.2 PROOF OF PROPOSITION
T
For (1), notice that can define %o -Ne and X. - ~ (O(0 if Yo O. Hence,
assume Ye > 0 without loss of generality. Then, we have
3
P(Y. 0) P(X. 0) P(X.
7e
t00
cxp
dt
Be
2m
70
exp
4 dl
+
1
exp
' dl
(9)
de
22T
0
22T
12
+
exp
dt
+
exp
V2T
J0
2t
Erf( 82 V270
1+1
2
When 2e < 0, we can set To = -Yc. Hence, we arrive at
Eri( B. 12h
+
P(Y > 0) = P(X. > 0)
(10)
2
For (2), let us denote X. ~ No,, 1), and X. = 2e + B. and Y. = max X.) where Y represents random variables corresponding to output feature yc in (qn.( in main text. Firstly, it is easy to see that P(X. > 0) 0 + Ex [Y.] = 0 and _x [V=3] 0. In the following we show that
Ex, [Y]] 0 and Ex. [Y-] 0 + B. s 0and = O. Similar with (1), assume % > 0 without loss of generality.
For the sufficiency, have
too
Ex, [2]
0.
exp
dē. +
(yo + Bc)
exp
dā
2t
22x
(11)
22
exx
2
Be
Be V2%
_c
+
(1
Erf[
2
2
where Er[[x] 3 for exp
2 di is the error function. From Eqn. 11 have
lim Eg [Y]
lim
Teep
2
lim Th
3a (1
+
Erf
3,
1)
= 0
(12)
- 7o +0+
22m
+
2
V
13

---

Under review as a conference paper at ICLR 2022
Table 6: Runing time comparison during training between BWCP, vanilla BN and SCP. The proposed BWCP achieves better trade-off between FLOPs reduction and accuracy drop although it introduces a little extra computational cost during training. 'F' denotes forward running time (s) while 'F+B' denotes forward and backward running time
(s). The results are averaged over 100 iterations. The GPU is NVIDIA GTXI080Ti, The CPU type is Intel Xeon E5-2682 v4.
Model
Mothod
CPU (s) (s)
CPU (F4B) (s)
GPU (F) (s)
GPU (F4B) (8)
Acc. Drop
FLOPs | (%)
vamilaa BN
0,478
0.015
0.031
ResNet-50
SCP
0.193
0.495
0.034
0.067
1.69
54.3
BWCP (Ours)
0.239
0.610
0.053
0.104
1.02
51.2
In the same way, we can calculate
5²
+ f (1att Bc)2
+00
1
Eg, [Y:]
exp
$ dī.
exp
- dē
2m
22
(13)
exp
2
2 + c
Be
(1 + Erf
1),
V2T
2
V2%
From Equ. we have
13
#2
lim Ez. [Y?]- 7o 7++
lim +0+ 22T
cxp
lim
2 Ye
+
(1 [1+Erf[[
Be
= 0 (14)
No 7-+0+
2
V 2
For necessity, we show that if EE,, [Y]] = and E,, (V= = O, then % - 0 and 3. S 0. In essence,
can be acquired by solving Eqn Eqn.qTI and Eqn. (3). To be specific, 3. (II Eqq. (I)
gives % = 0+. Substituting it into Eqn. I1). we can obtain Be S 0. This completes the proof.
A.3 TRAINING OVERIEAD OF BWCP
The proposed BWCP introduces a little extra computational cost during training. To see this, evaluate the computational complexity of SCP and BWCP for ResNet50 on ImageNet with input image size of 224 x 224. We can see from the table below that the training BWCP is slightly slower on both CPU and GPU than the plain ResNet and SCP. In fact, the computational burden mainly comes from calculating the covariance matrix and its root inverse. root inverse of the covariance matrix by Newton's 's iteration, which is fast and efficient. Although BWCP brings extra training overhead, it achieves better top-1 accuracy drop under the same FLOPs consumption.
our paper, we calculate the
A.4 PROOF OF PROPOSITION 2
First, we can derive that Xx =
3= ( x -
Hence, the newly defined scale and bias parameters are Σ and 3 Σn 3. When T =
we have Σ 13I - Σx) by 5 in main text. Hence we obtain,
1 331 TT--ENy-- 2
TxT
= 2
(3I
Oph Iyl
T
1
(37
1r>
1
Σ
773P157H
Σ
1075P0378
(15)
C
T5P
C Σ
P7
33
1
(3
)nc
2
|
| 2
1=1
1=1
14

---

Under review as a conference paper at ICLR 2022
Similarly, B can be given by
yy
8 231
E)ح
2 (31
p)B
|
T
c
= 2
(3g
11
1170115511
(16)
r
T3BpP1s
C
Ty3,pcs
3
171
33c
nCc
|
17
Taking each component of vector Eqn.
15
16
gives us the expression of te and &e in Proposition 2
A.5 PROOF OF PROPOSITION 3
Σの P. hhlll Therefore, lfel + 0 if [7o + For (I), through Eqn.0 we acquire (iel = 3 - Ty 1
On the other hand, by Eqn. 16). we have B. (3 ا )8, < S 0. Here we assume that Pod = 1 if d and 0 otherwise. Note that the assumption is plausible by Fig (c & f) in main text from which we see that the correlation among channel features will gradually decrease during training. We also empirically verify these two conclusions by Fig From Fig 4 we can that
Hel had where the equality holds iff 17e = 0, and B, is larger than B. if 33 is positive, and vice versa. By Proposition
I we arrive at
P 6) P(X. 0) =0
(17)
where the first '>' holds since 8 is small positive constant and '=' follows from 1%cl -> 0 and Be S 0. For (2), to show P(X, > 8) > P(Xe > 6), we only need to prove P(X, >
8-3 >
Tha
P(X. --3=), ), which is equivalent to
8- م= To this end, we calculate 17cl Txol
hel88
19cl3
hnel - (2
I.B: )70) - 1 Σ5= 1ri1
Bope Thill
170
Tal
l7cl
{3 E j=1 1r/
3300
177el - lrol
JrBrrePai mr
Σ/=1
2 3.pe
j=
1 プラ= j=1
C
1
(18)
Eg 75(1877 -3 BoBoees
ES (B,7o - 77.
1//13
My
1- -E-
J5prs
==
j=
11rl1
\ Eg (3,76- - -7;33)2
=
1//1 - Σg= 33
where the 'S" holds due the Cauchy-Schwarz inequality. By Eqn. 18). we derive that Irel(6 3.) < (8 (e) which is exactly what we want. Lastly, we empirically verify that 8 defined in Proposition is small positive constant. In fact, 8 represents the minimal activation feature value X. x. + B. 2/4 by definition). We visualize the value of ò in shallow and deep layers in ResNet-34 during the whole training stage and value of å of each layer in trained ResNet-34 on ImageNet dataset in Fig 5 As we can see, 8 is always small positive number in the training process. We thus empirically set 0 as 0.05 in all experiments.
To conclude, by Eqm. (7), BWCP
keep the activation probability of unimportant channel un- changed; by Eqn. BWCP can increase the activation probability of important channel. In this way, the proposed BWCP can pursuit a compact deep model with good performance.
15

---

Under review as a conference paper at ICLR 2022
Ialloged Aoee
0.
-0.2
-04
-15
IkF
Bky
202 222
182
560
94
16-
1.4
64
62
3.5
0.0
Figure 4: Experimental observation of how our proposed BWCP changes the values of Ye and B. in BN layers through the proposed BW technique. Results are obtained by tranining ResNet50 on ImageNet dataset. We investigate Ye and Be at different depths of the network including layer1.0.. .0.bnl, layer2.0.br 1r.e2...bn.,laaaeee .layer300.bn and layer4.0. 0.bn1. (a-d) show BW enlarges B, when 3 > O while reducing Be when B, 0. (e-h) show that BW consistently increases the magnitude of 7a across the network.
(a) RestNatt344layyrr...mmm
(b) RewNet- Nooleet-44llayer4...0
(c) ResNett 34- 34- layer
0.622
0.0050
0.0045
0.0040
0.01
0.0035
0.914
Www
0.010
0.008
0.00
60
Figure 5: Experimental observation of the values of defined in proposition Results obtained by tranining ResNet-34 on ImageNet dataset. (a & b) investigate at different depths of the network including layer1.0 1.0.bnl and layer4.0.u. 0nll respectively. (c) visualizes ô for each layer of ResNet-34. We see that 8 in proposition is always small positive constant.
Shortcut with Conv-BN
m me
Conv BN
Conv BN
Conv BN
BW
ReLU
BW
Sum
ReLU
mc
Conv BN
Conv BIN
ReLU
ReLU
m
Identity
Shortcut with Identity
Figure 6: Illustration of BWCP with shortcut in basic block structure ResNet. For shortcut with Conv-BN modules, simple strategy that lets BW layer in the last convolution layer and shortcut share the same mask. For shortcut with identity mappings, we use the mask previous layer.
16

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Forward Propagation of the proposed BWCP.
1: Input: mini-batch inputs E I xxxxxxx
2: Hyperparameters: momentum for calculating root inverse of covariance matrix, iteration number T.
4: calculate standardized activation: {xx C BWCP.
3: Output: the activations x obtained
Equ
5: calculate the output of BN layer: X = T/ce + 8c.
6: calculate normalized covariance matrix: EN = 1
2miij=1 KnijXnij
NHW
7: En I.
8: for k = to do
9: Σx (32x-1 E IN)
10: end for
11: calculate whitening matrix for training: 2
N Er.
calculate whitening matrix for inference: Σ t- - 1) + g
13: calculate whitened output: &nij ΣN Knij.
14: calculate equivalent scale and bias defined by BW: 9 = E by and B Σ 2ー
15: calculate the activation probability by Proposition
2 with & and 3, obtain soft masks {me]- by Eqn.
16: calculate the output of BWCP: x Sx mlc-
(A) BN
(b) BWCP
X
Figure 7: Illustration of forward propagation of (a) BN and (b) BWCP. The proposed BWCP prunes CNNs by replacing original BN layer with BWCP module.
A.6 SOLUTION TO RESIDUAL ISSUE
The recent advanced CNN architectures usually have residual blocks with shortcut connections (He et al. 2016 Huang et al., 2017). As shown in Fig the number of channels in the last convolution layer must the same in previous blocks due the element-wiss summation. Basically, there are two types of residual connections, i.e. shortcut with downsampling layer consisting of Conv- BN modules, and shortcut with identity. For shorteut with Conv-BN modules, the proposed BW technique is utilized in downsampling layer to generate pruning mask m Furthermore, we use simple strategy that lets BW layer in the last convolution layer and shortcut share the same mask as given by m, m mast where masst and m denote masks of the last convolution layer. For shortcut with identity mappings, we use the mask in the previous layer. In doing so, their activated output channels must be the same.
A.7 BACK-PROPAGATION OF BWCP
The forward propagation of BWCP can be represented by Eqn.(3-4) and Eqn. in the main text (see detail Table D. all of which define differentiable transformations. Here we provide the back-propagation of BWCP. By comparing the forward representation of BN and BWCP in Fig 7 for backward propagation of BWCP. For we need to back-propagate the gradient DL
BC.
Uxouu
to
simplicity, we neglect the subscript 'rij"".
By chain rules, we have
3C
JL ox
= 7
m
(19)
0x
0x
17

---

Under review as a conference paper at ICLR 2022
Table 7: Performance of our BWCP different base models compared with other approaches CIFAR-100 1-0 dataset.
Model
Motood
Biseline Acc. (18)
Acc. (56)
Acc. Drop
Channels
(16)
Model Size (16)
HLOPs (%6
Slimming
Liu al. 2017
77.24
74.52
2.72
60
29.26
47.92
Re4Nes-164
SCP
Kang Him pranp
77.24
76.62
0.62
57
28.89
36
77.24
76.77
0.47
41
21.58
39.84
73.33
DenseNet-40
Variatonal
SCP
Kang & Tran 4000
74.24
73.84
12.19
0.80
55.22
46.25
BWCF (Umis
74.24
74.18
0.06
$3.53
40.40
VGGINet-19
Slimming
BWC (UBis)
[Lu al.
2019
72.56
72.36
73.20
73.01
-0.4
-045
50
23
41,00
76.47
22 09
38,23
Slimming
2013
7531
0.16
66.30
Varistional
73.31
73.60
0c ENN' denotes the gradient w.r.l. x back-propagated through ΣN² To calculate it, we first where 28
obtain the gradient w.rl ΣN" as given by
0c
OLT
UCT
+
(20)
aΣ N
05
0
where
JC
UC
dm
0L ox
05y = x ④
m
ox
04
(x +
(21)
and
ac = m
7m
ac
(22)
03
0
Oxout
The remaining thing is to calculate arrive at
dm and dm 85
Based on the Gunbel-Softmax transformation, we
dm,
- (( (P( hol 0, otherwise
Be% if
(23)
0id
Om
(T ifd =c
me(l-m.)
(24)
03d
0, otherwise
where f(fe. 3c) is the probability density function of R.V. X. as written in Eqn.(2) of main text.
To proceed, we deliver the gradient w.r.t. EN in Eqn. 20 to by Newton Iteration in (6n. of main text. Note that Σ Er, we have
ac
3 T) 0L
(25)
NΣN
where ac GE
can be calculated by following iterations:
OC
dL
OC 0E
T) - -x%
-Σ
2 20
2 2
(Ex-1E)T TT.
TΣ
Given the gradient w.rlt Σn in Eqn. (25).
calculate the gradient w.r.t. back-propagated
through " in 19 follows
ac S
YT
OL
OL
0x
=
1r
NΣ
+
NEN
T))x
(26)
Based Eqn. 19 26 obtain the back-propagation of BWCP.
18

---

Under review as a conference paper at ICLR 2022
B MORE DETAILS ABOUT EXPERIMENT
B.1 DATASET AND METRICS
We evaluate the performance of our proposed BWCP various image classification benchmarks, including CIFAR 10/100 (Krizhevsky 2009) and ImageNet (Russakovsky et (015). The CIFAR-10 and CIFAR- 100 datasets have 10 and 100 categories, respectively, while both contain 60k color images with size of 32 x 32, in which 50k training images and 10K test images are included. Moreover, the ImageNet dataset consists of 1.28M training images and 50k validation images. Top-1 accuracy CIFAR 10/100. Top-1 and Top-5
used to evaluate the recognition performance of models accuracies are reported on ImageNet. We utilize the common protocols, i.e. number of parameters and Float Points Operations (FLOPs) to obtain model size and computational consumption.
For CIFAR-IO/ 10/100, we use ResNet (He et al. 2016), DenseNet (Huang et al. 2017). and VGGNet (Sr monyan & Zisserman 2014) as base model. For ImageNet, we use ResNet-34 and ResNet-50. We compare our algorithm with other channel pruning methods without a fine-tuning procedure. Note that extra fine-tuning process would lead remarkable improvement of performace (Ye et al.) 2020). For fair comparison, we also fine-tune our BWCP to compare with those pruning methods. The training configurations are provided Appendix B.2 The base networks and BWCP are trained together from scratch for all of our models.
B.2 TRAINING CONFIGURATION
Training Setting on ImageNet. All networks are trained using GPUs with a mini-batch of 32 per GPU. We train all the architectures from scratch for 120 epochs using stochastic gradient descent (SGD) with momentum 0.9 and weight decay le-4. We perform normal training without sparse regularization in Eqn. on the original networks for first 20 epochs by following (Ning et al. 2020). The base learning rate is set to 0.1 and is multiplied by 0.1 after 50, 80 and 110 epochs. During fine-tuning, we use the standard SGD optimizer with Nesterov momentum 0.9 and weight decay 0.00005 to fine-tune pruned network for 150 epochs. We decay learning rate using cosine schedule with initial learning rate 0.01. The coefficient of sparse regularization X1 and A2 are set to 7e-5 and 3.5e-5 to achieve Flops Reduction at a level of 40%, while x1 and X2 are set to 9e-5 and 3.5e-5 respectively to achieve FLOPs reduction at a level of 40%. Besides, proposed BW technique is calculated within each GPU. Like (Huang et al. 2019), we also use group-wise decorrelation with group size 16 across the network to improve the efficiency of BW.
covariance matrix in the
Training setting CIFAR-10 and CIFAR-100. We train all models on CIFAR- 10 and CIFAR-100 with batch size of 64 on a single GPU for 160 epochs with momentum 0.9 and weight decay le-4. The initial learning rate is 0.1 and is decreased by 10 times at 80 and 120 epoch. The coefficient of sparse regularization ê and A2 are set to 4e-5 and 8e-5 for CIFAR-10 dataset and 7 and 1.4e-5 for CIFAR-100 dataset.
B.3
MORE RESULTS OF BWCP
The results of BWCP on CIFAR- 100 dataset is reported in Table 7 As we can see, our approach BWCP achieves the lowest accuracy drops and comparable FLOPs reduction compared with existing channel pruning methods in all tested base models.
19


Reviews:
[{"review_id": "ukUyGzLD36", "reviewer": "Reviewer_qyNE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Based on the proposed stochastic pruning scheme, the full channel space is preserved during the training process, which avoid stuck in some local minimum due to improper selection of channels to be pruned.\n2. The batch whitening module, together with the regularization term on per-channel scaling factors and biases, automatically increases the activation probability of informative channels, so that unimportant channels can be identified more easily.\n3. The effectiveness of the proposed method is validated by extensive experiments on various network architectures and ablation studies.\n\nWeaknesses:\n1. The training efficiency of the proposed method is not discussed. Will the stochastic pruning scheme require additional training epochs, compared with other channel pruning methods?\n2. Section 4.4, Equation (7). As shown in Proposition 1, the activation probability is determined by $\\beta_{c} / \\left| \\gamma_{c} \\right|$, so it seems more reasonable to define the second regularization term as $\\lambda_{2} \\beta_{c} / \\left| \\gamma_{c} \\right|$, rather than $\\lambda_{2} \\beta_{c}$ (or, is it due to the optimization difficulty of the former one?).\n3. Section 4.4, last paragraph. Authors state that a sigmoid-like transformation is used to make activation probabilities approach 0 or 1 during training. How is this carried out exactly? It would be helpful to provide more technical details to improve the reproducibility of the proposed method.", "summary_of_the_paper": "In this paper, authors propose to compress convolutional layers via batch whitening channel pruning (BWCP). By apply batch whitening on incoming feature maps, unimportant channels are automatically identified and stochastically discarded during the training process. A sigmoid-like transformation is used to push activation probabilities to either 0 or 1, so that additional fine-tuning is not needed, unlike many other channel pruning methods.", "summary_of_the_review": "The training efficiency remains unclear. Although the stochastic pruning scheme can explore the channel space more sufficiently, it is also possible that more training time are needed for such exploration. More details should be provided here.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lyXekC0Qaf", "reviewer": "Reviewer_PUWD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1.\tThey propose a probabilistic pruning method, quite different from existing pruning methods which act in a deterministic manner. \n2.\tTheir method is shown effective on CIFAR-10/100 and ImageNet.\n\nWeaknesses:\n\n1.\tSevere idea overlap. \nThe main spirit of this paper, “unimportant channels are stochastically pruned with activation probability”, namely the so-called “probabilistic” pruning, has been explored in the literature. See [*1], they proposed to assign activation probabilities to different filters; more important filters have larger activation probability.  This is very similar to what is proposed in this paper: “we assign each channel with an activation probability (i.e. the probability of a channel being activated)”, “A larger activation probability indicates that the corresponding channel is more likely to be preserved”. Please clarify how your method is substantially different from [*1] (if any).\nBesides, there is no determined relationship between regularization strength and the final speedup in this method. Thus, “we should search for proper values for λ1 and λ2 to trade off between accuracy and FLOPs reduction, which is a drawback for our method”. However, in [*1], how many filters are imposed with probabilistic pruning is explicitly linked with the desired layer-wise sparsity, thus their method can achieve desired speedup for sure. In this sense, [*1] is also more convenient than the BWCP in practice.\n\n2.\tPerformance is not strong. Missing important comparison with more recent methods. \nIn Tab. 2, ResNet50, ImageNet. They do not have any 2021 papers compared. If compared to the more recent 2021 methods (e.g., [*3-*4]), BWCP is actually not as competitive. For example, [*3] achieves 75.36% top-1 accuracy under speedup 2.31x, while BWCP only achieves 75.18% under even less speedup (2.05x).\n\n3.\tIn terms of the proposed methodology itself, it is a BN-based pruning method. While not all deep CNNs have BN. For example, in image super-resolution (SR), BN is not used in SOTA SR network (such as [*2]). This limits the potential application of this method, while related works (such as [*1], HRank, DSA, [*3-*4], etc) do not have this limitation.\n\n[*1] Structured Probabilistic Pruning for Convolutional Neural Network Acceleration, BMVC, 2018\n\n[*2] Image super-resolution using very deep residual channel attention networks, ECCV, 2018\n\n[*3] Neural pruning via growing regularization, ICLR, 2021.\n\n[*4] Towards compact cnns via collaborative compression, CVPR, 2021\n", "summary_of_the_paper": "This paper presents a probabilistic channel pruning method (BWCP) for accelerating CNNs. The key newly proposed technique is “batch whitening”. They evaluate their method on CIFAR-10/100 and ImageNet compared to other recent filter pruning methods.", "summary_of_the_review": "There is a severe idea overlap between this work and previous work. Meanwhile, the empirical performance is not strong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VBT4AKs4oS2", "reviewer": "Reviewer_8cV6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Feature whitening reduces the correlation between features. It was shown that the whitened representation facilitates the training. Another property is that the whitening also leads to a sparse representation. This paper shows a nice application of this property in training compact neural networks. \n\nThis direction is very important, neural networks have been dense and large. Whitening has been found to be the more optimal normalization  in terms of training of neural networks. However, its benefit in compressing is barely known to the community. This paper is a good attempt in this direction.\n\nThe downsides: \n\n(1) Some reorganization could clarify the presentation. The reader may want to have a quick understanding of why whitening is chosen over standardization. Some plots similar to Fig. 10,11 in Ye et. al network deconvolution may be helpful. It will be beneficial to make the paper more self-contained, e.g. clearly formulate Gumble softmax  in eq 6.\n\n\n(2) It is not clear whether the improvement is due to whitening or pruning. Whitening based methods such as decorrelated Batch Norm and IterNorm by Huang et. al generally lead to better performance compared to standard normalization methods. The benefits of sparsity may have already been exploited by these methods. If this is the case, one might be able to directly train a narrower network without the Gumbel softmax trick. Can the authors provide more insights/experiments to this? How much better is this method compared to direct whitened training?\n\na typo in page 8: spare -->sparse", "summary_of_the_paper": "This paper proposes a way to train compact neural networks. The feature channels are whitened, then combined with a sampling method. At the testing time, this sampling is discarded and replaced with a deterministic selection. \n\nThe benefits of whitening in improving the convergence of NNs is widely explored. This paper explores another useful property of whitening,  the sparsity in the whitened results. Since the representation is sparser,  one might use fewer weights to reach similar performance. This paper presents such a solution.", "summary_of_the_review": "The paper has room for improvements, but it is a nice new attempt to the pruning of networks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cn-W9JBlxDm", "reviewer": "Reviewer_kp3m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of this paper can be summarized into the following points.\n1. This method has some good properties for pruning, like structural pruning, simultaneously pruning and training, and so on.\n2. The analysis of whitening shows that the activation probability of important channels will become larger, which might be helpful for channel pruning.\n3. The method generally achieves better results compared to similar methods.\n\nThe weakness of this paper are:\n1. Batch-whitening is built on batch normalization, which may limit the application scope of this paper. In language modelings and vision transformers, layer normalization is widely used instead of batch normalization. Does whitening can also be applied to such settings? In addition, for downstream tasks, like object detection, the image resolution becomes larger, and the allowed mini-batch size will decrease dramatically. When the mini-batch size is limited, dose batch-whitening suffers from the poor estimation of $\\Sigma$? In another word, the proposed method may perform poorly when batch normalization is unreliable. \n2. Some arguments are not well supported by the experimental results. In the paper, the authors argue that 'bach-whitening increase the activation probability of useful channels while keeping the activation probability of unimportant channels unchanged'. This is demonstrated in Fig.3 (a.b). However, increasing the activation probability may not improve pruning. In Table. 3, Fig. the relative performance changes between case (3) and BL is similar to case (1) and BWCP. If increasing the activation probability is helpful, we should see BWCP achieves a more obvious performance gain compared to the case (3). The current ablation study does not support the argument that increasing the activation probability benefits pruning. \n3. It seems that the batch-whitening process incurs additional computational costs since you need to keep updating the covariance matrix. Although newton iteration improves the efficiency, the training time is also increased. If we measure the training budget by the training time, other methods actually use a smaller training budget.\n4. The experimental results are not very significant compared to previous methods. The improvement over previous baselines is around 0.3 for ResNet-50. ", "summary_of_the_paper": "The paper presents a probabilistic channel pruning method based on batch whitening to accelerate and compress CNNs.", "summary_of_the_review": "In summary, I think this is a borderline paper. It has some good properties, but it is limited by using batch normalization. It also introduces additional costs at the training time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers consider the authors' approach to pruning of convolutional networks reasonable; but neither sufficiently novel nor sufficiently well explored for inclusion in the conference.  In particular, the reviewers would like to see a more explicit discussion of the effect on training time of the authors' method, and more discussion and comparison against previous probabilistic pruning methods.
Decision: Reject