Filter pre-pruning for improved fine-tuning of quantized deep neural networks
Authors: Jun Nishikawa
Ryoji Ikegaya
Venue:  (2021)
Keywords: Deep Neural Networks
Quantization
Quantize
Pruning
MobileNet
compression

Abstract: Deep Neural Networks(DNNs) have many parameters and activation data, and these both are expensive to implement. One method to reduce the size of the DNN is to quantize the pre-trained model by using a low-bit expression for weights and activations, using fine-tuning to recover the drop in accuracy. However, it is generally difficult to train neural networks which use low-bit expressions. One reason is that the weights in the middle layer of the DNN have a wide dynamic range and so when quantizing the wide dynamic range into a few bits, the step size becomes large, which leads to a large quantization error and finally a large degradation in accuracy. To solve this problem, this paper makes the following three contributions without using any additional learning parameters and hyper-parameters. First, we analyze how batch normalization, which causes the aforementioned problem, disturbs the fine-tuning of the quantized DNN. Second, based on these results, we propose a new pruning method called Pruning for Quantization (PfQ) which removes the filters that disturb the fine-tuning of the DNN while not affecting the inferred result as far as possible. Third, we propose a workflow of fine-tuning for quantized DNNs using the proposed pruning method(PfQ). Experiments using well-known models and datasets confirmed that the proposed method achieves higher performance with a similar model size than conventional quantization methods including fine-tuning.

Under review as a conference paper at ICLR 2021
FILTER PRE-PRUNING FOR IMPROVED FINE-TUNING OF QUANTIZED DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Neural (etWorks(DNNs) have many parameters and activation data, and these both are expensive to implement. One method reduce the size of the DNN is to quantize the pre- pre-trained model by using low-bit expression for weights and activations, using fine-tuning recover the drop in accuracy. However, generally difficult to train neural networks which use low-bit expressions. One reason that the weights in the middle layer of the DNN have a wide dynamic range and so when quantizing the wide dynamic range into few bits, the step size becomes large, which leads to large quantization error and finally large degradation in accuracy. To solve this problem, this paper makes the following three contributions without using any additional learning parameters and hyper- parameters. First, analyze how batch normalization, which causes the afore- mentioned problem, disturbs the fine-tuning of the quantized DNN. Second, based these results, we propose new pruning method called Pruning for Quantiza- tion (PfQ) which removes the filters that disturb the fine-tuning of the DNN while not affecting the inferred result as far as possible. Third, we propose work- flow of fine-tuning for quantized DNNs using the proposed pruning method(PrQ). Experiments using well-known models and datasets confirmed that the proposed method achieves higher performance with a similar model size than conventional quantization methods including fine-tuning.
INTRODUCTION
DNNs (Deep Neural Networks) greatly contribute to performance improvement various tasks and their implementation in edge devices required. On the other hand, a typical DNN (He all 2015 Simonyan & Zisserman, 20155) has the problem that the implementation cost is very large, and is difficult operate it on an edge device with limited resources. One approach this problem is to reduce the implementation cost by quantizing the activations and weights in the DNN.
Quantizing DNN using extremely low bits, such as or 2 bits has been studied by Courbariaux al. (015) and et al. (2019). However, it is known that while such a bit duction has been performed for a large model such as ResNet (He et al. 2015), it has not yet been performed for a small model such Howard et all 2019), and is very difficult to apply to this case. For models that are difficult to quan- tize, special processing for the DNN is required before quantization. On the other hand, although fine-tuning is essential for quantization with extremely low bit representation, few studies have been conducted on pre-processing for easy fine-tuning of quantized DNNs. In particular, has been ex- perimentally shown from previous works (Lan et al. 2211 Frankle & Carbin, 20188) that, regardless of quantization, some weights are unnecessary for learning or fact disturb the learning process. Therefore, focused on the possibility of the existence of weights that specially disturb the fine- tuning quantized DNN and to improve the performance of the quantized DNN after fine-tur tuning by removing those weights.
MobileNet (Howard all 2017: Sandler et al., 2018

---

Under review as a conference paper at ICLR 2021
Batch Normalzation(
(
Convolution
Convolution
BN(x) C
Activation
Quantization
CoO.
Mou;
BN output approximation
Pruning for Quantization (PfQ)
running var (in next BN)
2
Viz
Yiu
x ((33)
W (0.3) x ((0,2)
Ri 0
x ((2))
X W
+b(
x W
+
B
Wro0
( (
Quant (uun)
Figure I: This is a diagram the proposed pruning method PfQ. As mentioned in subsection 3.1 the filters with very small running variance approximate its output by A3. At this time, by pruning the filters and correcting the bias of the next convolution using B3 as described above, the DNN be pruned without affecting the output as far as possible.
RELATED WORK AND PROBLEM
2.1 CONVENTIONAL QUANTIZATION WORKS
In order to reduce the implementation cost of DNN in hardware, many methods of reduction by pruming (Molchanov et al. 2019. You all, 2019 He et al. 2019) and quantization (Gu al. 2019 Gong et al_ 2020: Fan et al.. 2020: Uhlich et al. 2019 Sheng et al. 2018: Nagel et al.. (019) have been studied. Research is also underway in network architectures such as depthwise convolu- tion and pointwise convolution (Howard et all 2017), which are arithmetic modules that maintain high performance even when both the number of operations and the model capacity are kept small (Howard et al., 2017: Sandler et al., 2018: Howard et all 2019). Knowledge Transfer (Hinton all 2015; Zagoruyko & Komodakis, 2016: Radosavovic et al. 2018 Chen all. 2019) also being studied as special learning method for achieving high performance in these compression technolo- gies and compressed architectures.
There have been various approaches to studies in DNN compression using quantization. DNN quan- tization is usually done for activations and weights in DNN, As basic approach to quantization, there are some methods for a pre-trained model of which one uses only quantization (Sheng al, 2018 Nagel et all 2 and another uses fine-tuning (Gong al., 2019 Gu et al. 2019: Jung al.
2019: Uhlich et al. 2019: 9 Gong et al., 2020 Fan et al. 2020). Cardinaux et al. (002)) and Han et al (015) propose a method for expressing values that are as precise as possible. However, the sim- plest expression method is easier to calculate and implement in hardware, so expressions to divide the range of quantized values by linear value (Jaobb all. 2018: Sheng et al., 2018 Nagel al.)
2019: Gu et al. 2019; GGong et all, 20019; 2020: Jung et al. 2019) or a power of 2 value (Lee et al.,
2017: Li al. 2019) are widely used present in quantization research.
It is known that fine-tuning is essential to quantize DNN with low bits in order racy as much as possible. Jacob et all (2018) shows that an accuracy improvement can be expected by fine-tuning with quantization. Since the quantization function is usually non non-differentiable the gradient in the quantization function is often approximated by STE (Bengio et all (013).. However, the case of quantization with low bits, the error very large in the approximation by STE, and is known that it is difficult to accurately propagate the gradient backward from the output side to the input side, making learning difficult. In order to solve this problem. Gong et al. (001)), Fan et al. (202h) and Darabi et al. (2018 ( proposed the backward quantization function that the gradient can be propagated to the input side as accurately as possible. et (019) and Banner et al (2018) considered activations and weights in DNN random variables, and proposed approach to op- timize the stochastic model behind DNN so that activations and weights in DNN become vectors which are easy quantize. Jung et al. (019) and Uhlich et al. (2019) considered that is important
decrease accu-
2

---

Under review as a conference paper at ICLR 2021
for performance determine the dynamic range of vectors be quantized appropriately, and it is proposed that this dynamic range is also learned as learning parameter.
2.2 PROBLEM OF WIDE DYNAMIC RANGE ON QUANTIZATION
the dynamic range appropriately in the quantization. In modern network chitectures, most of the computation is made up of a block of convolution and batch normalization (BN). It is known that the above block can be replaced by the equivalent convolution, and quanti- zation is performed after this replacement. Thus, it is possible to suppress the quantization error rather than quantizing the convolution and BN separately. On the other hand, it is also known that the dynamic range of the convolution may be increased by this replacement (Sheng et al., 2018 Nagel et al. 2019). In order to explain this in detail, the calculation formulas of convolution and BN in pre-training are described. First, the equation for convolution is described as follows
important
C(assk)
-E>
W(o.i,u.o) ( (i.J + b(o)
(1)
(a,,0)
where C(oj..) is the o-th convolution output feature at the coordinate (j, k), ((o,i,a,)) is the weight of the kernel coordinates (u, v) the i-th input map, and tu..-tt.kko))) is the input data. For sim- plicity, Cooja, is represented as vector (o) = (C(o, ().., ((o,N) 0 and ((o,i,a)) is represented as vector ((,,)) if is not necessary. Next, the calculation formula for BN described as
C(O) M()
(
(o ((0) + B(0),
where ((o)) is the o-th BN output feature M(o) (o) are respectively calculated by equation (3),
during the training or by equation 7 ( during the inference. *(o) 8(o) are respectively scaling factor and shift amounts of BN. The mean and variance are calculated as
N
1
(1)) =
(o,s)
(3)
N
(o))
((o.n)
(())
(4)
N
At the time of inference or the fine-tuning of quantization for the pre-trained model, BN can be folded into convolution by inserting equation ①1 into 2 The new weight (i(o,.,,,))) and bias (,0z) the convolution when BN folded the convolution are calculated by
No)
(o,s)
((,
(5)
Vv te
((o)
?(0
bio) + 1(0)
Ho) M(T) Vr(l +
Vv(a)
(0)
+c
where ( v(t) are respectively the mean (running mean) and variance (running variance) of BN (o) used at inference, and the values are calculated as
M = M + Moo) -1
(7)
N ((T) = v p+ )2 1
(8)
where 7 is an iteration index, and is omitted in this paper when not specifically needed, and p is hyper-parameter of BN in pre-training so that < p 1. The initial values of 7 and are
before learning in the same way as weights such as convolution.
Sheng et al. 2018 pointed out that the dynamic range of the weights of depthwise convolution is
N is the number of clements in channel C(o)-
Vector and scalar operations are performed by implicitly broadcasting scalars.
3

---

Under review as a conference paper at ICLR 2021
large, which leads to the large quantization error. They also pointed out the relationship that the filters (sets of weights) with wide dynamic range correspond to the running variances the BN with small magnitude. [agel et al. (2019) attacked this problem by adjusting scales of weights of consecutive depthwise and pointwise convolutions without affecting the outputs of the pointwise convolution. However, they cannot solve the problem in quantization completely.
In this study, we theoretically analyze the weights with the running variances which have small magnitude and show that certain weights disturb the fine-tuming tin-tuning of the quantized DNN. Based on this analysis, we propose a new quantization training method which can solve the problem and improve the performance (section[4) 4
3 PROPOSAL
3.1 WEIGHTS DISTURBING LEARNING
We analyze how the fine-tuning is affected when the following condition holds
(L,T) O,
(e)
v-(L,r) is the rumning variance (8) of eth-channel in Lth-BN. When ( holds regarding where (c)
significantly large т, the term of the left side and the first term of the right side in (8) close to
zero. Therefore, it is necessary that the second term of the right side in (8) close to zero to hold (
As the result, the variance (o())) 2 0 holds, the following equation
(C(en)
(L) (())
=0
(10)
holds for arbitrary n E (1,-..,N) N) and input data from Therefore, in the case of (o(c))² ~ 0, since (T())) )
4 in 2 is usually satisfy > 0.
<< holds, the following equation
(D) (c)
B
(11)
holds from 2 and 10). Since the output of BN is a constant value by |II regardless of the input data, the following convolution can be calculated as
ΣΣ ipic
w (L+1) (L+1) (a.i.n)) ( (Bj+uktt) Σ
(L+1)
Act(3(()) + 1
+6
(o.j.)
=
w (o.c..)))
(u.v)
(a.e)
Σ
w (L+1) (L+1)
(oaaan) "oosa.")" (k.+t.btt)) ((++1)
(13)
(u.r)
((L+1)
(0) b (1+1)
(0)
+ Uw (L+1)
(ww(tt))
(0,0) .g0)
(14)
U(w (1+1)
(a.c) (((()
Σ
w ( Act( (c)
(L+1)
(15)
(u,e)
where Act(+) is the activation function. being (14). The bias of (14) is updated as
This equivalent to the bias of the L + 1-th convolution
8 ((++))) = (()) b(6) (L+
+ U(w (b+1,-1111 (a.e) ; g((,r-1))
Ab + AU (W((Lt (1+1,r-1) s(b,T-1))
(c)
(16)
where 7 is an index of the iteration to express the bias before and after the update. At the step of quantization, since each of the first and second terms of (14 is quantized, the quantization error tends to be larger than when the whole of (4) is quantized. Further, addition to the quantization error of (14), equation (16) adds approximation error in approximating the gradient of the quantization function to the third and fourth terms, respectively. Therefore, it can be seen that the quantization
error of i ((6+1) tends to be larger than that of the other weights in the fine-tuning of the quantized DNN. From the above, it can be seen that the filters satisfying 99 disturb the fine-tuning.
4

---

Under review as a conference paper at ICLR 2021
3.2 PROPOSAL OF NEW PRUNING METHOD TO REMOVE WEIGHTS DISTURBING TRAINING
As mentioned in subsection weights that satisfy (o) ~ 0 extend the dynamic range and disturb the fine-tuning of the quantized DNN. Therefore, we propose a new pruning method, illustrated in Figure called Pruning for Quantization (PfQ), that solves the described problem by removing the weights that satisfies the following conditional
((,,) <E (0)
(17)
for the running variance v=(1,5) of the L-th BN, correcting the bias of the next convolution without affecting the output of the DNN as far possible. The reason behind this requirement for pruning is
that the term controlling the magnitude of the denominator becomes e at this time, increasing the dynamic range of that weight.
If the convolution has bias, we replace the value with the one calculated by (14). Thus, at the time of the fin-ttnning of the quantized DNN, the second term in (I6) is absorbed into the first term, and the fourth term disappears by pruning, thereby eliminating the bad effect
the fine-tuning
discussed in subsection 3.1 When a BN is used, a convolution before the BN often does not have a bias in the convolution immediately before the BN. In this case, the / of the BN is replaced by
(L-1))
= -1((+t1) + =P())
Σ
w (a.caa)) Act(s(())
(1+
(18)
(((+
(0) + (u,v)
When we quantize the activations, we use the quantized (ct(3(()) in 15 and 18) pruning method was summarized in Figure
The proposed
3.3 PROPOSAL OF QUANTIZATION WORKFLOW
Also, our quantization workflow can get the effect of BN. We propose a new quantization workflow (Algorithm D in this section. In general, DNNs are quantized simultaneously for activations and weights for the pre-trained model with float. However, the fine-tuning may not proceed well by this method since the variation from the pre-trained model is too large. Therefore, we fine-tune the quantized DNN gradually in the proposed quantization workflow. First, fine-tune the DNN for quantized activations only, then for quantized activations and weights the DNN. Furthermore, in general, BN is folded into the convolution when the weights quantized. This does not allow to get the effect from BN in the fine-tuning of the quantized DNN. On the other hand, our quantization workflow allows the first fine-tuning using BN without folding BN into the convolution since the fine-tuning of the DNN for quantized activations only is performed at first. By performing PrQ for the DNN before each fine-tuning, the problem claimed by (Sheng et all 2018) that the dynamic range of the weights are widen and the problem claimed by subsection 3.1 in this paper that some biases tends to accumulate the quantization and approximation error are solved. Regarding second PfQ, since BN still remains in the fine-tuning of the DNN for quantized activations, the fine-tuning may generate some filters with small variance. In order to remove these filters, the same processing the first PIQ is performed.
EXPERIMENTS
In this section, we experimentally confirmed the performance of the quantization method proposed in subsection 3.2 and 3.3 First, we explain the setup of the experiment. Next, in order to confirm the effect of the weights which disturb the fine-tuning, we compare the performance of the models whose activations are quantized and fine-tuned. The difference of these models is that some weights in the model are removed by PfQ or not. Finally, we compare the performance of our proposed quantization method with the other quantization methods.
5

---

Under review as a conference paper at ICLR 2021
Algorithm Proposal of Quantization Workflow
Input: pre-trained model, dataset, training hyperparameters (bitwidth, epochs e ew, the other training parameters)
Output: quantized model
1: Perform PfQ operation to the pre-trained model
2: for = 0 to ед do
3: Fine-tune the pre-trained model for quantized activations
4: end for
5: Perform PfQ operation to the fine-tuned model for the quantized activations
6: Fold BN
7: for i= 0 to ew do
8: Fine-tune the model for the quantized activations and weights
9: end for
return quantized model
4.1 EXPERIMENTAL SETTINGS
We experiment with the classification task using CIFAR-100 and imagenet1000 as datasets. The learning rate was calculated by the following equation
e w
)) (e 2 w)
L
X
1 + cos
Le
A
(19)
1
(e < w);
w
where the number of epochs, [-1 is the initial learning rate, is the number of warm-up epochs, and X is the period of the cosine curve. The learning rate was reduced by cosine curve [Loshchilov & Hutter, 2016) during learned 100 epochs in the each fine-tuning in Algorithm [ respectively. We used DSQ (Gong et al,, 2019) PACT (Choi et al] 2018 Wang et al. 2018) and DFQ (Nagel et al., 2019) as quantization methods to compare with our proposed method. The performance of DSQ and PACT are directly referenced from the values the paper, but regarding DFQ. the performance the fine-tuning of the quantized DNN not described in the paper. Therefore the performance of DFQ was obtained by our own experiment with 200 epochs fine-tuning.
w. In the following experiments, the proposed method
In this paper, we use the quantization function given by the following
round min(mar(z; m), M) m
Q(x)
scale +
(20)
scale
M m 2
scale
(21)
where x represents the input data (an activation or a weight) of the quantization function, n repre- sents the bit width, and m, M represent the lower limit and upper limit of the quantization range, spectively. We use STE to calculate the gradient. In this paper, we use MobileNetV (Howard et al.. 2017) and MobileNetV2 (Sandler et all 2018) the network architectures. At the time of quanti- zation, for quantization of activations, all activations except the last layer and addition layers (i.e. outputs of skip connection) were quantized, and for weights, all convolutions, depthwise convolu- tions and affines including the first and last layers were quantized.
In this study, = 0.00001 was used as 6 in (1)). This the default value of the BN in NNabla (Sony).
4.2
ABLATION STUDY
4.2.1 EFFECT OF DISTURBING WFIGHTS
In this section, made comparative experiment to confirm that PfQ eliminates the bad effect the fine-tuning of the quantized DNN described in subsection B.]] In addition, in order to confirm the effect of the bias correction in PfQ, made the experiment even in the where the bias

---

Under review as a conference paper at ICLR 2021
correction was excluded. We made experiments for MobileNetVI and MobileNetV2 for the clas- sification task of CIFAR-100. In the experiment of CIFAR-100 100 in this paper, the GPU was single (2080Ti) and the batch size 16. We made the pre-trained models of the two network architec- tures by the following experiment: the number of epochs was 600, the optimizer was momentum SGD, the moment
0.9, the learning rate settings in (T9) were 1 0.16, = 0, À 40, and the learning rate decay was not used. In the fine-tuning of the quantized DNN the following settings were used: the number of epochs was 100, the optimizer was the same as the pre trained model, the learning rate settings in 19) were I_ = 0.001, w 0, A 100, and the weight decay was 0.0001.
Table I: Topl Accuracy comparison between pruned and no pruned models
PfQ
Model
Activation bitwidth
Evaluation
original
PfQ
(w/o bias correction)
Accuracy
76.34
76.67
76.75
MobileNetV1
4
MAC
510M
367M
367M
Accuracy
74.17
75.14
75.26
MobileNetV1
3
MAC
510M
367M
367M
Accuracy
74.04
74.67
75.00
MobileNerV2
MAC
147M
93M
93M
Accuracy
71.36
72.17
72.28
MobileNetV2
3
MAC
147M
93M
93M
The experimental results are summarized in Table 0 Regarding the accuracy in Table I Table shows that the fine-tuning after using PfQ gives better performance in all experiments. and can be solved that the bad effect on the fine-tuning of the quantized DNN described in [31] In particular, PfQ (w/o bias correction), the accuracy higher than that of the original, and the difference is larger for 3 bits than 4 bits, indicating that the influence of the quantization error (i.e. the 4th term in (66) in the backward calculation large. MobileNetV1 and MobileNetV2 in the columns of PfQ in Table are pruned with the weights by 31.86% and 35.74%, respectively. Thus, the number of MACs is also reduced.
4.2.2 SOLVING DYNAMIC RANGE PROBLEM
In this section, we confirmed that PfQ can suppress the increase in the dynamic range of the weights after folding BN. The models that we used are MobileNetV1 and MobileNetV2 used in Table I
MobilenetV1
MobilenetV2
34
10
100
DC
022
500
500
800
500
500
000
OC10
KC0
0C11
PCU
DC22
PCZ2
DC13
PCU
UOL Lin Lnoope
pectine celt and dest coec
tivert 1ie
Lintt
Figure 2: This is a figure comparing the dynamic range of the weights after folding BN with re- spect using not using PfQ. The left figure is about MobileNetVI and the right one is about MobileNetV2. The blue bin is not using PfQ and the red one is using PfQ. The vertical axis the dynamic range value (i.e. weight figure of MobileNetVI and MobileNetV2, the left side on the horizontal axis is the layer on the input side.
weight min), and the horizontal axis is the layer. In each
From the results in Figure it can be that PfQ improves the wide dynamic range.

---

Under review as a conference paper at ICLR 2021
4.2.3 EFFECT OF PROPOSED QUANTIZATION WORKFLOW
In this section, we confirm the effect of Algorithm | Table 2 shows the results of comparing the performance between quantization using Algorithm M and the fine-tuning of the DNN with activa- tions and weights quantization simultaneously once after performing PfQ. The experiments are the classification task of CIFAR-100 using MobileNetVi and MobileNetV2. In the each fine-tuning in Algorithm the number of epochs was 100, the optimizer was momentum SGD, the moment was 0.9, the learning rate settings in (99) were w 0, A 100. Regarding the fine-tuning once with PfQ. the number of epochs is 200 in order to be the same number of the total epochs in Algorithm
Table 2: Effect of proposed quantization workflow
Bitwidth (Activatioo/Wcight)
fine-tune once
Model
Evaluation
Algorithm
(w/ PfQ)
Accuracy
74.69
75.31
MobileNetVI
A4/W4
MAC
367M
367M
Accuracy
71.56
70.85
MobileNetV1
A3/W3
MAC
367M
367M
Accuracy
72.78
72.7
MobileNetV2
A4/W4
MAC
93M
92.4M
Accuracy
66.29
68.96
MobileNetV2
A3W3
MAC
93M
93M
Table 2 shows that Algorithm D better. Although the performance of Algorithm | is lower than that of the fine-tuning once with PfQ for A3W3 of MobileNetV1 and A4W4 of MobileNetV2, the difference is small, and Algorithm[]] [ which can often reduce the MAC is considered to be better.
4.3 COMPARISON TO OTHER METHODS
In this section, we apply the proposed quantization workflow using PfQ to the pre-trained model. Here, we compare the performance of the proposed method with that of the other quantization methods. The experimental settings common to each are described below. We used MobileNetV] and MobileNetV2 as a network architecture. epochs was 100, the optimizer was momentum SGD, the moment was 0.9, the learning rate settings in (I9) were w rate settings in 19 were 0, À 200. The optimizer, the batch size, the initial learning rate 1-1 and the weight decay settings were the same settings as Table and Table 4
the each fine-tuning Algorithm II the number of
100. In the experiment of DFQ, the number of epochs was 200, the learning
4.3.1 CIFAR- 100 CLASSIFICATION EXPERIMENT
In this subsection, we compare the performance of the proposed quantization method with DFQ us- ing fine-tuning in the classification task of 110AR-100. Next, we explain the experimental settings. We used single GPU (2080Ti) for the learning in CIFAR-100. The same pre- model as subsection 1 was used. In our method, we used the same settings as subsection 1.2. the first fine-tuning for quantized activations only in Algorithm || In the settings of the second fine-tuning for quantized activations and weights in the Algorithm II the optimizer was SGD (i.e. the moment was 0), l-1 = 0.0005, the weight decay was 0.00001. In the settings DFQ, the optimizer was SGD, 1_1 = 0.001, and the weight decay was 0.0001.
The experimental results are summarized in T which shows that the performance of the pro- posed method is better than DFQ with fine-tuning- Also, 2 PfQ operations in the proposed quan- tization workflow pruned the weights by 36. 17% and 34.36% in the A4/W4 and A3/W3 models, respectively. Thus, the number of MACs was also reduced.
4.3.2 IMAGENET 000 CLASSIFICATION EXPERIMENT
In this subsection, compare the performance of the proposed quantization method with the other quantization method in the classification task of imagenet1000. The methods used for comparison DSQ and PACT which achieve high performance with bits in the activations and weights
8

---

Under review as a conference paper at ICLR 2021
Table 3: Topl Accuracy comparison CIFAR-100
Bitwidth
DFQ
Model
Evaluation
Ours
(Activation/Weight)
(w/ fine-tune)
Accuracy
71.57
75.31
MobileNetV1
A4/W4
MAC
510M
367M
Accuracy
61.58
70.85
MobileNetV1
A3/W3
MAC
510M
367M
Accuracy
72.14
72.7
MobileNetV2
A4/W4
MAC
147.2M
92.4M
Accuracy
68.05
68.96
MobileNetV2
A3/W3
MAC
147.2M
93.0M
for MobileNetV2, and DFQ. Next, the experimental settings described. The experiment of the imagenet 1000 was learned using GPUs (tesla v100 x4), the optimizer was momentum SGD, the moment 0.9, and the weight decay was 0.00001. The pre-trained model was learned by the following settings. The number of epochs was 120, the batch size was 256, the learning rate settings 104. The settings for the fine-tuning of the quantized DNN are described below. In our method, in addition to the above settings, the batch size was 64, and the initial learning rate (-1 0.01 in the first fine-tuning for quantized activations only in Algorithm [ In the second fine-tuning for quantized activations and weights in the [ the initial learning rate was 1_1 0.0001. In the experimental settings DFQ, the batch size was 128, the initial learning rate was 1_1 0.0001.
in (19) were 1-1 0.256, w 16,
Table 4: Topl Accuracy comparison imagenet1000
Bitwidth ActivationnWeiggn))
DFQ
Evaluation
PACT
DSQ
Ours
(w/ fine-tune)
Accuracy
57.44
61.39
64.8
67.48
A4/W4
MAC
313M
313M
313M
261M
The experimental results summarized in Table 4 which shows that the quantization performance of the proposed method is the best compared to the other quantization methods. In our approach, 2 PřQ operations in the proposed quantization workflow pruned the weights by 10.29%. Thus, the number of MACs was also reduced.
These results show that PfQ and the proposed quantization workflow make it easy to fine-tune the quantized DNN and make high-performance models even with low bits. On the other hand, in this method, STE is used as the gradient of the quantization function, and the problem of degradation of the gradient due quantization, which is often raised a problem in the study of quantization, is not addressed. Therefore, further performance improvement can be expected a technique to cope with gradient degradation during the fine-tuning of the quantized DNN such as DSQ used together with the proposal of this paper.
5 CONCLUSION
In this paper, we propose PfQ to solve the dynamic range problem in the fine-tuning of the quantized DNN and the problem that the quantization errors tend to accumulate in some bias, and we propose new quantization workflow including PfQ. Since PřQ is type of filter pruning, it also has the effect of reducing the number weights and the computational costs. Moreover, our method very easy to use because there conventional pruning and quantization methods.
additional parameeassmeters and learning parameters like the
REFERENCES
Ron Banner. Yury Nahshan, Elad Hoffer. and Daniel Soudry. ping for integer quantization of neural networks. CoRR, abs/18100 100.083,, 2018. URL
ACIQ: analytical clip-

---

Under review as a conference paper at ICLR 2021
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint (tf.::308.3333 2013.
Fabien Cardinaux, Stefan Uhlich, Kazuki Yoshiyama, Javier Alonso García, Lukas Mauch, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Iteratively training look-up tables for network quantization. IEEE Journal of Selected Topics in Signal Processing, 14(4))860-87) 2020.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3514-3522 2019.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR, abs/1 805.06085 2018. (RRt::/:://arx: org/ abs 06085
Matthieu Courbariaux, Yoshua Bengio, and Jean- David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. BNN+: improved binary network training.
CoRR, abs/1812.11804 2018. URL
http://arxiv. org/ wtwpg///a///....... 11800
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. ArXiv, abx/2004.07320 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net- works. CoRR, Abs/1803.00635, 2018.
Cheng Gong, Yao Chen, Ye Lu, Tao Li, Cong Hao, and Deming Chen. Veeq: Minimal loss dnn model compression with vectorized weight quantization. IEEE Transactions Computers, 2020.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 4852-4861, 2019.
Jiaxin Gu, Junhe Zhao, Xiaolong Jiang, Baochang Zhang, Jianzhuang Liu, Guodong Guo, and Ron- grong Bayesian optimized 1-bit cnns. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4909-4917, 2019.
Song Han, Huizi Mao, William Dally, et al. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffiman coding, 56(4):3-7, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/15122 122003385 2015.
03385
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv 2015.
preprint 1503.vv5500.0000
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Quoe Le, and Hartwig Adam. Searching for mobilenetv3. CoRR, abs/1905.02244, 2019. URL
http::///axxvv.ooo wttpv//orr//////11.0222222 1ttp///10000222
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, To- bias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017. URL
http://arxisw http://aaaavvv.ro/////////0
10

---

Under review as a conference paper at ICLR 2021
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for inference. In Proceedings of the IEEE Conference on Computer
efficient integer-aritmmetee Vision and Pattern Recognition, pp. 2704-2713, 2018.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. +350-4359, 2019.
Janice Lan, Rosanne Liu, Hattie Zhou, and Jason Yosinski. Lea: Loss change allocation for neural network training. In Advances in Neural Information Processing Systems, pp. 3619-3629 2019.
Edward H Lee, Daisuke Miyashita, Elaina Chai, Boris Murmann, and S Simon Wong. Lognet: Energy-efficient neural networks using logarithmic computation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5900-5904. IEEE, 2017.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non- uniform discretization for neural networks. In International Conference on Learning Representa- tions, 2019.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. abs/1608. 03983, 2016. URL http: httt::/tttt/iiii //tps//aavvvvvvviv httg//bb//66l/ 03983
CoRR,
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1264-11272, 2019.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantiza- tion through weight equalization and bias correction. CoRR, abs/ abs/1190.04421, 2019. URL
Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 41 4119-4128, 2018.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and seg- mentation. CoRR, abs/1801.0423 2018. /tt////cee///aatt....oo///////// /0001. 04381
Tao Sheng, Chen Feng, Shaojie Zhuo Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. A ciamiizattio-friennii separable convolution for mobilenets. CoRR, 0bbs/1803.08607, 2018. URL
http: wttp://////////////b////// 08607
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Sony. Neural network libraries (nnabla).
Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is good parametrization. arXiv preprint arXiv:/905. 1452, 2019.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware-aware automated quan- tization. CORR, 0/5/1811.08888 2.18.
08886 httptt:://axxw..or///aaaaaa
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems. pp. 2133-2144, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: mance of convolutional neural networks via attention transfer. arXiv preprint 2016.
Improving the perfor-
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 EXPERIMENTS USING VALIDATION SET
Algorithm Tanihhmmneee needs the fixed number of epochs in the experiments. So we made the experiments not using the fixed number of epochs and using the value of accuracy drop. The accuracy check during the training used validation set apart from training set and set. We split separate validation set out of the original training set by randomly sampling 50 images from the training set for each category. set the accuracy drop 0.5% the each fine-tuning Algorithm D and 1 1.0% in DFQ. Then the end numbers of epochs were 82 and 80 during the each fine-tuning in Algorithm and the end number of epoch of DFQ was 200 that was we set max epochs. The result is Table 5
Table 5: Using Accuracy Drop
Bitwidth (Activation/Weiign)
Model
Evaluation
DFQ
Ours
Accuracy
70.00
71.58
MobileNetV2
A4/W4
MAC
147.2M
92.4M
A.2 VISUALIZE Low VARIANCE CHANNELS
We claimed in subsection 3.1 that the outputs of some channels whose running variance close to zero are the constant values. In this subsection, we confirmed the fact and the result was Figure 3 Figure shows that the channels whose running variances are close to zero are the constant value. The model we used Figure 3 was the pre-trained model in subsection 4.2.1 The input images are in the CIFAR- 100 training set.
V
V 0
input image apple
ch 3
ch 5
ch 6
orange
Figure 3: This the visualized channels of the first BN outputs after the first convolution Mo- bileNetV2. The input images are apple and orange, respectively. The running variance of the channel 3,5,6 are close to zero and that of the channel 0,1,2 are not close to zero.
12


Reviews:
[{"review_id": "QKNhB_nddwr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a pruning for quantization (PfQ) scheme to improve the fine-tuning of the quantized network. PfQ removes the filters that disturb the fine-tuning of the DNN while retaining good performance. Some experiments are conducted to demonstrate the effectiveness of the proposed PfQ.\n\nPros:\n- The paper is well motivated and the method seems valid.\n\nCons:\n- The writing quality is poor. in its current form, a general readership will struggle to understand it. An expert familiar with the details of the field (not just the general area) can probably disentangle the text, but otherwise, it's too convoluted.\n- This paper is over-claimed. For example, how BatchNorm disturbs the fine-tuning of the quantized network has been pointed out by [1]. The analysis presented by the authors is too incremental.\n\nSome remarks:\n- Please note that BN can not be absorbed into the previous convolution layer in the training stage. The mean and variance in Eq.(2) should be running-moving average statistics, which should be pointed out in the beginning.\n- To improve the quality of this paper, more rigorous mathematical notations and some visual experiments should be provided.\n- It would be better to show some ablation studies to show how your method works.\n\n[1] Data-free quantization through weight equalization and bias correction. Markus Nagel et al.\n\n**********After rebuttal\n\nThe revised version has a better shape. In particular, I like the analytical experiment (Fig.2), which demonstrates that the proposed scheme can improve the wide dynamic range. Overall, this paper observes that BN with small variance influence quantization and proposes a protocol for training a quantized neural network combining filter pruning. \n\nSome issues still prevent it from being accepted. For example, PfQ is proposed to reduce the dynamic range. However, there is even no definition of the dynamic range in the paper which may make readers hard to understand the mechanism of PfQ.  Besides, the author claim that the weight widening the dynamic range in quantization is theoretically analyzed. But the analysis of Eqn. (14-16) is less rigorous. The readers may expect to see how weights in the case of $V_{c}^{L,\\tau} \\approx 0$ increase the dynamic range according to its definition compared to those weights in the other cases ($V_{c}^{L,\\tau} \\gg 0$).\n\nThe paper proposes an effective approach of quantization, which reduces the model size and improves accuracy. I would like to increase my rating to 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ks2tW-5vOZ9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effect of quantization during training together with batch normalization in quantized deep neural networks. The compound effect of convolution and batch normalization on the dynamic range of activations has implications on the progress of training. The authors propose a protocol for training a quantized neural network combining filter pruning, fine tuning and bias correction. The experiments show that the model size is reduced significantly, while keeping or improving the accuracy.\n\nStrengths\n- The paper is clearly presented, and the effect of batch normalization in the dynamic range of a quantize layer is interesting.\n- The method, in the paper settings, is effective in reducing the model size and sometimes improving accuracy.\n\nWeaknesses\n- There is not qualitative analysis of the problem of the dynamic range and how the proposed method alleviates it. \n- The novelty is limited in my opinion. The paper combines typical practices used in other works. For example, pruning filters with small magnitude is a common practice. it is also common to fine tune a network after pruning or compression, which not only reduces model size but often also improves the accuracy due to lower overfitting. And bias correction to account for the reconstruction error is also commonly used in similar cases (e.g. DFQ, Finkelstein2019, Masana2017). \n- The number of iterations in the experiments is a fixed number. It would be more convincing using a validation set with early stopping.\n- The pruning+fine tuning seems to be done once (twice in the proposed workflow). In that case the comparison may not be fair without several iterations of pruning+fine tuning, to compare models when they cannot be further pruned. \n\nOverall, I think the novelty of the paper is limited, with concerns about the experiments.\n\nQuestions\nPlease address weaknesses\n\nFinkelstein et al., Fighting Quantization Bias With Bias, ECV@CVPR 019 \nMasana et al., Domain-adaptive deep network compression, ICCV2017\n\n-- Post rebuttal\n\nI appreciate the response by the authors and the new experiments. I also read the other reviews and responses. I think the paper has improved in the revised version. However, I'm still concerned about the novelty, which still remains relatively incremental, as also pointed by other reviewers. I update my rating to 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "A9PzPA_joMn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to prune certain channels to improve the accuracy of quantized DNN model. The motivation comes from the observation that the channels which have small variance are actually harmful to the quantization-aware training. The authors show that these channels with small variance can be pruned without significant influence to the neural network, and the accuracy loss can be easily recovered with fine-tuning. \n\nPros:\n\nThis paper makes an important observation that channels with small variance lead to a problem that the filter weights will have a wide range after fusing the BN layer into convolution. Directly performing quantization on the weight tensor will lead to a large quantization scale and large quantization error.\n\nThe authors find that the channels with small variance can actually be removed. Based on their following BN operations, these channels can be seen as constant channels approximately, and these constants can be folded into the bias of the next layer. By this way, the neural network is not only more suitable to quantize, but also has less model size and computation.\n\nThe paper gives a study showing that removing the channels with small variance improves the accuracy of quantized model. The proposed method is also compared with several other quantization methods.\n\nCons:\n\nOn ImageNet, the experiments are only did on MobileNetV2, and only a few methods are compared. Although the current experiments show the benefits of PfQ, it is better to have more experiments to make the validation part more solid.\n\nTable 1 is to show the benefits of removing disturbing weights. Why use the activation quantization instead of weights quantization? It's better to analyze weight quantization here too.\n\nThere are some questions which are not clear in the paper:\n\n1. Will per-channel quantization have the problem discussed in this paper? What about the results of using / not using PfQ for per-channel quantization? The min-max quantization scale (eq 20) is used in this paper, what if we use learnable quantization (e.g., LSQ [1])? Is it still suffer from the channels having small variance?\n\n2. DFQ is used as the quantization method. Is this a quantization method cited in this paper? I didn't find the reference and the full name for DFQ.\n\n3. Is the proposed method only applied to DNNs which have BN layers? Is it possible to apply the proposed method to models without BN layers?\n\nOther comments:\nIt's easier for people to understand the key idea of this paper if moving Figure 1 to the first few pages.\n\nIn general I think this paper is a good work on quantization and model compression. It makes good observation of factors that influence quantization, and shows that pruning can actually help quantization in some cases. There are some issues on the experiment (See cons), it's better to validate the proposed method on more DNN architectures and compare with more quantization methods.\n\nReferences\n\n[1] Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R. and Modha, D.S., 2019. Learned step size quantization. arXiv preprint arXiv:1902.08153.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2q3M5RTvEY4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overall\nThis work present a Pruning mechanism for Quantization scenario. Duo to the low-bits effects, the quantized network is hard to train properly. Therefore, authors provide a new method call Pruning for Quantization (PfQ) and a workflow to solve the model compression problem practically. Comparing to some current quantization methods, PfQ obtains some gain from solution to the performance on benchmark datasets such as ImageNet and CIFAR100.\n\n### Pros\nClear idea about PfQ and the mathematical results provide the reason why the authors want to do so.\n\n### Cons\n1) Generally, the joint quantization (Q) and pruning (P) framework is not novel at present. For example, the following papers have tried to solve the model compression problem with P and Q jointly:\nTung, Frederick, and Greg Mori. \"Clip-q: Deep network compression learning by in-parallel pruning-quantization.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\nTung, Frederick, and Greg Mori. \"Deep neural network compression by in-parallel pruning-quantization.\" IEEE transactions on pattern analysis and machine intelligence (2018).\nWang, Ying, Yadong Lu, and Tijmen Blankevoort. \"Differentiable Joint Pruning and Quantization for Hardware Efficiency.\" European Conference on Computer Vision. Springer, Cham, 2020.\nAnd compression with the BN mechanisms are also available:\nGao, Xitong, et al. \"Dynamic channel pruning: Feature boosting and suppression.\" arXiv preprint arXiv:1810.05331 (2018).\nLiu, Yuan, et al. \"Local Normalization Based BN Layer Pruning.\" International Conference on Artificial Neural Networks. Springer, Cham, 2019.\nKang, Minsoo, and Bohyung Han. \"Operation-Aware Soft Channel Pruning using Differentiable Masks.\" arXiv preprint arXiv:2007.03938 (2020).\nTo support the claim that PfQ has the SOTA performance on benchmark datasets, first, PfQ should outperforms other BN based pruning techniques under the quantization settings. Then, PfQ should be better than those Pruning and Quatization Joint optimal methods for model compression. Especially, the readers would like to know why the variance based pruning approach is better than those BN weights based methods. It is not convincible enough for readers to follow the idea without analysis into details.\n2) It is lack of explanation why the workflow is needed for the model compression with PfQ. The reason why multiple PfQs are executed is missing. A more clear algorithm would be better for the presentation of the \"workflow\".\n3) Ablation studies: since the author design multiple round of PfQs, the paper is expected to show the studies on what will happen if the workflow just contains PfQ and reasonable finetune. Why the performance could not beat the current \"workflow\"? How would the variance distribution be after the \"workflow\"?\n4) More proof-reading will make the paper look better. For example,  in page 7, section 4.3.1, \"a single GPU (2080ti)\" => \"a single GPU (2080Ti)\". In the same sentence, \"for the learning in cifar100\" => \"for the learning in CIFAR-100\". \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four reviewers rate this article borderline. R3 finds the paper clearly presented and the method effective, but misses quantitative analysis of the dynamic range problem as well as novelty. Following the discussion and revision, she/he considers the paper improved and updated the score to 5, still being concerned about the novelty. R1 considers the paper makes an important observation but has concerns about experiments, rating it 6. R2 considers that the paper contributes a clear idea, but indicates that more analysis and supporting results are needed. She/he indicated a number of shortcomings in the initial review, and found the update good, hence tending to rate the paper higher after the responses, 6. R4 considers the paper well motivated and the method valid. However, he/she found the writing poor and over-claiming results, and that more rigorous mathematical notation would help. After the discussion and revision, he/she found the paper better and increased the score to 5, but still found issues preventing the paper from being accepted. In summary, the reviewers agree that the paper contains an interesting and well motivated method, but they also point at a number of shortcomings. The revision improved several of them but others persisted. Although the ratings improved after the discussion, the overall rating is borderline. This is a very competitive call, and hence I have to recommend reject at this time. 
Decision: Reject