Prune or quantize? Strategy for Pareto-optimally low-cost and accurate CNN
Authors: Kengo Nakata
Daisuke Miyashita
Asuka Maki
Fumihiko Tachibana
Shinichi Sasaki
Jun Deguchi
Venue:  (2020)
Keywords: CNN
Quantization
Pruning
Accelerator
Computational cost

Abstract: Pruning and quantization are typical approaches to reduce the computational cost of CNN inference. Although the idea to combine them together seems natural, it is being unexpectedly difficult to figure out the resultant effect of the combination unless measuring the performance on a certain hardware which a user is going to use. This is because the benefits of pruning and quantization strongly depend on the hardware architecture where the model is executed. For example, a CPU-like architecture without any parallelization may fully exploit the reduction of computations by unstructured pruning for speeding up, but a GPU-like massive parallel architecture would not. Besides, there have been emerging proposals of novel hardware architectures such as one supporting variable bit precision quantization. From an engineering viewpoint, optimization for each hardware architecture is useful and important in practice, but this is quite a brute-force approach. Therefore, in this paper, we first propose hardware-agnostic metric to measure the computational cost. And using the metric, we demonstrate that Pareto-optimal performance, where the best accuracy is obtained at a given computational cost, is achieved when a slim model with smaller number of parameters is quantized moderately rather than a fat model with huge number of parameters is quantized to extremely low bit precision such as binary or ternary. Furthermore, we empirically found the possible quantitative relation between the proposed metric and the signal to noise ratio during SGD training, by which the information obtained during SGD training provides the optimal policy of quantization and pruning. We show the Pareto frontier is improved by 4 times in post-training quantization scenario based on these findings. These findings are available not only to improve the Pareto frontier for accuracy vs. computational cost, but also give us some new insights on deep neural network.

Under review as a conference paper at ICLR 2020
PRUNE OR QUANTIZE? OPTIMALLY LOW-COST AND ACCURATE CNN
STRATEGY FOR PARETO-
Anonymous authors Paper under double-blind review
ABSTRACT
Pruning and quantization are typical approaches to reduce the computational cost convolutional neural network (CNN) inference. Although the idea of combin- ing both approaches seems natural, it is surprisingly difficult to determine the ef- fects of the combination without measuring performance on the specific hardware that the user will use. This is because the benefits of pruning and quantization strongly depend on the hardware architecture where the model executed. For example, CPU-like architecture with no parallelization may fully exploit the reduction of computations by unstructured pruning to improve speed, but a GPU- like massive parallel architecture would not. Further, there have been emerging proposals of novel hardware architectures, such as those supporting variable bit- precision quantization. From an engineering viewpoint, optimization for each hardware architecture is useful and important in practice, but this is in essence a brute-force approach. Therefore, in this paper, we first propose a hardware- agnostic metric for measuring computational costs. Using the proposed metric, we demonstrate that Pareto-optimal performance, where the best accuracy ob- tained at a given computational cost, is achieved when slim model with fewer parameters moderately quantized rather than a fat model with a huge num- ber of parameters is quantized to extremely low bit precision, such binary ternary. Furthermore, we empirically find a possible quantitative relation between the proposed metric and the signal-to-noise ratio during stochastic gradient de- scent (SGD) training, by which information obtained during SGD training pro- vides an optimal policy for quantization and pruning. We show the Pareto frontier is improved by 4x in a post-training quantization scenario based on these find- ings. These findings not only improve the Pareto frontier for accuracy versus computational cost, but also provide new insights into deep neural networks.
INTRODUCTION
Reducing execution cost of deep learning inference is one of the most active research topics for applying superhuman recognition in embedded loT devices and robots. A typical approach for em- ploying memory- and computation-effiiien components is separable convolution, which is com- bination of depth-wise and point-wise convolutions (landola et all 2016 Zoph et all, 2018; Zhang all 201; Howard et all 2017), structured/unstructuuee pruning of connections and activations, and quantizing activation, weight, and their vectors (Stock all R019; Jegou et al. д0ал Gong all 2014). Among these, separable convolution and structured pruning are similar, in that sepa- rable convolution can be viewed as convolutions pruned in a handcrafted manner. From a pruning viewpoint, since the separable convolution structure results from applying aggressive pruning to normal convolution, the result is drastic reductions in memory and computational cost at the ex- pense of greatly decreased accuracy Stockketal 2019). On the other hand, structured pruning and quantization are seemingly orthogonal approaches that can be naturally combined (Tung & Mori, 2018: Han all 2016). However, their interactions are still not well-studied. For instance, the use of single-bit representation is being actively explored nonnegligible accuracy drop inevitable in extreme quantization, some papers have proposed in- creasing the number of channels to compensate for the lack of expressivity etll 2017). In other words, a quantization approach can further reduce the number bits by compromising the increase in number of channels, or the increase in number of computations. This indicates that,
extreme quantization. Since

---

Under review as a conference paper at ICLR 2020
versely, reducing channels by pruning may limit capability for quantization. This discussion raises a controversial question: which is better, a fat model with smaller bit width or a slim model with larger bit width? Answering this question requires metric that fairly measures the effects of both pruning and quantization. One such metric in the literature is the inference speed when the model executed specific hardware. This metric is useful or ideal when the target hardware is known in advance but strongly depends on features of the hardware architecture. Yang et al (2IT) searched for an optimal architecture using inference time as the optimization objective and found different optimal architectures depending on the target device. For example, if the hardware cannot handle extremely low bit-widths (1 or 2 bits), instead treating them as 8-bit integers with upper bits filled with zeros, we cannot exploit the reduction of bit width to improve inference speed. From theoretical viewpoint, figuring out the extent to which we can reduce the computational complexity of deep neural networks is another important open question.
The discussion so far urges us to develop agnosti and theoretically reasonable metric for measuring computational costs of neural network architectures, In this paper, we propose the Frobe- nius norm of the effective value of weight parameters as one such metric. This metric is proportional to the total energy when the model is executed on ideal hardware, where energy consumption for (MAC) computation is proportional to the squared effective amplitude of the individual weight parameter used for the MAC computation. The basic idea of the metric is analogous to a highly efficient class-B amplifier circuit whose energy consumption is determined by the instant signal amplitude (Sechi, [976). This metric successfully reflects the effects of both quantization and structured/unstructure pruning in accordance with intuition.
single
Using the proposed metric, we empirically find that a slimmer model can achieve a far better Pareto frontier in a lower computational cost region than can a fatter model after quantization, while fat model is advantageous for achieving higher accuracy in larger computational cost region. Finally, we perform experiments under post-training quantization scenario (Banner et all, 2018) on Ima- geNet dataset (Deng et al,, 2009) to verify the validity of our claim, namely that is superior to quantize-only or prune-only for achieving better Pareto frontier.
Further, since this metric is relevant to the signal-to-nol noise ratio (S/N), it measurable during SGD training, in which the absolute value of weights and the random walk of weight parameters corre- spond to signal and noise, respectively. We observe that the dependencies of the metric on validation accuracy seem to be correlated between those during training and those applying quantization after training. From this observation, we point out some possibilities for which we could expect robust- ness of model for quantization from information obtained during training, could determine an optimal policy for quantization of that model, and we could develop a novel optimization or regularization scheme.
The main contributions of this paper are as follows:
We define hardware-agnostie metric for measuring the computational cost of pruned and quantized models.
We empirically find that models with fewer parameters achieve far better accuracy in a low computational cost region after quantization.
We show a potential quantitative relation between quantization noise and perturbation of weight parameters during SGD training.
And as implications, we hope to exploit our findings for
thorough comparison of various neural network architectures using the proposed hardware- agnostic metric.
development of method for extracting a quantization policy from information obtained during SGD training, and
development of training algorithm regularization scheme for producing robust models based on the relation between quantization noise and perturbation of weight parameters during SGD training.
2

---

Under review as a conference paper at ICLR 2020
Compun computational Cont
Reg.. Emerge compumption
CPU
GPU
Computationl Cost
ideal hardware
Ploating PaintPP) Szbrt
Computational
Dense
B-bit
Btarr
FP16
Jiooyy Integer
Dense
Sporse
Sparse
Binary
Computational Complexity
Computational Complexity
Computational Complexity
Figure 1: Left/Center: Computational cost strongly depends the hardware architecture on which the model is executed. Right: Proposed computational cost for analysis or theoretical research, assuming an ideal hardware architecture.
EFFECTIVE SIGNAL NORM
We seek metric that properly reflects the effects of both quantization and pruning. Convention- ally. quantization effectiveness is evaluated according to the number of bits required to achieve a given accuracy, or the accuracy achieved by using certain bit numbers for specific network architec- tures (Stock al M We cannot use this to compare efficiencies between different architecture models (e.g., MobileNet versus ResNet-18). The number of MAC computations or parameters can be used to compare different architectures, but the number of MAC computations does not consider quantization and the number parameters not directly related to inference time.
Recently, the use of actual or estimated inference speeds as metric for comparing network archi- tectures has been proposed (Yang et al, 201& Wang et all 2019%; Caietall all, 2019). This metric very useful when the target hardware is known in advance, and ideal for those who wish to the model that performs best on that hardware. However, this metric is strongly hardware dependent. Indeed, Yang et al] (201)); Wang et al] (20192); Cai all (2019) found that optimal architectures for different types of target hardware are totally different. Considering interest in, for example, the simplest realizable deep neural network model while achieving required accuracy, there is need for a hardware-agnostic metric.
The metric for model evaluation should correlate with energy consumed when the model is exe- cuted on ideal hardware. We assume that energy consumption by ideal hardware monotonically decreases when the bit width is reduced by quantization and when the number of nonzeros weight parameters is reduced by pruning. For example, hardware with an 8-bit integer MAC array can- be further accelerated even if the bit width reduced from 8 to or bits. Thus, the energy consumption measured using such hardware does not satisfy the aforementioned requirement and cannot our metric. Hardware like a CPU, which processes each computation serial, can natu- rally exploit the structured or unstructured sparsity of weight parameters by skipping computations with zeroed weights. However, because is difficult to parallelize computations while maintaining such strategy, it is generally difficult to benefit from sparsity in GPU-like hardware employing massively parallel MAC units. Hardware dedicated to sparse convolution (Lu et all 2019) tends to show better performance only when sparsity is sufficiently high, due to relatively large overheads for encoding and decoding sparse weight parameters in a special format.
Therefore, the benefit of sparsity from pruning and low bit width from quantization largely depends on the hardware architecture, so long as we consider only existing hardware. Because we require hardware-agnostic metric, assume ideal hardware in which energy consumption is linearly proportional to the number of nonzero weight parameters and monotonically depends on the bit width of weight parameters, as shown in Figure MI setting aside the feasibility of such ideal hardware.
2.1 DEFINITION OF EFFECTIVE SIGNAL NORM
We define a metric called the effective signal norm (ESN) as
ESN = Σ || f(w! (W" 1|
(1)
3

---

Under review as a conference paper at ICLR 2020
with W 1W/4411+0.5 where W' is the weight tensor and 4' is the quantization step size of the Ith layer; and c' is a coefficient depending on the layer, in that if = 1, ESN related to the number of parameters (ef. memory footprint), and if cf is the number of computations per parameter the /th layer, ESN is related to the number of computations (cf. FL.P).. f()) is an element-wise function that determines how the metric responds to the value of each weight parameter. We propose two functions for f().. The first is fW im) W based on the assumption that energy consumption increases with the square of the value for each weight parameter or for each computation. When c¹ the definition
ESN, Σ ml
This assumption is reasonable when we employ an analog (or in-memory) MAC computation en- gine (Shafiee et all, 2016: Miyashita et all, 2017), because energy consumption is proportional to the square of the signal amplitude when the signal represents an analog quantity such as voltage or cur- rent. Assuming ideal hardware, adopt definition where energy consumption varies according to the instant amplitude (ef. class-B amplifier), which is more energy efficient than the case where energy consumption is constant and the value is determined by the maximal amplitude (ef. class-A amplifier) (Sechi, (1766).
The second proposed function is /(Wit) = [og (abs (abs(W ((n)) + 1], where log, (.) and abs (.) are functions applied each element of tensor argument. This is based the assumption that energy consumption increases with the binary logarithm of the value for each weight parameter. When c¹ the definition is
ESN Σ (abis) 1711..
(3)
In a digital circuit, number is represented as binary digit (bits), so the energy consumption for moving or processing signals is roughly proportional to the number of bits, which is the binary logarithm of the value. therefore reasonable use Equation (C) for a digital circuit.
2.2 RELATION BETWEEN ESN AND S/N
The effective signal norm defined in Equation (2) is related to the signal-to-noise ratio (S/N) when quantization noise is dominant and noise approximated by uniform distribution (Gray
& Neuhoff, (199)).
W
W 10
| 12
ESN, Σ W
Σ
S/N:
W
12
where is the layer index, 5/M, the soniiallo-guanizziionnn-n ratio of the Ith layer as defined by
S/N, W - --WW I.. and ||W') is the number of nonzero elements in the tensor w' Appendix
W
presents the derivation of Equation (8)) This equation allows us calculate ESN,, long as S/N is defined.
2.3 ESN, DURING TRAINING
For example, we can define S/N by regarding perturbation of weight parameters during training as noise. Formally, we define the signal and noise at the jth epoch as
S, = ΣΣ lll N ΣΣ
W "int 1P:
(5)
where W j.j is the weight parameters the Ith layer at the ith iteration in the jth epoch, and W W a snapshot of the weight at the beginning of the jth epoch. Then, N, is the effective value of random walk noise for weight parameters in epoch.
4

---

Under review as a conference paper at ICLR 2020
-weight decay: 5e-4
-weight decay: 1e-3
--weight decay: 2e-3
992
100M
90
ESN,
10M
88
1M
86
84
100K
82
10K
80
40
80
120
160
60
80
100
120
140
160
Epoch
Epoch
Figure 2: Training curves of ESN, (left) and validation accuracy (right) on CIFAR-10. In the right graph, moving average curve between 3-epochs is overlapped on each plot.
VGG7, CIFAR-10
ResNet-20, CIFAR-10
DenseNet-BC 100-12, CIFAR-100
100
Val. Accuracy [%)]
90
Val. Accuracy [%6
80
Val. Accuracy
60
70
40
40
50
20
During training After quantization
30
During training Afferr quantization
20
During training After quantization
0
10
0
10K
100M
10M 100M ESN,
100K
1M
10M 100M
1G
ESN,
ESN,
Figure 3: Relation between validation accuracy and ESN, during training and after quantization. ESN, of the former is computed using Equation (5)) and that of the latter is computed using Equa- tion (2)
The left figure in Figure 0 shows the ESN, curve during training, which is calculated by Equa- tion (L)), and S/N as defined by Equation (5). In this experiment, we use ResNet-20 (He et all 2016) CIFAR-10 dataset (Krizhevsky, 2009). We use SGD with momentum 0.9, and we set the mini-batch size to 100 and the initial learning rate to 0.1, which is shifted by 1/10 at epochs and
120. We vary the weight decay factor from x 10 to 2 x 10-3, indicated by different colors in Figure
Interestingly, the ESN, curves look very similar the validation accuracy curves shown in the right figure in Figure 2 Both ESN, and validation accuracy steeply increase the point where the learning rate is decreased by 1/10, because decreasing the learning rate reduces perturbation (or random walk) of weight values during one epoch, increasing S/N. We can also see that as the weight decay factor increases, ESN, tends to decrease. This is also reasonable, because weight decay decreases 3 or the signal level, reducing S/N. More interestingly, even this trivial tendency appears correlated with validation accuracy: for example, after steep rise epoch 80, both ESN, and validation accuracy gradually decrease, converging to stable values after overshooting, There is similar relation between the amount of overshoot and weight decay factor, namely, we observed larger overshoot with smaller weight decay factor. These findings might be useful for developing a novel optimization algorithm, but we leave this for future work.
The blue plots in Figure E show ESN, versus validation accuracy. In this experiment, we use VGG7 (Simonyan Zisserman, 2014) and ResNet-20 on CIFAR-10, and DenseNet-BC: with / 100, & 12 (Huang all 201 CIFAR- 100. We employ SGD with momentum 0.9 and set the initial learning rate to 0.1, followed by cosine annealing without restart (Ilya Loshchrlov, 2017). The mini-batch size and weight decay factor respectively set to 125 and 5 x 10
The red curve shows the ESN. versus validation accuracy curve when applying quantization to weight parameters trained by the training process where the blue plots are obtained. As the figure shows, the red and blue curves seem be correlated. This suggests that the model acquires robust- ness quantization by having experienced similar perturbations due random walk during the
5

---

Under review as a conference paper at ICLR 2020
100
100
100
100
Eng
Accurmay
60
60
60
40
40
40
60
Val.
ResNet-14 14
VVC.P1 -VGG
Vall
-MennNak-34
-Re5NNc-220
VVCCELL
VVG 3x)
VVG770
Figure 4: Aceuracy versus (left and ESN, (right) for various network depths and widths
training process. If so, the ESN, versus validation accuracy curve during training is available as loose boundary on accuracy degradation due to decrease of ESN,, by quantization. We discuss this hypothesis in Section 22. again.
2.4 ESN VERSUS ACCURACY FOR VARIOUS MODEL SIZES
With our metric defined, we next attempt to investigate which better, a model that is slim (with the number of weight parameters and computations reduced by pruning) and mildly quantized (e.g., with 4-5 bits), or model that is fat (with large number of weight parameters and computations) and radically quantized (e.g., with 1-2 bits). We evaluate six network architectures with different depths and widths: ResNet-8, 14, 20 and VGG7 with different channel widths (1 x, 0.5x, 0.25x) on CIFAR-10 10 dataset. In this experiment, we employ networks with originally small number of parameters as slim models instead of pruning fat models. The network architecture of ResNet is based on the original one without bottleneck architecture (He all 2016). The detailed network architecture of VGG7 is shown in Appendix B
Figure shows validation accuracy versus ESN, and ESN,, Here, we consider differences ESN, and ESNa. For simplicity, we assume that all weight parameters have the same value >> 1, and that the number weight parameters is N. The definitions of ESN, and ESN, are expressed as follows,
(6) Therefore, when we compare ESN, and ESN,, both are linearly proportional to the number of weight parameters, whereas values of weight parameters affect the metric at squared and logarith- mic scales, respectively. This means that ESN, is more strongly dependent on the size the values than the number of weight parameters, and the opposite is true for FSSN In other words, ESNN, is more sensitive to pruning than to quantization, and ESN, is more sensitive to quantization than to pruning. Figure Al clearly shows these tendencies. We can see that even when using the ESN, metric, which is advantageous for quantization, the slimmer model shows better validation accuracy lower ESN regions. These experiments indicate that in lower ESN regions, or with lower energy consumption by ideal hardware, especially when employing digital computing, it is an advisable strategy to prune the model the limit where the desired accuracy is achieved, and then to apply quantization to obtain the highest possible accuracy. In contrast, it is bad strategy to apply quan- tization to fat model, even if it shows much better accuracy than the pruned slim model before quantization. Consequently, slim and mildly quantized model or a fat and radically quantized model better the answer the former one, which may in sense raise question trends in quantization research. Note that the above discussion also indicates that the importance of quantization will be relatively increased analog computing become practical in the future.
ESN, = 02 x N, ESN, = [loga v +1] x N.
the question posed at the beginning of this subsection wwether
2.5 TRAINING UNDER LOW-ESN CONDITIONS
We next observe how the weight parameters evolve in the training process. Figure El shows from left to right the loss, ESN,, and number of pruned filters (output channels). In this experiment, we train ResNet-20 on 1IFAR-10 dataset with initial weight parameters of trained model. We employ SGD with momentum 0.9, weight decay factor 5 x 10 4 and mini-batch size 200, updating for 20 epochs. The learning rate (LR) is constant, with value varied from 0.32 1.81 for each training. Note that we do not intentionally prune filters in this experiment; rather, filter pruning spontancously occurs along with sequential updates of weight parameters by SGD. The center graph shows that as

---

Under review as a conference paper at ICLR 2020
0.45 +LR: 0.32
pnuned eltorr
600
1K
Training
2.5
100
400
300
100
0.1
10 Epoch
15
20
5
10
15
20
5
10 Epoch
15
20
Epoch
Figure 5: Training curves for loss, ESN and number of pruned filters under different learning rate (LR) conditions. E is calculated by Equations (2!) and (S))
E
Pruned Quantized Mixed (Uurs)
85
Accume'
85
400
80
80
300
70
75
Pruned
Quantized
75
70
-Quantized
Pruned
200
Mixed (Ours)
Mixed (Ours)
100M
6DM
ESN
prtciisoooon
Figure 6: Accuracy versus ESN./4 a/d (left/center) and histogram of the maximal bit width for each filter (right) in the pruning only, quantization only, and come-tten-qqantize mixed cases.
the learning rate increases, ESN, decreases due to the larger noise. The number of pruned filters after epoch 20 is larger with the larger learning rate, which corresponds to smaller FSNN, In the training process, rather than continuing increase constant rate throughout the training process, the number of pruned filters steeply increases within a few epochs and then is maintained without noticeable change.
This indicates that under small ESN, condition controlled by the learning rate, the optimizer finds solution for reducing the number of parameters by pruning filters and prioritizing increases in am- plitude of the remaining parameters, rather than letting all parameters survive with small amplitude.
2.6 PRUNE then QUANTIZE
In the previous two subsections, we empirically showed that achieving optimal accuracy in a low ESN region requires preparation of slim model before applying quantization. A slim model can be generated by pruning a fat model or by designing a slim architecture. Based on the lottery ticket hypothesis (Frankle & Carbin, (OT), where fat overparameterized network can have higher potential for finding better optimal parameters by SGD training, we adopt the former method, namely, pruning weight parameters from pretrained fat model. We apply ADMM regularization for structured pruning (Wang et al., (4L9H), since this achieves sate-of-the-aa art performance.
In this experiment, we evaluate ResNet-20 on CIFAR-I0 dataset. We update pretrained weight parameters through the SGD algorithm with alternating direction method of multipliers (ADMM) regularization (Wang et all 20196) for few additional epochs. We then fine-tune the resulting pruned model over 160 epochs. In this fine tuning, we employ SGD with momentum 0.9 and set the mini-batch size to 100 and the initial learning rate to 0.1, followed by cosine annealing without restart. The weight decay factor is set to 5 x 10-2. Since the weight parameters are not yet quantized in this phase, we update all parameters, except that we set weights for the filters (output channels) and channels (input channels) pruned in the previous phase zero. Finally, apply quantization.
As Figure E shows, the Pareto frontier significantly improves by applying prune-then-quantize (red), as compared to applying extreme quantization (green). The right graph in Figure E shows his- togram of the maximal bit width under each filter (output channel) of the three models indicated by the broken line in the center graph (ESN, 39M(bits).. Since the pretrained model has 800 filters before pruning and quantization, summations of frequencies for each color are 800. In the pruning only (blue) case, the weight parameters in 555 filters, the largest number among the three methods,
7

---

Under review as a conference paper at ICLR 2020
Training
Loss
Quantization
D Quantratinn Nois 28 2h1
Random walk BIR-C.0 0.01
Quanizze
W: 32bit
W
Figure 7: Relation between random walk during SGD and quantization noise. Loss often steeply decreases when LR is shifted by (e.g., 1/10). In this situation, loss before shifting LR is possibly governed by random walk noise due to the large LR, shown in the left figure. When quantize weight parameters of the trained model, however, the loss or accuracy should be determined by noise induced by quantization, the two cases should not diverge too much, the loss (or accuracy) after quantization is possibly predicted by the dependency of loss (or accuracy) on ESN, during training.
shown in the right figure. If we can assume that the loss landscapes of
are pruned (represented as 0 bits). Most of remaining filters are bits. In the quantization only (green) case, the number of pruned filters is low 77, while the bit width of most remaining filters is bit. In the proposed prune-ther thenn-quattize mixed (red) case, the number of pruned filters is 288 and the bit widths of most remaining filters are 3 or 4 bits, so the remaining filters are mildly quantized. As this case shows, while the ESN, values are similar, the validation accuracies differ depending on the number and bit width of the weight parameters. The prune-then-quantize mixed method prop- erly produces a pruned and mildly quantized model, which achieves far better validation accuracy. especially in the low ESN region, as compared the model produced by extremely quantizing fat model without pruning.
2.7 ESN, FOR QUANTIZATION
As Section PK3 showed, there is a possible quantitative relation between ESN, during model training and ESN, of the pruned and quantized version of that model after training. This finding inspires us to exploit ESN, information obtained during training to determine the quantization policy, for example, how many bits to allocate for each layer. Figure / shows the intuition behind this idea, namely straightforward use of the ESN, values for each layer at a certain epoch during training as targets for quantization. Appendix B describes a preliminary experiment for verifying this idea.
EXPERIMENT ON IMAGENET
To verify the validity of our claims, performed experiments under post-training quantization scenario (Banner et all, 2013) on ImageNet dataset (Deng et all, 2009). Post- Post-training quantization assumes use-case scenario where user (e.g.. at edge side) quantizes a given pretrained CNN model with limited number of unlabeled training samples for obtaining a lightweight model suited to a particular situation. As concluded in Section 2.4 in low ESN region or with low energy consumption, we can achieve higher accuracy by quantizing slim model than by quantizing fat model. even if the latter more accurate, To provide optimal accuracy in a low computational- cost region, we apply the proposed prune-then-quantee method to the post-training quantization scenario.
Figure shows the result of applying our method to ResNet- 18 and ResNet-50 on ImageNet dataset. mb We used pretrained mo as basic models with 1 channel width, and prune the models to various channel widths (0.25x, 0.5x, and 0.75x). In this experiment, we determine which filters (output channels) to prune based on the signal norm of weights calculated by Equation (5). We prune filters in order from the smallest signal norm, and tune each model for the pruned structure through the SGD algorithm over epochs. The mini-batch size is set to 64, and the learning rate is initialized
8-5c 1-5cc16 06cce,,th,
8

---

Under review as a conference paper at ICLR 2020
(0 (0255)-Dumm -urs (o5x)==uum (8.75x) +Ous (ix] HiBamee et (2618)
ReaMeeb-18 18
Resllet-18
Reshieh-1
60
60
15
50
50
50
10M
500M
106
ESS..
ESS
Rea Neee 50
Renilee-S3
Res
70
70
70
65
Top
134
30M4
100M
16
ESR.
0.5%
ESK:
Figure 8: (Left/center) top-1 accuracy on ImageNet versus ESN and (right) estimated ESN', applying the mune-then-quantize method to (upper) ResNet-18 and (lower) ResNet-50.
to 0.1 and divided by 10 at epochs 2 and 3. We then quantize each model. For quantization, we only fine-tune statistic parameters in BatchNorm layers (called running ing mean and running_ var in pytorch (Paszke et all 2017)) with no labeled data (Sasakı et all, 2019), which is the same condition as conventional work (Banner all 2018).
In addition to evaluations by ESN, and ESN,, to compare our method with the conventional work, we estimate its computational cost using Equation (II) with of and f(-) properly defined, as
ESN' ΣΣ M (loe((max(abs(W..)) + 1],
where MAC,. is the number of computation per filter and W int is the mth filter of W int. By applying the max(.) function, we restrict the number of bits to be same in each filter.
We apply this m to recently proposed efficient MAC array architecture capable of handling variable bit precision (Makretal) 2018).
As Figure & shows, significantly using slim models after pruning some filters improves the Pareto frontier as compared to Banner et al (20T).. For example, in ResNet-18, 57 % top-1 accuracy is achieved at 2 lower computational cost, and in ResNet-50, 63 % top-1 accuracy is at 4x lower computational cost. These results suggest that we should adopt a slim model to achieve best accuracy in low computational cost regions, instead of quantizing weight parameters to extremely low bit precision.
CONCLUSION
We proposed a hardware-agnostir metric called the effective signal norm (ESN) to measure compu- tational costs. Using this metric. demonstrated that a slim model with fewer weight parameters achieves better Pareto frontier performance in low computational cost regions than does an extremely quantized fat model. We also showed possible quantitative relation between weight perturbation during SGD training and quantization noise or robustness against quantization.
By defining this metric, on the hardware architecture side we can aim at realizing hardware whose energy consumption is proportional to the metric. On the algorithmic side, we can reduce the metric. We therefore expect consensus-sharing regarding the metric for computational cost to accelerate research progress for both algorithms and hardware architectures.
In Maki et all (2018), this metric is defined "MACxbit", which is essentially same ESN.

---

Under review as a conference paper at ICLR 2020
5 RELATED WORKS
Quantization Courbariaux & Bengio (2016); Rastegan al. (2016); Zhn et all (2017) quantize weight parameters and activations as or 2. Since such extreme quantization deteriorates accuracy to a nonnegligible extent, quantization methods with, for example, 4-8 bits are also actively ex- plored (Banner et all 2018; Lin et all, 2017) to avoid the accuracy drop. MMyashita et all (2016) attempts quantize weight parameters and activations a logarithmic domain, aiming not only at reducing information loss but also replacing multiplication with bit-shift to simplify computation. To reduce the average bit width as much as possible while maintaining accuracy, and to exploit it to ac-
celerate inference speed, Maki et all (2018) proposes variable bit-width quantization co-optimized with hardware architecture. Although nonuniform quantization (Tung & Mori, 2013; Han et all, 2016) and vector product quantization (Goog et all 2014: Jegou all, 2011 Stock_et.al 2019)
are also actively studied, these approaches are effective for reducing the memory footprint but not for directly reducing computational costs. Use cases include quantization-awen
training (Courbariaux & Bengio, 2016: Rastegari et ard 2016; Zhu et all 2017: Lin et all 201 Zhang et arj 2013) and post-training quantization (Banner all 2018). Quantization-awe
training achieves better accu- racy with lower bit widths, and is almost inevitable for those with extreme quantization. However, it tends to incur higher training costs in most cases. The post-training quantization scenario, which is executable with less computational resources and smaller training dataset, has potentially more applications, so we also target this scenario.
Pruning Pruning methods include structured pruning, which prunes whole layers, filters, or chan- nels to maintain regular structure so that computations are easily parallelized, and unstructured pruning. which randomly prunes individual weight parameters. Since it is difficult to benefit from unstructured pruning unless sparsity is sufficiently large, due to its incompatibility with paralleliza- tion (Lu et all 201), we target structured pruning. Many pruning methods have also been proposed, such as criteria- based approaches (1 CLeCun et all (190), regularization- based approaches (Han et all, 20L5: Wang et all 2019h) and methods employing reinforcement learning (Zhong all, 2018). In this paper, we argue that a pruned slim model performs better after quantization in lower computa- tional cost regions. Any pruning method be applied to produce a pruned slim model.
Metric for measuring cost In most quantization papers, the standard metric is accuracy achieved under a certain bit width. For pruning, the standard metric the number of parameters or the number of computations (FLOP). Some papers use inference time when the model runs on specific hardware (Cai all 2019; Yang et all, 2018), but this metric strongly hardwaree-ependdet. In this paper, we propose hardware-agnoste metric.
Noise during training Relations between learning rate, batch size, and noise during training are widely discussed (Keskar all, 2017; Xing et all, 2018). To make the model more robust to quantiza- tion, some papers propose intentional addition of noise to gradient or weight parameters (Spallanzani ata 201% Baskin et all 20167.5)
REFERENCES
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. ACIQ: analytical clipping for integer quantization of neural networks. CoRR, abs/1810.05723, 2018. URL http::/arxiv..ot
0os/1810.0572
Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Giryes, Avi Mendelson, and Alexander M. Bronstein. NICE: noise injection and clamping estimation for neural network quantization. CORR, abs/18 100.0000,,,,,, 2018a. URL http:://rxxxt
abs/1810.00 001621
Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes. Alexander M. Bron- stein, and Avi Mendelson. UNIQ: uniform noise injection for the quantization of neural networks. CoRR, abs/1804. 10969, 2018b. URL
10996//00110111.9..... 10969.
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for efficient deployment. CoRR, abs/1908 1908.09791.11.. 2019. RRI.ttt::/ //ax//aaxiiaa wwt.g/o.../g///o. 908 09791.
10

---

Under review as a conference paper at ICLR 2020
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. URL htto://arxiy:
org/ htw/abb/aa 1602. 328
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, June 2009, doi: 10.1 (.L109/CCVP.2200955000
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:/7
Spenreview. hot.:eneeeetttommm/oeeee--33----
A. Gersho. Quantization, IEEE Communications Society Magazine, 15(5):16-16, Sep. 1977. doi: (101////MCOMM.117..99999999
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. Compressing deep convolutional networks using vector quantization. CoRR, abs/1412.6115, 2014. URI. fttp:t////axxiv.orr
abs 11112211.615
R. M. Gray and D. L. Neuhoff. Quantization. IEEE Transactions Information Theory, 44(6): 2325-2388 Oct 1998. doi: 0.1100911.720544
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnet (eds.), Advances in Neural Information Processing Systems 28, pp. 1135-1143, Curran Asso- ciates, Inc., 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
K. He, Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10. 10.1109/CVPR.2016..0
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, has/1704.04861.. 2017. URL Atto://arxiv.orot
7W4 04861
G. Huang, Z. Liu, L. d. Maaten, and K. Q. Weinberger. Densely connected convolutional net- works. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- July 2017. doi: 10.1109/CVPR.2011..
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size. CoRR, abs/1602.07360, 2016. URL http::/ wttp/////////0///000666
Frank Hutter Ilya Loshchilov. SGDR: stochastic gradient descent with warm restarts. International Conference on Learning Representations (ICLR), 2017.
H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117-128, Jan 2011. ISSN 0162-8828. doi: 10.1 10.1109/PAML.2000.57
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On latch-batch training for deep learning: Generalization gap and sharp minima. Inter- national Conference Learning Representations (ICLR), 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky (ed.), Advances in Neural Information Processing Systems 2, pp. 598-605 Morgan-Kaufmamn Kaufmann, 1990. nis. wtwps.cccc/ee httppppp.ccpppp///5000a000
URL http: //ppppapp/e/eeee/eeee
-bra n- n-damage. .pdf

---

Under review as a conference paper at ICLR 2020
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp. 345-353. Curran Associates, Inc., 2017.
L. Lu, J. Xie, R. Huang, J. Zhang, W. Lin, and Y. Liang. An efficient hardware accelerator for sparse convolutional neural networks on fpgas. In 2019 IEEE 27th Annual International Symposium on Field-Prog regrambrogee Custom Computing Machines (FCCM), pp. 17-25, April 2019. doi: (A1109/FCCM.2211.00000
A. Maki, D. Miyashita, K. Nakata, F. Tachibana, T. Suzuki, and Deguchi. Fpga-based cnn proces- with filtr--wie-optmmized bit precision. In 2018 IEEE Asian Solid-Stale Circuits Conference (A (A-SSCC), pp. 47-50, Nov 2018. doi: 10.1109/ASCC..211... 851779994......79933.......
D. Miyashita, S. Kousai, T. Suzuki, and J. Deguchi. A neuromorphic chip optimized for deep learning and cmos technology with time-domain analog and digital mixed-signal processing. IEEE Journal of Solid-State Circuits, 52(10))2679-2666 Oct 2017. ISSN 0018-9200. doi: 10.11099/SS..2211.2221111
Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. CoRR, abs/1603.01025, 2016. URL httt::/arxiv..oo
008//603.000
Travis Oliphant. NumPy: A guide to NumPy. USA: Trelgol Publishing, 2006.
URL http:
7 www.. aumpy.ora/
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
S. Sasaki, A. Maki, D. Miyashita, and J. Deguchi. Post training weight compression with distribution-based filter-wise quantization step. In 2019 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), pp. 1-3, April 2019, doi: 101100090co0Chhpp:2099887711311
F. N. Sechi. Lincarised class-b transistor amplifiers. IEEE Journal of Solid-State Circuits, 11(2): 264-270, April 1976. ISSN 0018-9200. doi: 10.1109/1SSC.197 10507 13.
A. Shafice, A. Nag. N. Muralimanohar, R. Balasubramonian. J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar. Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars. In 2016 ACM//EEE 43rd Annual International Symposium on Computer Architecture doi: 10. 201611..
(ISCA), pp. 14 24, June 2016.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. CORR, 0bss/1409.15555 2014.
Matteo Spallanzani, Lukas Cavigelli, Gian Paolo Leonardi, Marko Bertogna, and Luca Benini. Additive noise anncaling and approximation properties of quantized neural networks. CoRR, abs/1905. 10452, 2019. URL
abs/19/5. 10452.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. CoRR, abs//990,,0566, 2019. URL
ht.t:://arxvv.ot www.aawwww.orr//bb.oaoaaaa/a wog./bogga...................... http:///bbb/990008811
F. Tung and G. Mori. Clip-q: Deep network compression learning by in-parallel pruning- quantization. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7873-7882, June 2018. doi: 10. 20W.CVVR..018.008888
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware- automated quan- tization with mixed precision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019a.
12

---

Under review as a conference paper at ICLR 2020
Yanzhi Wang, Shaokai Ye, Zhezhi He, Xiaolong Ma, Linfeng Zhang, Sheng Lin, Geng Yuan, Sia Huat Tan, Zhengang Li, Deliang Fan, Xuchai Qian, Xue Lin, and Kaisheng Ma. Non- structured DNN weight pruning considered harmful. CoRR, abs/1907.02124, 2019b. URL
http:://rxiis wwwpxiiiorr//a.... http///a////..0011111111
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. CoRR, abs/1802.08877, 2018. URL bttp://azxiv.org/aaae 08770
Tien-Ju Yang. Andrew Howard, Bo Chen, Xiao Zhang. Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 209,304, Cham, 2018. Springer International Publishing. ISBN 978-3-030- 01249-6.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference on Computer Vision (ECCV), 2018.
X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
June 2018. doi: 10. 2.1.109//VR.221..0000
Jing Zhong, Guiguang Ding, Yuchen Guo, Jungong Han, and Bin Wang. Where prune: Using Istm to guide and-to-end pruning. In Proceedings the the Twenty-Seventh International Joint Conference
on Artificial Intelligence, IJCAI-18, pp. 3205-3211. International Joint Conferences Artificial Intelligence Organization, 7 2018. doi: 10.24963/jcca..2018888 URL nttps://coi. ora/
10.24963/11ca1 2018
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. Interna- lional Conference on Learning Representations (ICLR), 2017.
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In 2018 IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8697-8710 June 2018. doi: 10.1 1119/C/VR.2018.0000000
13

---

Under review as a conference paper at ICLR 2020
A DETAILS OF THE QUANTIZATION PROCESS
We basically apply midrise-type quantization (Gersho, 1977), but modify it as follows. In the quantization phase, we prune filters in which all weight parameters are within 4/22, as described in the following pseudocode with Numpy (Olphant (006) notation.
Quantize
W: (filter (out_ch), channel (in_ch), ky, kx)
pruned numpy. aalp all (abs (W) < delta/2, axis= (1, 21, ))) Wiit = numpy. floor (W/delta) 0.5
Wiint [pruned, :, :, :] =
B ESN, FOR OPTIMAL QUANTIZATION POLICY
We attempt to exploit the ESN, obtained during training to determine a quantization policy using VGG7 where numbers of filters in all convolutional layers are reduced 0.25x on CIFAR- 10. The left figure in Figure g show the validation accuracy versus ESN, during training and after quantization, As an initial policy, we apply the same bit width for all layers. The results shown in gray do not fit the blue plot during training, possibly indicating the policy is suboptimal. We next investigate the ESN, of each layer and find that the misfit caused by the sixth layer, as shown in the right figure. When we consider validation accuracy versus the sixth-layer ESN,, shown in green symbols, validation accuracy after quantization deteriorates at a much higher ESN, than during training. An interpretation is that since this deterioration is caused by other layers (e.g.. the first) rather than the sixth layer, weight parameters in the sixth layer should be more aggressively quantized. Based this observation, we modify the quantization policy such that the bit width of the sixth layer become smaller than that in other layers, thereby improving performance as shown by the red plot in the left figure. This preliminary experimental result suggests we can use ESN, information during training to find an optimal quantization policy.
C RESNET-18 VERSUS RESNET-50
The proposed metric allows us to compare different network architectures. In Figure I00. we com- pare the Pareto frontier of validation accuracy versus ESN, curves for ResNet-18 and ResNet-50. The plots in Figure [T] extracted from Figure & This result shows that ResNet-50 has better Pareto frontier than does ResNet-18. One convincing reason is that ResNet-50 employs a parameter- efficient bottleneck structure, whereas ResNet-18 18 does not. In contrast, ResNet-18 shows better performance for ESN, < 2G, partly due to the higher sensitivity of ESN to the number of parame- ters. This suggests use of ResNet-18 when employing digital computing this ESN, region. Such comparisons enabled by the proposed metric are useful when searching for or developing efficient structures.
100
95
Val. Accuracy [%6]
Accuracy [%
90
80
85
60
80
40
During training
75
layer during training 6th layer during training layer after quantization
70
20
After quantization (opt.)
65
After quantization (init.)
+ 6th layer after quantization
60
10K 100K 1M 10M 100M 1G
1K
100K 10M
ESN,
ESN, of each layer
Figure 9: Relation between validation accuracy versus ESN, during training and after quantization.
14

---

Under review as a conference paper at ICLR 2020
Re (0.25x) *Res50 (0.5x)
<0Re550 (0.75x) c-Re550 (1x)
Res 18 (0.25x)
Res18 (0.5x)
R8 (0.75x) C-Re518 (1x)
Top Top-1Accurace Accuracy [%]]
Top-1 T Accuracy [G
70
70
60
60
50
50
1M
10M
100M ESN,
500M
ESN,
Figure 10: Comparison ResNet-18 18 and ResNet-50 with respect to top-1 accuracy on ImageNet versus ESN, a/d (left/right).
(0.25x)
(0.5x)
(0.75x)
(1x)
Zhang (2018)
*Lin (2017)
Wang (2018)
ResNet-18
ResNet-50
70
Top- Accurrcyy Pk]
60
55
16
10G
2G
20G
ESN;
ESN(
Figure 1I: Comparison of other methods with respect to top-1 accuracy on ImageNet versus esti-
mated ESN'
D COMPARISON WITH OTHER METHODS
In Figure I superimpose results from other works Figure N. Linetal (2077) and Zhang al (2018) show better accuracy in smaller ESN', regions than ours, but these apply quantization-aa aware training, whereas result is obtained by post-training quantization. Although expect that the findings presented this paper will also improve performance of quantization-aware training, we leave explicit demonstration of this future works.
E DERIVATION OF EQUATION
The mean squared noise energy N due to quantization over the all elements in tensor W is com- puted as
N W A Wint [
(8)
where Wint W/4 + 4 is the quantization step size and ||]] Wo is the number of nonzero el- ements W. When quantization noise is approximated by a uniform distribution, the mean squared noise energy (Gray & Neuhofi, (99%)
4/2 1
2
N
n2
2d
(9)
A"
12
15

---

Under review as a conference paper at ICLR 2020
From Equations H and (9).
12
1
|W 4 Wint
(10)
[
Then,
W
[W] 2
Wil
1|
||/
(11)
W
A
Wint
12
Here we again assume that quantization noise is uniformly distributed. In this equation, since ||]]] and ||W A W are respectively the signal and noise norm, the term to be S/N, and thus,
||
considered
I|W [W-A-WWmt wm
||
inc
S/N-
(12)
I
A mml
12
12
We obtain Equation (+) by summing this value over all layers.
F DETAILS OF NETWORK ARCHITECTURES
The network architecures of VGG7 (1x, 0.5x, 0.25x) which we use in section are shown in table M1 These architectures are based on the original model of VGG (Simonyan & Zisserman 2014)
Table I: Detailed network architecture of VGG7 for CIFAR-10. Cf and Co represent the number of input and output channels, respectively in conv-layer. fc-layer, they represent the number of input and output neurons
VGG
(1x)
VGG (0.5x)
VGG (0.25x)
Layer name
kernel size
padding
stride
C₁
Co
C₁
Co
C₁
Co
convl
3
128
3
64
3
32
3 x 3
conv2
128
128
64
64
32
32
3 x 3
1
maxpool
128
128
64
64
32
x
0
conv3
256
256
128
128
64
64
3 x
1
conv4
256
256
128
128
64
64
3 x 3
1
maxpool
256
256
128
128
64
x 2
0
conv5
512
512
256
256
128
128
3 x
conv6
512
512
256
256
128
128
3 x 3
maxpool
512
512
256
256
128
128
x
0
fc
8192
10
4096
10
2048
10
16


Reviews:
[{"review_id": "rylzNrlaYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new metric to evaluate both the amount of pruning and quantization. This metric is agnostic to the hardware architecture and is simply obtained by computing the Frobenius norm of some point-wise transformation of the quantized weights. They first show empirically that this Evaluation metric is correlated with the validation accuracy. Then use this metric to provide some general rules for pruning/quantizing to preserve the highest validation accuracy. Finally, they derive a strategy to perform pruning by monitory the signal to noise ratio during training and show experimentally that such method performs better than competing ones.\n\nPros: - Extensive experiments were performed to test the methods, and the results seem promising.\n\nCons: - The paper is not very clear, and the structure is somehow confusing.\n- It is not easy at first to understand the experimental setup and requires to make a lot of guesses in my opinion. \n- The paper didn't motivate properly the use of a hardware-agnostic metric in the context of the quantization and pruning. Isn't the ultimate goal of pruning/quantization is to optimize the run time/energy consumption of the specific device with the least compromise on the accuracy?\n\nI feel that the paper currently jumps between very different ideas: \n\t- Evolution of the proposed metric during training: 2.3 and 2.5. While the 2.3, the take-home message is relatively clear:  the ESN is correlated with the validation accuracy, I don't fully get the point of section 2.5: It suggests that the optimizer does some sort of pruning just by choosing a higher learning rate.   \n\t- Finding an optimal strategy for pruning/quantizing a network: 2.4 and 2.6. Those two sections are relatively clear, although I have some questions about the experiments.\n\t- Developing a new strategy based on the proposed metric to quantize and prune a network in a Pareto optimal sense: This is briefly and not very well explained in section 2.7, which sends back to 2.3, but it is hard to understand how it is exactly done. It seems that section 3 provides some empirical evidence supporting this strategy, but the description of the method is hidden in the experimental details.\n\n\nSome questions: \n- In figure 3, the blue dots represent validation vs ESN at each training iteration? What about the red plot, is it obtained by quantization of the parameters at different stages of training, or is it using the final parameters? Which equation was used to compute the red curve (2) or (4)?  How much quantization was performed? If the quantization was chosen to match the level of noise then it seems natural to expect such behavior in figure 3.  \n\n- In figure 4, how much pruning was performed for each network and was it the same quantization? In other words how each point in the plot was obtained?  The authors come to the conclusion that one should 'prune until the limit of the desired accuracy and then quantize', but it is hard for me to reach the same conclusions as I don't see the separate effect of pruning and quantization in those figures. Or maybe pruning is implicitly done by choosing a small network? In this case, it makes more sense, but still, some clarifications are needed.\n\n- Which equation for the ESN was used to produce figure 5? Equation (2) or (4)? \n\n- What is the Pareto frontier? I think it is worth first introducing this concept and describing more precisely how those curves are obtained. For someone who is not very familiar with these concepts, which is my case, it makes the reading very hard. \n\n- How was the number of pruned filters computed in figure 5 (right)? I don't expect the solutions to be sparse during training, especially that no sparsity constraint was imposed, or was it?\n\t\n\n\n-------------------------------------------------------------------------------\nRevision:\n\nThank you for all the clarifications and the effort to make provide a clearer version of the paper. \n\nRegarding section 2.7: ESNa FOR QUANTIZATION: Would it make sense to include the paragraph 2.7 at the end of 2.3, since it related to it and doesn't seem to require any of the intermediate subsections.\n\nresponse to Comment 3: Unfortunately, I'm not convinced by the explanation about the effect of the lr on sparsity. The decay coefficient controls the saparsity indeed, but not the lr. That is because unlike the lr, the decay coefficient defines the cost functions to be optimized:  L+dc ||W||^2, while the lr corresponds simply to the discretization of some gradient flow.   For instance, in a deterministic and convex setting, the solution that is obtained would be the same, regardless of the chosen lr  ( provided the lr is smal enough so that the algorithm converges) see for instance [1].  In a non-convex and stochastic setting do the authors have a particular reference in mind? I'm not aware of such behavior. I would expect a similar sparsity if dc is kept fixed and only lr changed. Is it likely that with smaller lr, the algorithm just didn't have time to converge? This would explain why the obtained solutions were less sparse.\nresponse to answer 5:  it is indeed well known that L1 norm induces sparsity, however l2 doesn't, it just encourages the weights to be smaller. In the optimization litterature sparsity of x% means x% of the parameters  are exactly 0. This is achieved with l1 norm, however l2 norm would only enforce that the coefficients are small but not necessarily 0 (see [1])\n[1]:  ROBERT TIBSHIRANI, Regression Shrinkage and Selection via the Lasso.\n\nAlthough the paper improved in terms of clarifications and experimental details, I still think it will benefit from additional work on careful interpretation of the results.\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rylETCDaFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an effective signal norm metric that measures the cost of the neural networks under both compute(ESN_a) or memory(ESN_d) in an ideal hardware setting. The authors then show that the slimmer models with fewer parameters are better than fatter models.\n\nStrength\n\nFirst of all, I like the discussion about different hardware(CPU) setups and corresponding cost. I also like the usage of pareto frontier to compare experimental methods. The proposed metric was simple but the author justifies it with respect to possible hardware setups in an ideal setting.\n\nWeakness\n\nThe main drawback of the paper is the lack of novelty in proposed method. The metric itself appeared to be the main contribution, but as the author said, the metric was based on an ideal hardware setup that ignores the memory hierarchy and data transfer cost, which could be the bottleneck in reality.\n\nMore importantly, it is hard to use the empirical evaluation to justify the conclusion “fatter models are worse”. As we know that extremely low bit quantized models are very hard to train, and it is not surprising -- by setting the number of bits to extremely low bits, the models cannot recover in a few rounds of re-training. \n\nThis being said, I cannot think of a better alternative. While I do not think the experimental results offers new insights(other than it is hard to re-train lower bits models). How to quickly train low bits models remains an open question.\n\nFinally given that there are already quite a few methods that prunes model based on real hardware evaluations, it would be great to compare to these methods as well.\n\nBecause the above weakness of the paper, I think this paper should not be accepted to the program.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lnx65qqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a hardware-agnostic metric called effective signal norm (ESN) to measure the computational cost of convolutional neural networks. This metric aims to fairly measure the effects of pruning and quantization. What’s more, based on the metric, the authors demonstrate that models with fewer parameters achieve far better accuracy after quantization. A large number of experiments are carried out to prove the effects of the metric and related conclusions, however, several experiments and arguments are confusing. \n\nPlease see my detailed comments below.\n\nPositive Points:\n1. The authors propose a hardware-agnostic metric named effective signal norm (ESN), aiming to measure the computational cost of the pruned and quantized models.\n\n2. Based on the proposed metric, the authors conclude that a moderately quantized slim model with fewer weight parameters achieves better performance rather than an extremely quantized fat model with huge number of parameters.\n\n3. This paper presents many interesting assumptions and possibilities, which can be further researched and explored in the future.\n\n\nNegative Points:\n1. The authors argue that ESN is a hardware-agnostic metric. However, ESN is based on ideal hardware, where the energy consumption is linearly proportional to the number of non-zero weight parameters and monotonically depends on the bit-width of weight parameters. Therefore, ESN is not suitable for existing hardware.\n\n2. Both of ESN_a and ESN_d are based on the assumptions instead of the fact, which is not convincing. What’s more, the assumptions are hard to be proved.\n\n3. The experiments are not strong enough to support the author's conclusions. For example, the authors argue that “models with fewer parameters achieve far better accuracy in low computational cost region after quantization”. However, this conclusion is only based on the ESN metric. Since ESN is based on some assumptions, this conclusion is not convincing.\n\n4. Please give more details about the quantitative relation between the quantization noise and the perturbation of weight parameters during SGD training.\n\n5. In terms of Figure 2, the author's description is not objective. The similarity between the ESN_a curves and the validation accuracy curves is not very high. Besides, the authors mention that “after steep rise at 80 epoch, both ESN_a and validation accuracy decrease gradually”. However, the performance degradation in Figure 2 is not obvious. Therefore, the conclusions drew by the observation are also unreliable.\n\n6. The ESN can not make an accurate evaluation of a model. The trade-off between accuracy, speed, model size is often required in model compression. However, the ESN can not make the trade-off between them. Moreover, the ESN does not have a clear application scenario.\n\nMinor issues:\n1.\tThere are lots of spelling and grammar mistakes in the paper, such as “an ideal hardware ”, “a extremely quantized fat model” and so on.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a hardware-agnostic metric called effective signal norm (ESN) to measure the computational cost of convolutional neural networks. They then demonstrate that models with fewer parameters achieve far better accuracy after quantization. The main novelty is on the metric ESN. However, ESN is based on ideal hardware, and thus not suitable for existing hardware. Assumptions made in the paper are hard to be proved. Experimental results are not convincing, and related pruning methods are not compared. Finally, the paper is not written clearly, and the structure and some arguments are confusing.
Decision: Reject