On the Pareto Efficiency of Quantized CNN
Authors: Ting-Wu Chin
Pierce I-Jen Chuang
Vikas Chandra
Diana Marculescu
Venue:  (2020)
Keywords: convolutional neural networks quantization
model compression
efficient neural network

Abstract: Weight Quantization for deep convolutional neural networks (CNNs) has shown promising results in compressing and accelerating CNN-powered applications such as semantic segmentation, gesture recognition, and scene understanding. Prior art has shown that different datasets, tasks, and network architectures admit different iso-accurate precision values, which increase the complexity of efficient quantized neural network implementations from both hardware and software perspectives. In this work, we show that when the number of channels is allowed to vary in an iso-model size scenario, lower precision values Pareto dominate higher precision ones (in accuracy vs. model size) for networks with standard convolutions. Relying on comprehensive empirical analyses, we find that the Pareto optimal precision value of a convolution layer depends on the number of input channels per output filters and provide theoretical insights for it. To this end, we develop a simple algorithm to select the precision values for CNNs that outperforms corresponding 8-bit quantized networks by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.   

Under review as a conference paper at ICLR 2020
ON THE PARETO EFFICIENCY OF QUANTIZED CNNS
Anonymous authors Paper under double-bling review
ABSTRACT
Weight quantization for deep convolutional neural networks (CNNs) has shown promising results in compressing and accelerating CNN-powered applications such as semantic segmentation, gesture recognition, and scene understanding. Prior art has shown that different datasets, tasks, and network architectures ad- mit different iso-accurate precision values, which increase the complexity of ef- ficient quantized neural network implementations from both hardware and soft- ware perspectives. In this work, we show that when the number of channels is allowed vary such that networks of different precision values have the same model size, lower precision values outperform higher precision ones in a Pareto sense (accuracy vs. model size) for networks with standard convolutions. Relying on comprehensive empirical analyses, we find that the optimal precision value of a convolution layer depends on the number of input channels per output filters and provide theoretical insights for it. To this end, we develop simple algorithm to select the precision values for CNNs that outperforms corresponding 8-bit quan- tized networks by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.
INTRODUCTION
Recent success of convolutional neural networks (CNNs) in computer vision applications such as image classification and semantic segmentation, have fueled many important applications in energy- constrained devices, e.g., virtual reality headsets, drones, and robots. As a result, improving the energy-efficiency of CNNs while maintaining their attractive features (e.g., accuracy for task) has gained tremendous research momentum in recent years.
Among the efforts of improving CNNs' efficiency, weight quantization was shown to be an effective technique (Zhou et al. (2016) 2017): Hou & Kwok (2018) Ding al. (2019)). The majority of research efforts in quantization have been devoted to develop better quantization algorithms such that lso-figure-of-merr (i.e., accuracy) is achieved with lowest possible weight precision value. Nevertheless, the iso-accurate precision value depends on the dataset, task, and network architecture of interest, which greatly increases the neural network implementation complexity from both hard- ware and software perspectives. For example, hardware and software implementations optimized for executing an 8 bit network are sub-optimal for executing bit network, and vice versa. The design optimization complexity further increases for recently proposed mixed- mixed-precision networks (Wang ctt al. 2019); Wu et al. (2018)) 8b); Dong et al. (2019)).
The key observation we have is that most prior literature in this space studies quantization for fixed network architectures, which is reasonable for evaluating the effectiveness of quantization algo- rithms, but unnecessary when considering the Pareto efficiency (accuracy vs. model size) of neural networks. In this work, we relax the restriction of fixing the network architecture and allow the number of channels of the CNN under consideration to vary. More concretely, multiplied (Howard et al. 2017)) as a tool to compare the performance different weight precision values under the same model size.
the width-
Overall, we systematically analyze the model size and accuracy trade-offs considering both weight precision values and the number of channels for various modern networks architectures (variants of
WWidth-multiphe grows or shrinks the number of channels across the layers with identical proportion for certain network, e.g., grow the numher of channels for all the layers by 2x.

---

Under review as a conference paper at ICLR 2020
ResNet, VGG, and MobileNet) and datasets (CIFAR and ImageNet) and have the following non- trivial and novel contributions:
We are the first to empirically show that when considering channel counts, lower precision weight values outperform higher precision weight values in a Pareto sense (accuracy vs. model size) for networks with standard convolutions. This is intriguing since it implies that scaling up (in terms of model size) along the channel count dimension is more effective for accuracy than the precision value dimension.
We are the first to show that the fan-in channel counts per output filter for a convolution layer determine the effectiveness of accuracy improvement when the model is scaled along the weight precision dimension and provide both theoretical and empirical reasoning for this.
We are the first to show that with a simple model scaling rule (the proposed DualPrecision), one can achieve a more accurate model (given the same model size) even compared to mixed-precision prior art that uses deep reinforcement learning to search for layer-wise weight precision values. Moreover, the results are validated on the large-scale dataset, i.e., ImageNet. This is a manifestation of our two previous findings.
The remainder of the paper is organized as follows. Section 2 discusses related work. Section discusses the methodology used to discover our findings. Section 4 shows lower precision values are preferable for networks with standard convolutions. Section 3 discusses how fan-in channel count per output filter affects precision scaling for convolution layers. Section discusses DualPrecision, our simple yet effective model scaling rule. Section 7 concludes the paper.
2 RELATED WORK
Several techniques for improving the efficiency of CNNs have been recently proposed. For in- stances, pruning removes the redundant connections of a trained neural network (Zhuang et al. 20188; Ye et al. (2018); Theis et al. (2018); Li et al. (2017); Frankle & Carbin (2019); CChin et al. 2019 Yu et al. 2018)) neural architecture search (NAS) tunes the number of channels, size of kernels, and depth of a network (Tan et al. 22118;; Stamoulis et al. (2019); Cai et al. (018); Sta- moulis al. (2018)) and convolution operations can be made more efficient via depth- wise con- volutions (Howard et al. (017)) group convolutions (Huang et al. (2017); Zhao et al. (2019h), and shift-based convolutions (He et al. (2019); [Wu et al. (2018a)). In addition to the aforemen- tioned techniques, network quantization introduces opportunity for hurdwareesoftwaan co-design to achieve better efficiency for CNNs.
There are in general two directions for weight quantization in prior literature, post-training quan- tization (Nagel et al. (2019); Meller et al. 2019);; Zhao et al. (2019)); Sheng et al. (2018) and quantization-ane awarr training Rastegari al. 2016 Zhu et al. (201 7): Jacob et al. (2018) y Jung al.
2019); Yuan et al. (2019): Hou & Kwok (2018); Choi et al. (2018). The former assumes training data is not available when quantization is applied. While being fast and craining-dtta-fees its per-
formance is worse compared to quantization-wware training. In contrast, our work falls under the category of quantization-awwe training.
In quantization-aware training, (Rastegari et al. (016)) introduces binary neural networks, which lead to significant efficiency gain by replacing multiplications with XNOR operations, but at the expense of significant accuracy degradation. Later, (Zhu et al. (2017)) propose ternary quantization and (Zhou et al. (2016): Jacob et al.) (2018)) bridge the gap between floating point networks and binarized neural networks by introducing fixed-point quantization. Building upon prior art, the vast majority of existing work focuses on reducing the accuracy degradation by improving the training strategy (Zhou et al. (2017); Yang et al. 2211 Louizos al. (2019); Ding et al. (2019) and better quantization schemes (Jung et al. (2019) y Wang et al. (2019); ) Yuan et al. (2019)) ). However, prior art studies quantization by fixing the network architecture, which may lead sub-optimal precision decisions in terms of Pareto efficiency (model size vs. accuracy).
Related to work, Mishra et al. (2018) have also considered the impact of channel count in quan- tization. In contrast, our work has the following novel features. First, we find that in CNNs with standard convolutions, lower precision values outperform higher ones in Pareto sense. Second,
2

---

Under review as a conference paper at ICLR 2020
find that the Pareto optimal precision value depends the number of input channels per output filter and provide theoretical insights for it. Last, we propose an algorithm to select the precision values of given network which result outperforms 8 bit fixed-poine and mixed-preciiss baselines.
METHODOLOGY
We conduct all of experiments image classification datasets including CIFAR- 100 and Ima- geNet. All the experiments are trained from scratch to ensure different precision values are trained equally long- While we do not start from pre-trained model, note that baseline fixed- point models (i.e., 4 bit for CIFAR and 8 bit for ImageNet) achieve iso-accurate results compared to their floating-point counterparts. For all the experiments on CIFAR, run the experiments three times and report the mean and standard deviation. The training hyper-parameters are detailed in Appendix A
3.1
QUANTIZATION
While our work focuses on weight quantization, we still quantize the activations since they are normally quantized for efficient deployment (Jacob al. (2018)). For activation quantization, we use the technique proposed in prior art (Jacob et al. (2018) and use 4 bit for CIFAR- 1-00 and 8 bit for ImageNet experiments. We note that the precision value chosen such that iso-accurate results can be achieved when compared to the floating-poin baselines.
For weight quantization, we use a straight-through estimator (Bengio et al. (2013) to conduct quantization-aware training. Specifically, for precision beyond 2 bit (b > 2), we use the follow- ing quantization function for weights during the forward pass:
Q( clamp( W 1.:, ai, ai)
x 8i, 8, 2b-1
(1)
8;
where [-] denotes the guund-to-neares--eiihhho function, W E Cout d = Cin Kw Kn, denotes the real-value weights for the 7th output filter of a convolution layer that has Crn. channels and Kw K₁. kernel size. C.ue denotes the vector of clipping factors which are selected to minimize [IQ Wa.113 by assuming W.; ~ NO, o²I). More details about the determination of a, is in Appendix B
For special cases such as bit and 1 bit, we use schemes proposed in prior art. Specifically, let us first define:
1 d
W
W
For precision, we follow trained ternary networks (Zhu et (2017) and define the quantization function as follows:
Q(W..) (sign W v.)) o M.,) (IW..)
[0, Wi, < O.7lW...l- Mij 11, otherwise.
For bit precision, we follow DoReFaNets (Zhou et al. 2016 and define the quantization function follows:
Q(W sign W..) (W..).
For the backward pass for all the precision values, we use straight-through estimator in prior to make the training differentiable. That is,
Q(
(5)
aw.
In the sequel, we quantize the first and last layers 8 bit. They are fixed throughout the experiments. We note that it is a common practice to leave the first and the last layer un-quantized (Zhou et al. 20166), however, we find that using 8 bit can achieve iso-accurate results.

---

Under review as a conference paper at ICLR 2020
3.2 COST METRICS
To measure the cost of CNN models, we use the size of the model (Caize) defined as:
Cuize Σ ())C(())
(6)
where O denotes the total number of filters and b(i) denotes the precision for filter i, C.. denotes the number of channels for filter i, and (w) and (t) denote the kernel height and width for filter i. We choose model size as a metric because it is relevant to both machine learning and systems community. Specifically, model size is of interest for the machine learning community since represents a proxy of model complexity. On the other hand, for the systems community, model size related to latency and energy for CNNs with weight fetch dominating memory accesses (e.g., the streaming inference scenario where inference is done with single data instance per batch). We note that this metric is also adopted in the MicroNet Challenge (Gale et al. (2019)) held at NeurIPS 2019.
4 DIFFERENT PRECISION VALUES HAVE DIFFERENT PARETO EFFICIENCY
To be precise the following discussion, we define Pareto domination as follows:
Definition 4.1 (Pareto domination) When comparing precision values A and B for a network fam- dly F, say A Pareto dominates B if.
(A, (B,(N((,s)) s)) Vs.
where Ace evaluates the validation accuracy of a network, (AA, uses width-multtpliir to find network in F such that it has A precision value and s model size.
We study three kinds of commonly adopted CNNs, namely, ResNets with Basic Block (He et al. (2016)) VGG (Simonyan & Zisserman (2014)), and MobileNetV2 (Sandler et al., (2018)). These networks differ the convolution operations, connections, and filter counts. For ResNets, we ex- plored the network from 20 layers up to 56 layers in step of six layers. For VGG, we investigate VGG with eleven layers. Additionally, we also study MobileNetV2, which is a mobile-friendly net- work. We note that modify the stride count in of the original MobileNetV2 to match the number of strides of ResNet for CIFAR. The used architectures are discussed in detail in Appendix C
For CIFAR- 100, only study precision values below 4 bit since the latter achieve iso-accurate results compared to its floating-poin counterpart. Specifically, we consider bit, bit, and 1 bit precision values. To compare the Pareto efficiency of different precision values, multiplier to align the model size among them. For example, one can make a 1-bit CNN 2x wider align with the model size of a 4-t CNN For each network, we sweep the width-multiplier to consider points at multiple model sizes. As it can be observed from Figure I across the three types of networks we study, there exists some precision value that Pareto dominates others. For ResNets and VGG, it is 1 bit. In contrast, for MobileNetV2, it is 4 bit. The results for ResNets and VGG are particularly interesting, since we observe that the lower precision value Pareto dominates the higher precision ones. This implies that for networks such as ResNets and VGG, scaling the model along the channel dimension always more preferable in accuracy-vs-S5 vsssie trade-off compared to scaling the model along the weight precision value dimension.
the width-
5 THE OPTIMAL PRECISION VALUE DEPENDS ON THE NUMBER OF FAN-IN CHANNELS
With the empirical results from Section we have learned that lower precision values are better for two of the networks we study but not for MobileNetV2, which has reversed behavior. In this section, we are interested in identifying the underlying cause for this different trend. Through a
2 Increase the width of layer increases the number of output filters for that layer as well as the number of channels for the subsequent layer. Thus, number of parameters and number of operations grow approximately guadratically with the mutth-uultiplir.
4

---

Under review as a conference paper at ICLR 2020
75
70.0
720
70
00.0
50
57.5
5.0
0.5
4.0
(a) ResNets (Layer 20 56 in step 6).
(b) VGGI1
(c) MobileNetV2 (modified stride for CIFAR)
Figure 1: Comparisons of the Pareto efficiency for different precision under three kinds of CNNs. zWyA denotes x-bit weight quantization and y-bit activation quantization. The experiments are done on the CIFAR-10 1-0 dataset. For each network, we sweep the width-multiplier to cover points multiple model sizes.
series of controlled experiments, we empirically identify that more channels per output filter leads to lower optimal precision value. In addition, provide theoretical insights behind this empirical result.
5.1 DEPTH-WISE CONVOLUTION
As it can be observed in Figure 1 MobileNetV2 a special case where higher precision val- Pareto dominate lower ones. When com- paring MobileNetV2 to the other two networks, there are many differences, including how con- volutions are connected, how many convolution layers are there, how many filters each of them, and how many channels for each con- volution. To narrow down which of these im- pacts the reversed trend, we first consider the inverted residual blocks, i.e., the basic compo- nent in MobileNetV2. To do so, we replace all basic blocks (two consecutive convolutions) of ResNet26 with the inverted residual blocks. We refer to this new network as Inv-ResNet26. As shown in Figure 2 the Pareto efficiency trend of Iny-ResNer26 resembles the of MobileNetV2 and recall that in case of ResNet26, lower precision values Pareto dominate higher ones. Thus, we can infer that the inverted residual block itself or its components are responsible for such a reversed trend.
75
1W4A
70
2W4A
Top-1 Accurccy
65
4W4A
loguolCawlIK
Figure The trade-off curves Inv-ResNet26.
Since inverted residual block is composed of point-wise convolution and depth-wise sepa- rable convolution, we further consider the case of depth-wise separable convolution (DWSConv). To identify whether DWSConv can cause the trend reversion, we use VGGI1 as starting point
LEAA
630
03.0
575
275
500
(a) Variant
(b) Variant
(c) Variant C
(d) Variant C without
quantizing depth-wise convolutions
Figure 3: The trade-off curves for the three variants of VGG11 that investigated.
5

---

Under review as a conference paper at ICLR 2020
and gradually replace each of the convolution with DWSConv. We note that replacing all convolu- tions with DSWConvs results in an architecture that resembles MobileNetV (Howard et al. (2017). Specifically, we introduce three variants of VGG11 that have increasing number of convolutions replaced by DWSConvs. Starting with the second layer, variant A has one layer replaced by DWS- Conv, variant B has four layers replaced by DWSConvs, and variant C has all of the layers except for the first layer replaced by DWSConvs (the architectures are detailed in Appendix C
As shown in Figure 3 as the number of DWSConv increases (from variant A to variant C), the opti- mal precision value shifts from bit to 4 bit, which implies that depth-wise separable convolutions or the layers within it are affecting the optimal precision value. To identify which of the layers of the DWSConv (i.e., the depth-wise weit convolution or the point-wise convolution) is more important in affecting the optimal precision value, we keep the precision value of depth wise convolutions fixed at 4 bit and quantize other layers. As shown in Figure [d] the optimal curve shifts from 4 bit being the best back to 1 bit, with similarly performing 2 bit. Thus, depth-wise convolutions appear to directly affect the optimal precision trends.
5.2 SENSITIVITY ANALYSIS
In our setup, to obtain lower precision network that has the same model size a higher pre- cision network we follow two steps: (1) quantize the network weights to lower-precision val- ues and (2) grow the network with width-multiplier to the model size of the higher-precision one. The two steps introduce accuracy differences of AAcc = Acciow - Acchigh and AAce
AcCtow.groun grown Acciown respectively. Since depth-wise convolutions introduce reverse trend Pareto efficiency, which is the result of AAcco + Accc, the reason can poten- tially be due to them being quantization-umfiie unfriendly, growing-unfriente unfriendly, both.
To further diagnose the reason why depth-wise convolutions have a reverse Pareto efficiency trend, analyze the accuracy differences for networks with and without quantizing depth-wise convo- lutions, Le., Figure 3c and Figure [3d] Specifically, we use width-multipliers of 1 Xx, 1.25x, 1.5x, 1.75x, and 2x for the 4-bit variant C as networks of higher precision. Thus, AAceo is evaluated against the corresponding 1-bit quantized model and A Acce is measured by comparing the 1-bit model and its 2x grown counterpart. As shown in Table ! when quantizing depth-wise convo- lutions, A AAcc becomes more negative such that AAcco + AACce < O. This implies that the main reason for the optimal precision value change that depth-wis wise convolutions are quantization- unfriendly when going below 4 bit. We note that we expected that quantizing the depth- convo- lutions would incur smaller A Acco compared to their no-quantization baseline because we essen- tially quantized more layers. However, depth-wise convolutions only account for 2% of the model size but incur on average near 4x more accuracy degradation when quantized.
We note that Sheng et al. (2018) also find that depth-wise separable convolutions are quantization- unfriendly. However, their results are based on post-training layer- quantization. As mentioned in their work (Sheng et al. (2018)). the quantization challenges in their setting could be resolved by quantization-awer -wware training, which is the scheme considered in this work. As a result, our finding is different and novel.
5.3
QUANTIZATION AND WEPTH-WISE CONVOLUTIONS
Having uncovered that depth-wise convolutions introduce large accuracy degradation when weights are quantized below bit, in this section, we investigate depth-wise convolutions from a quantization perspective. When comparing depth-wiss wise convolutions and standard convolutions in the context of
Table I: Analysis of the impact of depth-wise convolutions with AAcc = Acc - Accase and A ACcG = Acc Accibit by comparing variant C with and without quantizing the depth- wise convolutions from 4 bit to bit.
VARIANT C
A Arrg
1.00x
Adrc
A.Aing
1,25x
AAre
1.50x
AAenn
&
1.75x
AArto
AAreg
2.00x
A Aoca
AArr
AVERAGE
Adora
DWCONV
-1.54
+2.61
-1.77
+1.74
-1.58
+1.55
QUANT: ALL
.8.60
+4.39
-7.60
+3.41
-7.74
+3.19
-8.51
+4.09
-7.49
+2.25
-8.01
+3.47

---

Under review as a conference paper at ICLR 2020
quantization, they differ in the number of elements to be quantized, i.e. Cin for depth-wise convolutions and Ci >> for standard convolutions.
Why does the number of elements matter? In quantization aware training, needs to estimate some statistics of the vector to be quantized (i.e., in Equation 1 and | in Equations 3 based on the elements the vector. The number of elements affect the robustness of the estimate that further decides the quantized weights. More formally, we provide the following proposition.
Proposition 5.1 Let Rdd be the weight vector be quantized where we has distribution of (N o2) V without assuming samples are drawn independently and d = K Ka- If the average correlation of the weights is denoted by the variance of /wo can be written as follows:
o² + (d- (d-1)po² 20
Var(
d
d
The proof is in Appendix D This proposition states that, as the number of elements (d) increases, the variance of the estimate can be reduced due to the first term. The second term depends on the correlation between weights. Since the weights might not be independent during training, the variance is also affected by their correlations.
We empirically validate Proposition by looking into the sample variance of |w] across the course of t for different d values by growing (Kw, Kr) or Cin- To do so, we consider the 0.5x VGG variant by changing the number of elements of the depth-wise convolutions. Since = (Oin X Kw x Kτ) for convolution layer, consider the original depth-wise convolution, i.e., and increasing channels with d = x 3 x 3 and 16 x 3 x 3, and increasing and
1 x
x 12 x 12. The numbers selected such that growing the channel has the same d for the corresponding higher kernel size.
kernel size with = 1 x 6
analyze the layer-level sample variance by averaging for all the filters in the same layer. First, ob- serve that one can reduce the variance by increasing the number of elements along both the channel and kernel size dimen- sions. Second, we find that increasing the number of channels is more effective than increasing the kernel size in reduc- ing the variance, which could be due to a different correlation of the weights, Le.. intra-channel weights have larger correla- tion than inter-channel weights.
In Figure
0.006
x1x6x
1 Verilw D
0.005
x
x2
0.004
(16xxx33
0.003
0.002
0.001
0.000
Depth corvv. layer index
Figure 4: The average estimate Var([w]) for each depth-wiss wise convolution under different (Onn x K., K2) values.
However, lower variance might not neces- sarily imply lower quantization error for the quantized models. Thus, we conduct the AAce analysis for different d values. More specifi- cally, we want to understand how d affects the accuracy difference between lower precision (1 bit) and higher precision (4 bit) models (AAcoq) and the accuracy difference between the lower pre- cision (1 bit) and its grown (2x) counterpart (AAccg). As shown in Table 2 we empirically find that lower variance reflects larger AAccQ (less degradation). On the other hand, when comparing channel counts and kernel sizes, we observe that increasing the number of channels is more effective than increasing the kernel size in reducing accuracy degradation (larger A Accq). Moreover, we find that increasing kernel size reduces Aceg more than increasing the number of channels; this may be because larger kernel is harder to optimize and the CIFAR dataset does not benefit from larger receptive field. Indeed, from the last row of Table we can observe that increasing the kernel size reduces the accuracy for the 4 bit models.
Overall, from the Pareto efficiency perspective, whether the lower precision can have better accuracy when grown to the same model size as the
interested in AAcc which determines
We treat the calculated [m]] each training step as sample and calculate the sample variance across training steps.

---

Under review as a conference paper at ICLR 2020
Table 2:
Analysis of the impact of with AAcc and AAce for VGG variant C. AAcco Accıii - Accasit and Acca = 2cc Accibit-
VARIANT C
(1 3
(1 x 6)
(4
(1 x2 12)
(16 x 3 x3
AAcc
-22.77 + 1.16
99.21 0.68
-6,42 0.41
-7.07 1007:0.11 0.16
-4.45 1 0.49
AAceg
8.72 + 0.59
6.79 0.88
7.20
4.49
5.40 + 0.66
AAce
-3.66 0.58
0.27
0.79 0.13
-2.58 0.33
0.95 + 0.39
AcC4sar
56.24 0.37
55.63 = 0.47
58.51 + 0.39
53.31 = 0.22
60.97 + 0.30
higher precision model. In this case, find empirically that, as the number of channels per output filter increases, Ace increases. This implies that higher fan-in channel counts per output filter can benefit more from using lower weight precision values.
DUALPRECISION: PRECISION SELECTION FOR CNNS
From previous results, we find that the optimal precision value depends on the number of fan-in channels per output filter in a convolution layer and as the number of fan-in channels grows, the optimal precision value becomes smaller. Together with the observation that convolution layers in modern CNNs, except for depth-wise convolutions, have many channels per filter, we propose DualPrecision, which uses one precision value (presumably higher) for depth-wise convolutions and another precision value (presumably lower) for other convolution layers. Once the precision values are found, we use width-multipliers to grow or shrink the network the desired model size.
With this heuristic, the search space of precision selection becomes small that grid search is feasible, i.e., [B] [B] for networks with depth-wise convolutions and [BB otherwise. denotes the set of considered precision values and is typically small, e.g., {1. 2, 4, 8). We note that the search space for mixed precision (Wang et al. (2019); Wu et al. (2018b) is [B|" with L being the number of layers. In DualPreciston.
can explore the grid more efficiently by using heuristics that incorporate our findings. Specifically, we find that one precision value Pareto dominates others and result, train the network faster.
compare precision values at the regime of low computational cost to
We evaluate the proposed DualPrecision with ResNet50 and MobileNetV2 the ImageNet dataset. Since we keep the precision of the first and last layer quantized at 8 bit, scaling them in terms of width will grow the number of parameters much more quickly than other layers. As result, we keep the number of channels for the first and last channel fixed for the ImageNet experiments. We first conduct grid search (BB) {1,
8))) for ResNet50 and MobileNetV2 by scaling them down with width- andth-multiplierss so to make the grid search faster. Once the optimal precision is decided, we use width-man wodth-muutipllerss to traverse the trade-off curve. Specifically, we use the model size of the 0.25x 8-bit model to conduct grid search for both networks. For ResNet50, there are only four precision values to be searched while MobileNetV2 has such values. The grid search results are shown in Appendix E
Similar to our CIFAR experiments, we find that for networks with standard convolutions, i.e., ResNet50, the lower the precision value the better accuracy is. Thus, the selected precision is
Table 3: ImageNet results for DualPrecision. All the activations are quantized to bit. We note that the results for 8 bit iso-aceurate with their full-precision counterparts, which makes 8 bit strong baseline.
NETWORKS
METRODS
ToP-1 (%)
C.ose (10")
Top-1 (%)
Caize (10%)
Tup-1 (%)
Curee (10*)
63.85
102.87
76.70
204.18
RESNET50
PLEXIRLE DUALPRESCIION
WANG RT AI. 2019)
74.30
63.60
76.04
102.90
77.23
204.18
75.44
63.13
76.70
102.83
77.58
204.08
8 BIT
$2.17
11.99
64.39
15.44
71.73
27.50
MOBILENETV2
FLEXIBLE WANG
2019
55.20
12.10
65.00
15.54
72.13
27.71
DUALPRE REC ION YOU
59.51
12.15
68.01
15.53
73.91
27.56
8

---

Under review as a conference paper at ICLR 2020
1 bit. On the other hand, for MobileNetV2, find that 4 bit for standard convolution and 4 bit for depth-wise convolution perform the best. We consider two baselines to benchmark the proposed approach including 8-bit fixed-point and mixed peeceddpreiisoo networks (Wang al. (2019)) with width- multipliers. For sixxed-rrecssion we follow (Wang et al. (2019)) and use a reinforcement learning approach search for iso-accurate networks (iso- compared to the 8-bit fixed-point models). Then, we use width-multipliers on top of the searched network to obtain models of different sizes. We consider networks of three sizes, i.e., the size of 0.25x, 0.5x and 1x 8-bit fixed-point models. As shown in Table B our proposed simple heuristic outperforms both baselines by significant margin for both networks considered.
7 CONCLUSION
In this work, we discuss the Pareto efficiency of quantized convolutional neural networks (CNNs). We find that a lower weight precision value produces a more accurate network than higher weight precision one when the model size is aligned using a width-multiplier (i.e., growing or shrinking the number of channels proportionally.) for CNNs with standard convolutions. Furthermore, from both theoretical and empirical analyses, find that the fan-in channel counts per output filter of convolution layer determine the optimal precision value for that layer, which explains our observed phenomenon that depth-wise convolutions are less quantization-frieet frienddly compared their standard counterparts. Based on our findings, we propose DualPrecision a simple yet effective heuristic for precision selection of given network. We show empirically that, when applied on ImageNet, Dual- Precision outperforms the 8-bit fixed-point baseline and prior art in mixed- mixed-precision by significant margin.
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3433 2013.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint /812:: 00332, 2018.
Ting- Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Legr: Filter pruning via learned global ranking. arXiv preprint arXiv: 123668 2019.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). arXiv preprint arXiv: 1807.06964, 2018.
Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. arXiv preprint arXiv: 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://
wwt.treviie.nnee/ee forum///orrrm fiiimmidd-rr--- D31-33FFF7
Trevor Gale, Erich Elsen, Sara Hooker, Olivier Temam, Scott Gray, Jongsoo Park, Cliff Young, Utku Evei, Niki Parmar, and Ashish Vaswani. Micronet challenge hosted at neurips 2019. https: // mi miieonet-challengeee. github. 1 2019. Accessed: 2019-08-30.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.

---

Under review as a conference paper at ICLR 2020
Yihui He, Xianggen Liu, Huasong Zhong, and Yuchun Ma. Addressnet: Shift-based primitives for efficient convolutional neural networks. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 222-1122.. IEEE, 2019.
Lu Hou and James T. Kwok. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations, 2018. URLhttps:/ wpwp//eeeeeeeeeeeeeeeeeee. /tpeneee///penreviiwww
net/ forum?
(d-BkrSv01A:
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1704.4470, 2017.
Gao Huang, Shichen Liu, Laurens der Maaten, and Kilian Weinberger. Condensenet: An efficient densenet using learned group convolutions. group, 3(12):11, 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for effi- cient integer-rrihmneti-onny inference. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learnnng to quantize deep networks by optimizing quantization intervals with task loss. In The IEEE Conference on Computer Vision and Patlern Recognition (CVPR), June 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representation (ICLR), 2017.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019. htt.hhts:///// /tttps:s//peenreeeee...tt
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but differ- ent: Recovering neural network quantization error through weight factorization. In Kamalika the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4486 4495, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL htR.htttp://prroeeddigg. mml.
Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings the
press/v97 er html
Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. WRPN: Wide reduced-precisioe networks. In International Conference on Learning Representations, 2018. URL https:// openreview net. forum? d=B1 7vaaeA7
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization 2019.
through weight equalization and bias correction. arXiv preprint
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. A quantization-frienlly separable convolution for mobilenets. In 2018 1st Workshop Energy Efficient Muchine Learning and Cognitive Computing for Embedded Applications (EMC2), pp- 14-18. IEFE, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
10

---

Under review as a conference paper at ICLR 2020
Dimitrios Stamoulis, Ting- Wu Rudy Chin, Anand Krishnan Prakash, Haocheng Fang. Sribhuvan Sajja, Mitchell Bognar, and Diana Marculescu. Designing adaptive neural networks for energy- constrained image classification. In Proceedings of the International Conference on Computer- Aided Design, pp. 23. ACM, 2018.
Dimitrios Stamoulis. Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than hours. arXiv preprint arXiv:1904.02335 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le. Mnasnet: Platform- aware neural architecture search for mobile. arXiv preprint 1rXiv 1166,, 2018.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: 011880000777... 2018.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612-8620 2019.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial con- volutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9127-9135, 2018a.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint 18Xiv:/8/2.00090 2018b.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian- sheng Hua. Quantization networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-orrm-ess--nffonniss assumption in channel pruning of convolution layers. International Conference Learning Rep- resentation (ICLR), 2018.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. Nisp: Pruning networks using neuron importance score propagation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Banng-Yung
Xin Yuan, Liangliang Ren, Jiwen Lu, and Jie Zhou. Enhanced bayesian compression via deep reinforcement learning. In The IEEE Conference on Computer Vision and Patlern Recognition (CVPR), June 2019.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. fornia, USA, 09-15 Jun 2019a. PMLR. http:: httt:://proceedinga ml press/v977 zhao19c. html
Long Beach, Cali-
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Building efficient deep neural networks with unitary group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11303-11312, 2019b.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precnsion weights. International Conference on Learning Representations, 2017. URL wttphhttss.. h/tp:s:///pprrree revi ew n forum? (i-rummmi--HHHJmmmm 1
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: (666600666)) 2016.
11

---

Under review as a conference paper at ICLR 2020
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In International Conference on Learning Representations, 2017. URL /
net//orum?id=s] __1_l1
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrinination-awaare channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pp. 883-894, 894, 2018.
12

---

Under review as a conference paper at ICLR 2020
A TRAINING HYPER-PARAMETER HYPER-PARAMETERS
For CIFAR, we use learning rate of 0.05, cosine learning rate decay, linear learning rate warmuu (from 0.05) with 5 epochs, batch size of 128, total training epoch of 300, weight decay of 5e" SGD optimizer with Nesterov acceleration and 0.9 momentum.
For ImageNet, we have identical hyper-parameter as CIFAR except for the following hyper- parameters. Batch size of 256, 120 total epochs for MobileNetV2 and 90 for ResNets, weight decay 4e and 0. label smoothing.
B CLIPPING POINT FOR QUANTIZATION-AWAR TRAINING
As mentioned earlier, CoCn denotes the vector of clipping factors which is selected minimize I!)(W,:) W by assuming W. ~ N(0,0²1 More specifically, we run simulations for weights drawn from a zero-mean Gaussian distribution with several variances and identify the best a. arg min., I|Q.; (W..) W.. empirically. According to our simulation, we find that one can infer a, from the sample mean WW,, which is shown in Figure As result, for the different
precision values considered, we find Wi.d via simulation and use the obtained to calculate o; on-the-fly throughout training.
of
12
3 bit
--- bit
bit
bit
10
bit
10.
bit
bit
bit
bit
bit
B65..
0.5
1.0
1.5
2.0
2.5
3.0
o of).
of .70,
Figure 5: Finding best for different precision values empirically through simulation using Gaus- sian with various o².
C NETWORK ARCHITECTURES
For the experiments in Section 4 the ResNets used are detailed in Table 4 Specifically, for the points in Figure Ta we consider ResNet20 to ResNet56 with width-multipliers of 0.5x, 1x, 1.5x, and 2x for the 4-bit case. Based on these values. we consider additional width- hidth-muliipliess 2.4x and 2.8x for the 2-bit case and 2.5x, 3.5x, and 3.9x for the 1-bit case. We note that the right-most points Figure Ta is 10x ResNet26 for the bit case. On the other hand, VGG11 1 is detailed in Table for which we consider width-multipliers from 0.25x x to 2x with step of 0.25 for the 4 bit case (blue dots in Figure (b). The architecture of MobileNetV2 used in the CIFAR-I00 experiments follows the original MobileNetV2 (Table 2 in Sandler al. (2018)) but we change the stride of all the bottleneck blocks to except for the fifth bottleneck block, which has a stride of 2. As a result, we down-sample the image twice in total, which resembles the ResNet design for the CIFAR experiments (He et al. (2016)). Similar to VGG1 we consider multh-hultspierss from 0.25 x to 2x with step of ().25 for MobileNetV2 for the bit case (blue dots in Figure Ic).
13

---

Under review as a conference paper at ICLR 2020
Table 4: ResNet20 to ResNet56
LAYERS
20
32
50
56
STEM
CoNv2D (16,3,3) STRIDE
30,v2 3,3) STRIdE
STAGE 1
x
4x
5x
6
7x
8x
9x
CoNv2D(16, 3, 3, STRIDE
C3,, 3, STRIDE CoNv2D(32, 3, STRIDE
STAGE 2
4x
5x
6x
7x
8x
CoNv2D(64, 3, 3) STRIDE 2 CoNv2D(64, 3, 3) STRIDE
STAGE 3
3
4x
5x
6x
7x
8x
9x
Table 5: Inv-ResNet26
STEM
Conv2D (16,3,3) STRIDE
CoNv2D(16 6, 1, 1) STRIDE
STAGE
x
DWCONV2D(16 x 6, 3, 3) STRIDE 1
CoNv2D(16, 1) STRIDE
CONv2D(32 x 6, 1, 1) STRIDE 1
STAGE 2
4
DWCONv2D(32 x 6, 6,3,3) STRIDE 2
CoNv2D(32, 1, STRIDE
C6NV2D(64 x 6, 1, 1) STRIDE 1
STAGE 3
4
DWC x 6, 3, 3) STRIDE 2
CONV2D(64, 1, 1) STRIDE
D PROOF FOR PROPOSITION 5.1
Based on the definition of variance, we have:
27
Var
w.
= E
W
E
|
d
1=1
1=1
i=1
2
20
E
LD
1=1
2
20
E
wi
d
1=1
0²
20
10
d
d
E GRID SEARCH ON IMAGENET
From Table can observe a trend similar to the CIFAR-100 experiments, i.e., for networks without deith-wise convolutions, the lower precision the better, and for networks with depth-wise convolutions, there sweet spots for depth-wise convolution and other convolutions. Specifically, the final precision value selected for MobileNetV2 is bit for both depth-wise convolutions and standard convolutions. On the other hand, the selected precision value for ResNet50 is 1 bit.
14

---

Under review as a conference paper at ICLR 2020
Table 6: VGGs
VGGII
VARIANT A
VARIANT
VARIANT
CONV2D (64,3,3)
MAXPOOLING
CoNv2D(128, 1,
CoNv2D(128, 1,1)
CoNv2D(128, 1,
CONV2D (128,3,3)
DWCONv2D(128,3.5
DWCONv2D(128,3,3)
DWCoNv2D(128,3.5
MAXPOOLING
CoNv2D(256, 1,
CONV2D(256, 1)
CONV2D (256,3,3)
Conv2D (256,3,3)
DWCONv2D(256, 3,3)
DWCONv2D(256,33 3, 3)
CoNv2D(256,
CONV2D(256, 1,
CONv2D (256.3.3)
CONv2D (256.3.3)
DWCONv2D(256,3.2
DWCONv2D(256,335
MAXPOOLING
CoNv2D(512, 1,
CoNv2D(512, 1,
CONv2D (512.3.3)
CoNv2D (512.3.3)
DWCoNv2D(512,3/) 3,3)
DWCONN220(12,333
1,, 1,
CONv2D (512.3.3)
CoNv2D (512,3,3)
CoNv2D (512.3.3)
DWCONv2D(512,3.3)
MAXPOOLING
CoNv2D(612, 1,
CoNv2D (512.3.3)
CoNv2D (512.3.3)
CoNv2D (512.3.3)
DWCONv2D((12,3333
CONV2D(512, 1,1)
CONV2D (512.3.3)
CoNv2D (512,3.3)
CoNv2D (512.3.3)
DWCONv2D(512.3 3)
MAXPOOLING
Table 7: Grid search of DualPrecision for MobileNerV2 and ResNet50 with the model size aligned the 0.25x bit models. Each cell reports the top-1 accuracy of the corresponding model on ImageNet.
PRECISION FOR CONVS DWCONVS
MOBILENETV2
RESNET50 NONE
8
4 BIT
BIT
BIT
BIT
52.17
53.89
50.51
48.78
71.11
BIT
56.84
59.51
57.37
55.91
74.65
BIT
53.89
57.10
55.26
54.04
75.12
BIT
54.82
58.16
56.90
55.82
75.44
F MEMORY FOOTPRINT FOR INFERENCE
We calculate and report the memory footprint needed for the proposed DualPrecision models and the baseline 8-bit models to do inference with a single image per batch. Specifically, the memory footprint of inference equals the largest input feature maps plus the largest output feature maps plus the weight sizes for the entire network. As shown Figure 6 DualPrecision outperforms the base- line. That is, considering the streaming inference setting (a single image per batch), DualPrecision requires less memory to achieve equal accurate results compared to the 8-bit models.
15

---

Under review as a conference paper at ICLR 2020
ResNet50
MobileNetV2
75
bit DualPrecision
70
accurscy
6.5
Top-3
60
95
DuslPrecision
150
Memory footprint for inference (10%)
Memaar footprint for inference (10*)
(a) Memory footprint for ResNet50.
(b) Memory footprint for MobileNetV2.
Figure 6: Memory footprint needed for inference under the single-image-per-biit setting.
ResNet50
MobileNetv2
75
70
accuracy
65
Top-1
60
72
55
DualPrecision
DuslPrezision
71
Modiel size in bits (10F)
Model size bils (10")
(a) Model size for ResNet50.
(b) Model size MobileNetV2.
Figure 7: The visualization of Table 3.
16


Reviews:
[{"review_id": "rkgWqCv6FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper studies the accuracy vs model-size trade-off of quantized CNNs under different channel width multipliers. The authors demonstrated that while all-to-all convolution works well under low bit settings, depthwise conv needs a different sweet spot. The authors then proceed to use the insight to design quantized cnns that have two different schemes for depthwise and normal conv.\n\n\nStrength\n\nThe paper is well written and motivated. By adding network width to the search space,  using the simple heuristics, the authors provide a better results than previously DRL based search method.\n\nWeakness\n\nOne of my main concerns is the direct usage of total number of bits as an equivalent measurement between models. While it is useful to measure the storage cost for weights. The choices of bit width will likely affect the computing cost in a non-trivial way, depending on the target hardware platform. It is unclear whether equal model size would mean equal inference latency in practice (most likely they would not be). Providing empirical implementations of these models will shed light into this question.\n\nThese weakness makes it a borderline paper.\n\nQuestion:\n\nHow do you handle batchnorm layer, do you use floating points?\n\nHow many bits did you use for accumulating the results?\n\nUpdate after rebuttal:\n\nI have read the authors' response. I would like to keep my current review as it is. Also note that the authors uses floating pt for batchnorm and 32 bit for accumulation, such additional cost might out-weights the benefit of choosing ultra low bits in the low bits regime, making the study less relevant for practice reasons\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xxtycQ5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigated the Pareto efficiency of quantized convolutional neural networks (CNNs). \nThe authors pointed out that  when the number of channels of a network are changed in an iso-model size scenario, the lower precision value can Pareto dominate the higher ones for CNNs with standard convolutions. Furthermore, the authors showed that depth-wise convolutions are quantization-unfriendly when going low- precision even in quantization-aware training. We further provide theoretical insights for it and show that the number of input channels per output filter affects the Pareto optimal precision value. The authors then proposed DualPrecision, a simple yet effective heuristic for precision selection of a given network. When applied on ImageNet, DualPrecision outperforms the 8 bit fixed point baseline and prior art in mixed-precision by a significant margin.\n\n1. The wording issues \n- The paper used words like \"iso-model\" and \"Pareto\" at many parts of the draft, making it quite hard to read..  \n\n2. The experiments seem fair. However, the improvement seems quite marginal\nThe proposed method improved the baseline by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.\n\nI have to admit that I am unfamiliar with this particular topic and only knows a few classic papers, like for example Xnor-net: Imagenet classification using binary convolutional neural networks", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeR13sCFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author studies the quantization strategy of CNNs in terms of Pareto Efficiency. Through a series of experiments with three standard CNN models (ResNet, VGG11, MobileNetV2), the authors demonstrated that lower precision value can be better than high precision values in term of Pareto efficiency under the iso-model size scenario. They also study cases with and without depth-wise convolution, and propose a new quantization method, DualPrecision. DualPrecision empirically outperformed 8-bit quantization and flexible quantization methods on ImageNet.\n\nComment:\nI am not at all familiar with quantization methods, therefore no knowledge of relevant related works. If, however, the authors did a thorough job of surveying related works and chose sensible baselines, I think the experiments demonstrate the usefulness of the new DualPrecision technique.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the trade-off between the model size and quantization levels in quantized CNNs by varying different channel width multipliers. The paper is well  motivated and draws interesting observations but can be improved in terms of evaluation. It is a borderline case and rejection is made due to the high competition. 
Decision: Reject