Title: ON THE PARETO EFFICIENCY OF QUANTIZED CNNS

Abstract: Weight quantization for deep convolutional neural networks (CNNs) has shown promising results in compressing and accelerating CNN-powered applications such as semantic segmentation, gesture recognition, and scene understanding. Prior art has shown that different datasets, tasks, and network architectures admit different iso-accurate precision values, which increase the complexity of efficient quantized neural network implementations from both hardware and software perspectives. In this work, we show that when the number of channels is allowed to vary such that networks of different precision values have the same model size, lower precision values outperform higher precision ones in a Pareto sense (accuracy vs. model size) for networks with standard convolutions. Relying on comprehensive empirical analyses, we find that the optimal precision value of a convolution layer depends on the number of input channels per output filters and provide theoretical insights for it. To this end, we develop a simple algorithm to select the precision values for CNNs that outperforms corresponding 8-bit quantized networks by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.

1. INTRODUCTION:

Recent success of convolutional neural networks (CNNs) in computer vision applications such as image classification and semantic segmentation, have fueled many important applications in energyconstrained devices, e.g., virtual reality headsets, drones, and robots. As a result, improving the energy-efficiency of CNNs while maintaining their attractive features (e.g., accuracy for a task) has gained tremendous research momentum in recent years.

Among the efforts of improving CNNs' efficiency, weight quantization was shown to be an effective technique (Zhou et al. (2016; 2017) ; Hou & Kwok (2018) ; Ding et al. (2019) ). The majority of research efforts in quantization have been devoted to develop better quantization algorithms such that an iso-figure-of-merit (i.e., accuracy) is achieved with lowest possible weight precision value. Nevertheless, the iso-accurate precision value depends on the dataset, task, and network architecture of interest, which greatly increases the neural network implementation complexity from both hardware and software perspectives. For example, hardware and software implementations optimized for executing an 8 bit network are sub-optimal for executing a 4 bit network, and vice versa. The design optimization complexity further increases for recently proposed mixed-precision networks (Wang et al. (2019) ; Wu et al. (2018b) ; Dong et al. (2019) ).

The key observation we have is that most prior literature in this space studies quantization for fixed network architectures, which is reasonable for evaluating the effectiveness of quantization algorithms, but unnecessary when considering the Pareto efficiency (accuracy vs. model size) of neural networks. In this work, we relax the restriction of fixing the network architecture and allow the number of channels of the CNN under consideration to vary. More concretely, we use the widthmultiplier1 (Howard et al. (2017) ) as a tool to compare the performance of different weight precision values under the same model size.

Overall, we systematically analyze the model size and accuracy trade-offs considering both weight precision values and the number of channels for various modern networks architectures (variants of ResNet, VGG, and MobileNet) and datasets (CIFAR and ImageNet) and have the following nontrivial and novel contributions:

• We are the first to empirically show that when considering channel counts, lower precision weight values outperform higher precision weight values in a Pareto sense (accuracy vs. model size) for networks with standard convolutions. This is intriguing since it implies that scaling up (in terms of model size) along the channel count dimension is more effective for accuracy than the precision value dimension.

• We are the first to show that the fan-in channel counts per output filter for a convolution layer determine the effectiveness of accuracy improvement when the model is scaled along the weight precision dimension and provide both theoretical and empirical reasoning for this.

• We are the first to show that with a simple model scaling rule (the proposed DualPrecision), one can achieve a more accurate model (given the same model size) even compared to mixed-precision prior art that uses deep reinforcement learning to search for layer-wise weight precision values. Moreover, the results are validated on the large-scale dataset, i.e., ImageNet. This is a manifestation of our two previous findings.

The remainder of the paper is organized as follows. Section 2 discusses related work. Section 3 discusses the methodology used to discover our findings. Section 4 shows lower precision values are preferable for networks with standard convolutions. Section 5 discusses how fan-in channel count per output filter affects precision scaling for convolution layers. Section 6 discusses DualPrecision, our simple yet effective model scaling rule. Section 7 concludes the paper.

2. RELATED WORK:

Several techniques for improving the efficiency of CNNs have been recently proposed. For instances, pruning removes the redundant connections of a trained neural network (Zhuang et al. ( In quantization-aware training, (Rastegari et al. (2016) ) introduces binary neural networks, which lead to significant efficiency gain by replacing multiplications with XNOR operations, but at the expense of significant accuracy degradation. Later, (Zhu et al. (2017) ) propose ternary quantization and (Zhou et al. (2016) Related to our work, Mishra et al. (2018) have also considered the impact of channel count in quantization. In contrast, our work has the following novel features. First, we find that in CNNs with standard convolutions, lower precision values outperform higher ones in a Pareto sense. Second, we find that the Pareto optimal precision value depends on the number of input channels per output filter and provide theoretical insights for it. Last, we propose an algorithm to select the precision values of a given network which as a result outperforms 8 bit fixed-point and mixed-precision baselines.

3. METHODOLOGY:

We conduct all of our experiments on image classification datasets including CIFAR-100 and Ima-geNet. All the experiments are trained from scratch to ensure different precision values are trained equally long. While we do not start from a pre-trained model, we note that our baseline fixedpoint models (i.e., 4 bit for CIFAR and 8 bit for ImageNet) achieve iso-accurate results compared to their floating-point counterparts. For all the experiments on CIFAR, we run the experiments three times and report the mean and standard deviation. The training hyper-parameters are detailed in Appendix A.

3.1 QUANTIZATION While our work focuses on weight quantization, we still quantize the activations since they are normally quantized for efficient deployment (Jacob et al. (2018) ). For activation quantization, we use the technique proposed in prior art (Jacob et al. (2018) ) and use 4 bit for CIFAR-100 and 8 bit for ImageNet experiments. We note that the precision value is chosen such that iso-accurate results can be achieved when compared to the floating-point baselines.

For weight quantization, we use a straight-through estimator (Bengio et al. (2013) ) to conduct quantization-aware training. Specifically, for precision beyond 2 bit (b > 2), we use the following quantization function for weights during the forward pass:

EQUATION

where • denotes the round-to-nearest-neighbor function, W ∈ R Cout×d , d = C in K w K h denotes the real-value weights for the i th output filter of a convolution layer that has C in channels and K w × K h kernel size. a ∈ R Cout denotes the vector of clipping factors which are selected to minimize Q(W i,: )-W i,: 2 2 by assuming W i,: ∼ N (0, σ 2 I). More details about the determination of a i is in Appendix B.

For special cases such as 2 bit and 1 bit, we use schemes proposed in prior art. Specifically, let us first define:

EQUATION

For 2 bit precision, we follow trained ternary networks (Zhu et al. (2017) ) and define the quantization function as follows:

EQUATION

For 1 bit precision, we follow DoReFaNets (Zhou et al. (2016) ) and define the quantization function as follows:

EQUATION

)

For the backward pass for all the precision values, we use a straight-through estimator as in prior art to make the training differentiable. That is,

EQUATION

In the sequel, we quantize the first and last layers to 8 bit. They are fixed throughout the experiments. We note that it is a common practice to leave the first and the last layer un-quantized (Zhou et al. (2016) ), however, we find that using 8 bit can achieve iso-accurate results.

3.2. COST METRICS:

To measure the cost of CNN models, we use the size of the model (C size ) defined as:

EQUATION

where O denotes the total number of filters and b(i) denotes the precision for filter i, C in (i) denotes the number of channels for filter i, and K w (i) and K h (i) denote the kernel height and width for filter i. We choose model size as a metric because it is relevant to both machine learning and systems community. Specifically, model size is of interest for the machine learning community since it represents a proxy of model complexity. On the other hand, for the systems community, model size is related to latency and energy for CNNs with weight fetch dominating memory accesses (e.g., the streaming inference scenario where inference is done with single data instance per batch). We note that this metric is also adopted in the MicroNet Challenge (Gale et al. ( 2019)) held at NeurIPS 2019.

4. DIFFERENT PRECISION VALUES HAVE DIFFERENT PARETO EFFICIENCY:

To be precise in the following discussion, we define Pareto domination as follows: 

Acc(N (A, s)) > Acc(N (B, s)) ∀s,

where Acc evaluates the validation accuracy of a network, N (A, s) uses width-multiplier to find a network in F such that it has A precision value and s model size.

We study three kinds of commonly adopted CNNs, namely, ResNets with Basic Block (He et al. (2016) ), VGG (Simonyan & Zisserman (2014) ), and MobileNetV2 (Sandler et al. (2018) ). These networks differ in the convolution operations, connections, and filter counts. For ResNets, we explored the network from 20 layers up to 56 layers in a step of six layers. For VGG, we investigate VGG with eleven layers. Additionally, we also study MobileNetV2, which is a mobile-friendly network. We note that we modify the stride count in of the original MobileNetV2 to match the number of strides of ResNet for CIFAR. The used architectures are discussed in detail in Appendix C.

For CIFAR-100, we only study precision values below 4 bit since the latter can achieve iso-accurate results compared to its floating-point counterpart. Specifically, we consider 4 bit, 2 bit, and 1 bit precision values. To compare the Pareto efficiency of different precision values, we use the widthmultiplier to align the model size among them. For example, one can make a 1-bit CNN 2× wider to align with the model size of a 4-bit CNN2 . For each network, we sweep the width-multiplier to consider points at multiple model sizes. As it can be observed from Figure 1 , across the three types of networks we study, there exists some precision value that Pareto dominates others. For ResNets and VGG, it is 1 bit. In contrast, for MobileNetV2, it is 4 bit. The results for ResNets and VGG are particularly interesting, since we observe that the lower precision value Pareto dominates the higher precision ones. This implies that for networks such as ResNets and VGG, scaling the model along the channel dimension is always more preferable in accuracy-vs-size trade-off compared to scaling the model along the weight precision value dimension.

5. THE OPTIMAL PRECISION VALUE DEPENDS ON THE NUMBER OF FAN-IN CHANNELS:

With the empirical results from Section 4, we have learned that lower precision values are better for two of the networks we study but not for MobileNetV2, which has a reversed behavior. In this section, we are interested in identifying the underlying cause for this different trend. Through a series of controlled experiments, we empirically identify that more channels per output filter leads to lower optimal precision value. In addition, we provide theoretical insights behind this empirical result.

5.1. DEPTH-WISE CONVOLUTION:

Figure 2 : The trade-off curves for Inv-ResNet26.

As it can be observed in Figure 1 , MobileNetV2 is a special case where higher precision values Pareto dominate lower ones. When comparing MobileNetV2 to the other two networks, there are many differences, including how convolutions are connected, how many convolution layers are there, how many filters in each of them, and how many channels for each convolution. To narrow down which of these impacts the reversed trend, we first consider the inverted residual blocks, i.e., the basic component in MobileNetV2. To do so, we replace all basic blocks (two consecutive convolutions) of ResNet26 with the inverted residual blocks. We refer to this new network as Inv-ResNet26. As shown in Figure 2 , the Pareto efficiency trend of Inv-ResNet26 resembles the one of MobileNetV2 and recall that in case of ResNet26, lower precision values Pareto dominate higher ones. Thus, we can infer that the inverted residual block itself or its components are responsible for such a reversed trend.

Since an inverted residual block is composed of a point-wise convolution and a depth-wise separable convolution, we further consider the case of depth-wise separable convolution (DWSConv).

To identify whether DWSConv can cause the trend reversion, we use VGG11 as a starting point As shown in Figure 3 , as the number of DWSConv increases (from variant A to variant C), the optimal precision value shifts from 1 bit to 4 bit, which implies that depth-wise separable convolutions or the layers within it are affecting the optimal precision value. To identify which of the layers of the DWSConv (i.e., the depth-wise convolution or the point-wise convolution) is more important in affecting the optimal precision value, we keep the precision value of depth-wise convolutions fixed at 4 bit and quantize other layers. As shown in Figure 3d , the optimal curve shifts from 4 bit being the best back to 1 bit, with a similarly performing 2 bit. Thus, depth-wise convolutions appear to directly affect the optimal precision trends.

5.2. SENSITIVITY ANALYSIS:

In our setup, to obtain a lower precision network that has the same model size as a higher precision network we follow two steps: (1) quantize the network weights to lower-precision values and (2) grow the network with width-multiplier to the model size of the higher-precision one. The two steps introduce accuracy differences of ∆Acc Q = Acc low -Acc high and ∆Acc G = Acc low,grown -Acc low , respectively. Since depth-wise convolutions introduce a reverse trend in Pareto efficiency, which is the result of ∆Acc Q + ∆Acc G , the reason can potentially be due to them being quantization-unfriendly, growing-unfriendly, or both.

To further diagnose the reason why depth-wise convolutions have a reverse Pareto efficiency trend, we analyze the accuracy differences for networks with and without quantizing depth-wise convolutions, i.e., Figure 3c and Figure 3d . Specifically, we use width-multipliers of 1×, 1.25×, 1.5×, 1.75×, and 2× for the 4-bit variant C as networks of higher precision. Thus, ∆Acc Q is evaluated against the corresponding 1-bit quantized model and ∆Acc G is measured by comparing the 1-bit model and its 2× grown counterpart. As shown in Table 1 , when quantizing depth-wise convolutions, ∆Acc Q becomes more negative such that ∆Acc Q + ∆Acc G < 0. This implies that the main reason for the optimal precision value change is that depth-wise convolutions are quantizationunfriendly when going below 4 bit. We note that we expected that quantizing the depth-wise convolutions would incur smaller ∆Acc Q compared to their no-quantization baseline because we essentially quantized more layers. However, depth-wise convolutions only account for 2% of the model size but incur on average near 4× more accuracy degradation when quantized.

We note that Sheng et al. ( 2018) also find that depth-wise separable convolutions are quantizationunfriendly. However, their results are based on post-training layer-wise quantization. As mentioned in their work (Sheng et al. (2018) ), the quantization challenges in their setting could be resolved by quantization-aware training, which is the scheme considered in this work. As a result, our finding is different and novel.

5.3. QUANTIZATION AND DEPTH-WISE CONVOLUTIONS:

Having uncovered that depth-wise convolutions introduce large accuracy degradation when weights are quantized below 4 bit, in this section, we investigate depth-wise convolutions from a quantization perspective. When comparing depth-wise convolutions and standard convolutions in the context of quantization, they differ in the number of elements to be quantized, i.e., C in = 1 for depth-wise convolutions and C in >> 1 for standard convolutions.

Why does the number of elements matter? In quantization-aware training, one needs to estimate some statistics of the vector to be quantized (i.e., a in Equation 1 and |w| in Equations 3,4) based on the elements in the vector. The number of elements affect the robustness of the estimate that further decides the quantized weights. More formally, we provide the following proposition.

Proposition 5.1 Let w ∈ R d be a the weight vector to be quantized where w i has distribution of N (0, σ 2 ) ∀ i without assuming samples are drawn independently and d = C in K w K h . If the average correlation of the weights is denoted by ρ, the variance of |w| can be written as follows:

EQUATION

)

The proof is in Appendix D. This proposition states that, as the number of elements (d) increases, the variance of the estimate can be reduced due to the first term. The second term depends on the correlation between weights. Since the weights might not be independent during training, the variance is also affected by their correlations.

We In Figure 4 , we analyze the layer-level sample variance by averaging it for all the filters in the same layer. First, we observe that one can reduce the variance by increasing the number of elements along both the channel and kernel size dimensions. Second, we find that increasing the number of channels is more effective than increasing the kernel size in reducing the variance, which could be due to a different correlation of the weights, i.e., intra-channel weights have larger correlation than inter-channel weights.

However, lower variance might not necessarily imply lower quantization error for the quantized models. Thus, we conduct the ∆Acc analysis for different d values. More specifically, we want to understand how d affects the accuracy difference between lower precision (1 bit) and higher precision (4 bit) models (∆Acc Q ) and the accuracy difference between the lower precision (1 bit) and its grown (2×) counterpart (∆Acc G ). As shown in Table 2 , we empirically find that lower variance reflects larger ∆Acc Q (less degradation). On the other hand, when comparing channel counts and kernel sizes, we observe that increasing the number of channels is more effective than increasing the kernel size in reducing accuracy degradation (larger ∆Acc Q ). Moreover, we find that increasing kernel size reduces Acc G more than increasing the number of channels; this may be because a larger kernel is harder to optimize and the CIFAR dataset does not benefit from larger receptive field. Indeed, from the last row of Table 2 , we can observe that increasing the kernel size reduces the accuracy for the 4 bit models.

Overall, from the Pareto efficiency perspective, we are interested in ∆Acc, which determines whether the lower precision can have better accuracy when grown to the same model size as the higher precision model. In this case, we find empirically that, as the number of channels per output filter increases, ∆Acc increases. This implies that higher fan-in channel counts per output filter can benefit more from using lower weight precision values.

= Acc 1bit -Acc 4bit and ∆Acc G = Acc 1bit,2× -Acc 1bit . VARIANT C (1 × 3 × 3) (1 × 6 × 6) (4 × 3 × 3) (1 × 12 × 12) (16 × 3 × 3) ∆AccQ -12.

6. DUALPRECISION: PRECISION SELECTION FOR CNNS:

From previous results, we find that the optimal precision value depends on the number of fan-in channels per output filter in a convolution layer and as the number of fan-in channels grows, the optimal precision value becomes smaller. Together with the observation that convolution layers in modern CNNs, except for depth-wise convolutions, have many channels per filter, we propose DualPrecision, which uses one precision value (presumably higher) for depth-wise convolutions and another precision value (presumably lower) for other convolution layers. Once the precision values are found, we use width-multipliers to grow or shrink the network to the desired model size.

With this heuristic, the search space of precision selection becomes so small that grid search is feasible, i.e., |B| × |B| for networks with depth-wise convolutions and |B| otherwise. B denotes the set of considered precision values and is typically small, e.g., {1, 2, 4, 8}. We note that the search space for mixed precision (Wang et al. (2019) ; Wu et al. (2018b) ) is |B| L with L being the number of layers. In DualPrecision, one can explore the grid more efficiently by using heuristics that incorporate our findings. Specifically, we find that one precision value Pareto dominates others and as a result, one can compare precision values at the regime of low computational cost so as to train the network faster.

We evaluate the proposed DualPrecision with ResNet50 and MobileNetV2 on the ImageNet dataset.

Since we keep the precision of the first and last layer quantized at 8 bit, scaling them in terms of width will grow the number of parameters much more quickly than other layers. As a result, we keep the number of channels for the first and last channel fixed for the ImageNet experiments. We first conduct grid search (|B| = {1, 2, 4, 8}) for ResNet50 and MobileNetV2 by scaling them down with width-multipliers so as to make the grid search faster. Once the optimal precision is decided, we use width-multipliers to traverse the trade-off curve. Specifically, we use the model size of the 0.25× 8-bit model to conduct grid search for both networks. For ResNet50, there are only four precision values to be searched while MobileNetV2 has 16 such values. The grid search results are shown in Appendix E.

Similar to our CIFAR experiments, we find that for networks with standard convolutions, i.e., ResNet50, the lower the precision value the better accuracy is. Thus, the selected precision is 1 bit. On the other hand, for MobileNetV2, we find that 4 bit for standard convolution and 4 bit for depth-wise convolution perform the best. We consider two baselines to benchmark the proposed approach including 8-bit fixed-point and mixed-precision networks (Wang et al. (2019) ) with widthmultipliers. For mixed-precision, we follow (Wang et al. (2019) ) and use a reinforcement learning approach to search for iso-accurate networks (iso-compared to the 8-bit fixed-point models). Then, we use width-multipliers on top of the searched network to obtain models of different sizes. We consider networks of three sizes, i.e., the size of 0.25×, 0.5× and 1× 8-bit fixed-point models. As shown in Table 3 , our proposed simple heuristic outperforms both baselines by a significant margin for both networks considered.

7. CONCLUSION:

In this work, we discuss the Pareto efficiency of quantized convolutional neural networks (CNNs). We find that a lower weight precision value produces a more accurate network than higher weight precision one when the model size is aligned using a width-multiplier (i.e., growing or shrinking the number of channels proportionally.) for CNNs with standard convolutions. Furthermore, from both theoretical and empirical analyses, we find that the fan-in channel counts per output filter of a convolution layer determine the optimal precision value for that layer, which explains our observed phenomenon that depth-wise convolutions are less quantization-friendly compared to their standard counterparts. Based on our findings, we propose DualPrecision, a simple yet effective heuristic for precision selection of a given network. We show empirically that, when applied on ImageNet, Dual-Precision outperforms the 8-bit fixed-point baseline and prior art in mixed-precision by a significant margin.

A TRAINING HYPER-PARAMETERS:

For CIFAR, we use a learning rate of 0.05, cosine learning rate decay, linear learning rate warmup (from 0 to 0.05) with 5 epochs, batch size of 128, total training epoch of 300, weight decay of 5e -4 , SGD optimizer with Nesterov acceleration and 0.9 momentum.

For ImageNet, we have identical hyper-parameters as CIFAR except for the following hyperparameters. Batch size of 256, 120 total epochs for MobileNetV2 and 90 for ResNets, weight decay 4e -5 , and 0.1 label smoothing.

B CLIPPING POINT FOR QUANTIZATION-AWARE TRAINING:

As mentioned earlier, a ∈ R Cout denotes the vector of clipping factors which is selected to minimize

EQUATION

2 by assuming W i,: ∼ N (0, σ 2 I). More specifically, we run simulations for weights drawn from a zero-mean Gaussian distribution with several variances and identify the best 

EQUATION

C NETWORK ARCHITECTURES:

For the experiments in Section 4, the ResNets used are detailed in Table 4 . Specifically, for the points in Figure 1a , we consider ResNet20 to ResNet56 with width-multipliers of 0.5×, 1×, 1.5×, and 2× for the 4-bit case. Based on these values, we consider additional width-multipliers 2.4× and 2.8× for the 2-bit case and 2.5×, 3×, 3.5×, and 3.9× for the 1-bit case. We note that the right-most points in Figure 1a is a 10× ResNet26 for the 4 bit case. On the other hand, VGG11 is detailed in Table 6 for which we consider width-multipliers from 0.25× to 2× with a step of 0.25 for the 4 bit case (blue dots in Figure 1b ). The architecture of MobileNetV2 used in the CIFAR-100 experiments follows the original MobileNetV2 (Table 2 in Sandler et al. ( 2018)) but we change the stride of all the bottleneck blocks to 1 except for the fifth bottleneck block, which has a stride of 2. As a result, we down-sample the image twice in total, which resembles the ResNet design for the CIFAR experiments (He et al. (2016) ). Similar to VGG11, we consider width-multipliers from 0.25× to 2× with a step of 0.25 for MobileNetV2 for the 4 bit case (blue dots in Figure 1c ). Based on the definition of variance, we have:

Var( 1 d d i=1 |w i |) := E   1 d d i=1 |w i | 2 -E 1 d d i=1 |w i | 2   = E   1 d d i=1 |w i | 2 - 2σ 2 π   = 1 d 2 E d i=1 |w i | 2 - 2σ 2 π = σ 2 d + d -1 d ρσ 2 - 2σ 2 π .

E GRID SEARCH ON IMAGENET:

From Table 7 , we can observe a trend similar to the CIFAR-100 experiments, i.e., for networks without depth-wise convolutions, the lower precision the better, and for networks with depth-wise convolutions, there are sweet spots for depth-wise convolution and other convolutions. Specifically, the final precision value selected for MobileNetV2 is 4 bit for both depth-wise convolutions and standard convolutions. On the other hand, the selected precision value for ResNet50 is 1 bit. 

F MEMORY FOOTPRINT FOR INFERENCE:

We calculate and report the memory footprint needed for the proposed DualPrecision models and the baseline 8-bit models to do inference with a single image per batch. Specifically, the memory footprint of inference equals the largest input feature maps plus the largest output feature maps plus the weight sizes for the entire network. As shown in Figure 6 , DualPrecision outperforms the baseline. That is, considering the streaming inference setting (a single image per batch), DualPrecision requires less memory to achieve equal accurate results compared to the 8-bit models.

:

Width-multiplier grows or shrinks the number of channels across the layers with identical proportion for a certain network, e.g., grow the number of channels for all the layers by

2×.

Increase the width of a layer increases the number of output filters for that layer as well as the number of channels for the subsequent layer. Thus, number of parameters and number of operations grow approximately quadratically with the width-multiplier.

We treat the calculated |w| at each training step as a sample and calculate the sample variance across training steps.

Figures and Tables:

); Ye et al. (2018); Theis et al. (2018); Li et al. (2017); Frankle & Carbin (2019); Chin et al. (2019); Yu et al. (2018)), neural architecture search (NAS) tunes the number of channels, size of kernels, and depth of a network (Tan et al. (2018); Stamoulis et al. (2019); Cai et al. (2018); Stamoulis et al. (2018)), and convolution operations can be made more efficient via depth-wise convolutions (Howard et al. (2017)), group convolutions (Huang et al. (2017); Zhao et al. (2019b)), and shift-based convolutions (He et al. (2019); Wu et al. (2018a)). In addition to the aforementioned techniques, network quantization introduces an opportunity for hardware-software co-design to achieve better efficiency for CNNs. There are in general two directions for weight quantization in prior literature, post-training quantization (Nagel et al. (2019); Meller et al. (2019); Zhao et al. (2019a); Sheng et al. (2018)) and quantization-aware training (Rastegari et al. (2016); Zhu et al. (2017); Jacob et al. (2018); Jung et al. (2019); Yuan et al. (2019); Hou & Kwok (2018); Choi et al. (2018)). The former assumes training data is not available when quantization is applied. While being fast and training-data-free, its performance is worse compared to quantization-aware training. In contrast, our work falls under the category of quantization-aware training.

Definition 4.1 (Pareto domination) When comparing precision values A and B for a network family F, we say A Pareto dominates B if,

Figure 1: Comparisons of the Pareto efficiency for different precision under three kinds of CNNs. xWyA denotes x-bit weight quantization and y-bit activation quantization. The experiments are done on the CIFAR-100 dataset. For each network, we sweep the width-multiplier to cover points at multiple model sizes.

Figure 3: The trade-off curves for the three variants of VGG11 that we investigated.

empirically validate Proposition 5.1 by looking into the sample variance of |w| across the course of training 3 for different d values by growing (K w , K h ) or C in . To do so, we consider the 0.5× VGG variant C by changing the number of elements of the depth-wise convolutions. Since d = (C in × K w × K h ) for a convolution layer, we consider the original depth-wise convolution, i.e., d = 1 × 3 × 3 and increasing channels with d = 4 × 3 × 3 and d = 16 × 3 × 3, and increasing kernel size with d = 1 × 6 × 6, , and d = 1 × 12 × 12. The numbers are selected such that growing the channel has the same d for the corresponding higher kernel size.

Figure 4: The average estimate Var( |w|) for each depth-wise convolution under different d = (C in × K w × K h ) values.

empirically. According to our simulation, we find that one can infer a i from the sample mean |W i,: |, which is shown in Figure 5. As a result, for the different precision values considered, we find c = |Wi,:| a * i via simulation and use the obtained c to calculate a i on-the-fly throughout training.

Figure 5: Finding best a i for different precision values empirically through simulation using Gaussian with various σ 2 .

Analysis of the impact of depth-wise convolutions with ∆Acc Q = Acc 1bit -Acc 4bit and ∆Acc G = Acc 1bit,2× -Acc 1bit by comparing variant C with and without quantizing the depth-wise convolutions from 4 bit to 1 bit.

Table content: <table><tr><td>VARIANT C</td><td colspan="12">1.00× ∆AccQ ∆AccG ∆AccQ ∆AccG ∆AccQ ∆AccG ∆AccQ ∆AccG ∆AccQ ∆AccG ∆AccQ ∆AccG 1.25× 1.50× 1.75× 2.00× AVERAGE</td></tr><tr><td>4 BIT DWCONV</td><td>-1.54</td><td>+2.61</td><td>-2.76</td><td>+2.80</td><td>-1.77</td><td>+1.74</td><td>-1.82</td><td>+1.64</td><td>-1.58</td><td>+1.55</td><td>-1.89</td><td>+2.07</td></tr><tr><td>QUANT. ALL</td><td>-8.60</td><td>+4.39</td><td>-7.60</td><td>+3.41</td><td>-7.74</td><td>+3.19</td><td>-8.61</td><td>+4.09</td><td>-7.49</td><td>+2.25</td><td>-8.01</td><td>+3.47</td></tr></table>

Analysis of the impact of d with ∆Acc Q and ∆Acc G for VGG variant C. ∆Acc Q

Table content: <table/>

ImageNet results for DualPrecision. All the activations are quantized to 8 bit. We note that the results for 8 bit are iso-accurate with their full-precision counterparts, which makes 8 bit a strong baseline.

Table content: <table><tr><td>NETWORKS</td><td>METHODS</td><td colspan="6">TOP-1 (%) Csize (10 6 ) TOP-1 (%) Csize (10 6 ) TOP-1 (%) Csize (10 6 )</td></tr><tr><td/><td>8 BIT</td><td>71.11</td><td>63.85</td><td>74.86</td><td>102.87</td><td>76.70</td><td>204.18</td></tr><tr><td>RESNET50</td><td>FLEXIBLE (WANG ET AL. (2019))</td><td>74.30</td><td>63.60</td><td>76.04</td><td>102.90</td><td>77.23</td><td>204.18</td></tr><tr><td/><td>DUALPRECISION (OURS)</td><td>75.44</td><td>63.13</td><td>76.70</td><td>102.83</td><td>77.58</td><td>204.08</td></tr><tr><td/><td>8 BIT</td><td>52.17</td><td>11.99</td><td>64.39</td><td>15.44</td><td>71.73</td><td>27.50</td></tr><tr><td>MOBILENETV2</td><td>FLEXIBLE (WANG ET AL. (2019))</td><td>55.20</td><td>12.10</td><td>65.00</td><td>15.54</td><td>72.13</td><td>27.71</td></tr><tr><td/><td>DUALPRECISION (OURS)</td><td>59.51</td><td>12.15</td><td>68.01</td><td>15.53</td><td>73.91</td><td>27.56</td></tr></table>

ResNet20 to ResNet56

Table content: <table><tr><td>LAYERS</td><td>20</td><td>26</td><td>32</td><td>38</td><td>44</td><td>50</td><td>56</td></tr><tr><td>STEM</td><td colspan="3">CONV2D (16,3,3) STRIDE 1</td><td/><td/><td/><td/></tr><tr><td>STAGE 1 3 ×</td><td>CONV2D(16, 3, 3) STRIDE 1 CONV2D(16, 3, 3) STRIDE 1</td><td colspan="6">4× 5× 6× 7× 8× 9×</td></tr><tr><td>STAGE 2 3 ×</td><td>CONV2D(32, 3, 3) STRIDE 2 CONV2D(32, 3, 3) STRIDE 1</td><td colspan="6">4× 5× 6× 7× 8× 9×</td></tr><tr><td>STAGE 3 3 ×</td><td>CONV2D(64, 3, 3) STRIDE 2 CONV2D(64, 3, 3) STRIDE 1</td><td colspan="6">4× 5× 6× 7× 8× 9×</td></tr></table>



Table content: <table><tr><td/><td/><td>: Inv-ResNet26</td></tr><tr><td>STEM</td><td/><td>CONV2D (16,3,3) STRIDE 1</td></tr><tr><td>STAGE 1 4 ×</td><td>  </td><td>CONV2D(16 × 6, 1, 1) STRIDE 1 DWCONV2D(16 × 6, 3, 3) STRIDE 1</td></tr><tr><td/><td> </td><td>CONV2D(16, 1, 1) STRIDE 1</td></tr><tr><td>STAGE 2 4 ×</td><td>  </td><td>CONV2D(32 × 6, 1, 1) STRIDE 1 DWCONV2D(32 × 6, 3, 3) STRIDE 2</td></tr><tr><td/><td> </td><td>CONV2D(32, 1, 1) STRIDE 1</td></tr><tr><td>STAGE 3 4 ×</td><td>  </td><td>CONV2D(64 × 6, 1, 1) STRIDE 1 DWCONV2D(64 × 6, 3, 3) STRIDE 2</td></tr><tr><td/><td> </td><td>CONV2D(64, 1, 1) STRIDE 1</td></tr><tr><td colspan="3">D PROOF FOR PROPOSITION 5.1</td></tr></table>

Grid search of DualPrecision for MobileNetV2 and ResNet50 with the model size aligned to the 0.25× 8 bit models. Each cell reports the top-1 accuracy of the corresponding model on ImageNet.

Table content: <table><tr><td>PRECISION FOR</td><td/><td colspan="2">MOBILENETV2</td><td/><td>RESNET50</td></tr><tr><td>CONVS \ DWCONVS</td><td>8 BIT</td><td>4 BIT</td><td>2 BIT</td><td>1 BIT</td><td>NONE</td></tr><tr><td>8 BIT</td><td colspan="4">52.17 53.89 50.51 48.78</td><td>71.11</td></tr><tr><td>4 BIT</td><td colspan="4">56.84 59.51 57.37 55.91</td><td>74.65</td></tr><tr><td>2 BIT</td><td colspan="4">53.89 57.10 55.26 54.04</td><td>75.12</td></tr><tr><td>1 BIT</td><td colspan="4">54.82 58.16 56.90 55.82</td><td>75.44</td></tr></table>

Peer review (iclr2020):
This paper studies the accuracy vs model-size trade-off of quantized CNNs under different channel width multipliers. The authors demonstrated that while all-to-all convolution works well under low bit settings, depthwise conv needs a different sweet spot. The authors then proceed to use the insight to design quantized cnns that have two different schemes for depthwise and normal conv. Strength The paper is well written and motivated. By adding network width to the search space, using the simple heuristics, the authors provide a better results than previously DRL based search method. Weakness One of my main concerns is the direct usage of total number of bits as an equivalent measurement between models. While it is useful to measure the storage cost for weights. The choices of bit width will likely affect the computing cost in a non-trivial way, depending on the target hardware platform. It is unclear whether equal model size would mean equal inference latency in practice (most likely they would not be). Providing empirical implementations of these models will shed light into this question. These weakness makes it a borderline paper. Question: How do you handle batchnorm layer, do you use floating points? How many bits did you use for accumulating the results? Update after rebuttal: I have read the authors* response. I would like to keep my current review as it is. Also note that the authors uses floating pt for batchnorm and 32 bit for accumulation, such additional cost might out-weights the benefit of choosing ultra low bits in the low bits regime, making the study less relevant for practice reasons