On the Pareto Efficiency of Quantized CNN

1 INTRODUCTION . Recent success of convolutional neural networks ( CNNs ) in computer vision applications such as image classification and semantic segmentation , have fueled many important applications in energyconstrained devices , e.g. , virtual reality headsets , drones , and robots . As a result , improving the energy-efficiency of CNNs while maintaining their attractive features ( e.g. , accuracy for a task ) has gained tremendous research momentum in recent years . Among the efforts of improving CNNs ’ efficiency , weight quantization was shown to be an effective technique ( Zhou et al . ( 2016 ; 2017 ) ; Hou & Kwok ( 2018 ) ; Ding et al . ( 2019 ) ) . The majority of research efforts in quantization have been devoted to develop better quantization algorithms such that an iso-figure-of-merit ( i.e. , accuracy ) is achieved with lowest possible weight precision value . Nevertheless , the iso-accurate precision value depends on the dataset , task , and network architecture of interest , which greatly increases the neural network implementation complexity from both hardware and software perspectives . For example , hardware and software implementations optimized for executing an 8 bit network are sub-optimal for executing a 4 bit network , and vice versa . The design optimization complexity further increases for recently proposed mixed-precision networks ( Wang et al . ( 2019 ) ; Wu et al . ( 2018b ) ; Dong et al . ( 2019 ) ) . The key observation we have is that most prior literature in this space studies quantization for fixed network architectures , which is reasonable for evaluating the effectiveness of quantization algorithms , but unnecessary when considering the Pareto efficiency ( accuracy vs. model size ) of neural networks . In this work , we relax the restriction of fixing the network architecture and allow the number of channels of the CNN under consideration to vary . More concretely , we use the widthmultiplier1 ( Howard et al . ( 2017 ) ) as a tool to compare the performance of different weight precision values under the same model size . Overall , we systematically analyze the model size and accuracy trade-offs considering both weight precision values and the number of channels for various modern networks architectures ( variants of 1Width-multiplier grows or shrinks the number of channels across the layers with identical proportion for a certain network , e.g. , grow the number of channels for all the layers by 2× . ResNet , VGG , and MobileNet ) and datasets ( CIFAR and ImageNet ) and have the following nontrivial and novel contributions : • We are the first to empirically show that when considering channel counts , lower precision weight values outperform higher precision weight values in a Pareto sense ( accuracy vs. model size ) for networks with standard convolutions . This is intriguing since it implies that scaling up ( in terms of model size ) along the channel count dimension is more effective for accuracy than the precision value dimension . • We are the first to show that the fan-in channel counts per output filter for a convolution layer determine the effectiveness of accuracy improvement when the model is scaled along the weight precision dimension and provide both theoretical and empirical reasoning for this . • We are the first to show that with a simple model scaling rule ( the proposed DualPrecision ) , one can achieve a more accurate model ( given the same model size ) even compared to mixed-precision prior art that uses deep reinforcement learning to search for layer-wise weight precision values . Moreover , the results are validated on the large-scale dataset , i.e. , ImageNet . This is a manifestation of our two previous findings . The remainder of the paper is organized as follows . Section 2 discusses related work . Section 3 discusses the methodology used to discover our findings . Section 4 shows lower precision values are preferable for networks with standard convolutions . Section 5 discusses how fan-in channel count per output filter affects precision scaling for convolution layers . Section 6 discusses DualPrecision , our simple yet effective model scaling rule . Section 7 concludes the paper . 2 RELATED WORK . Several techniques for improving the efficiency of CNNs have been recently proposed . For instances , pruning removes the redundant connections of a trained neural network ( Zhuang et al . ( 2018 ) ; Ye et al . ( 2018 ) ; Theis et al . ( 2018 ) ; Li et al . ( 2017 ) ; Frankle & Carbin ( 2019 ) ; Chin et al . ( 2019 ) ; Yu et al . ( 2018 ) ) , neural architecture search ( NAS ) tunes the number of channels , size of kernels , and depth of a network ( Tan et al . ( 2018 ) ; Stamoulis et al . ( 2019 ) ; Cai et al . ( 2018 ) ; Stamoulis et al . ( 2018 ) ) , and convolution operations can be made more efficient via depth-wise convolutions ( Howard et al . ( 2017 ) ) , group convolutions ( Huang et al . ( 2017 ) ; Zhao et al . ( 2019b ) ) , and shift-based convolutions ( He et al . ( 2019 ) ; Wu et al . ( 2018a ) ) . In addition to the aforementioned techniques , network quantization introduces an opportunity for hardware-software co-design to achieve better efficiency for CNNs . There are in general two directions for weight quantization in prior literature , post-training quantization ( Nagel et al . ( 2019 ) ; Meller et al . ( 2019 ) ; Zhao et al . ( 2019a ) ; Sheng et al . ( 2018 ) ) and quantization-aware training ( Rastegari et al . ( 2016 ) ; Zhu et al . ( 2017 ) ; Jacob et al . ( 2018 ) ; Jung et al . ( 2019 ) ; Yuan et al . ( 2019 ) ; Hou & Kwok ( 2018 ) ; Choi et al . ( 2018 ) ) . The former assumes training data is not available when quantization is applied . While being fast and training-data-free , its performance is worse compared to quantization-aware training . In contrast , our work falls under the category of quantization-aware training . In quantization-aware training , ( Rastegari et al . ( 2016 ) ) introduces binary neural networks , which lead to significant efficiency gain by replacing multiplications with XNOR operations , but at the expense of significant accuracy degradation . Later , ( Zhu et al . ( 2017 ) ) propose ternary quantization and ( Zhou et al . ( 2016 ) ; Jacob et al . ( 2018 ) ) bridge the gap between floating-point networks and binarized neural networks by introducing fixed-point quantization . Building upon prior art , the vast majority of existing work focuses on reducing the accuracy degradation by improving the training strategy ( Zhou et al . ( 2017 ) ; Yang et al . ( 2019 ) ; Louizos et al . ( 2019 ) ; Ding et al . ( 2019 ) ) and better quantization schemes ( Jung et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Yuan et al . ( 2019 ) ) . However , prior art studies quantization by fixing the network architecture , which may lead to sub-optimal precision decisions in terms of Pareto efficiency ( model size vs. accuracy ) . Related to our work , Mishra et al . ( 2018 ) have also considered the impact of channel count in quantization . In contrast , our work has the following novel features . First , we find that in CNNs with standard convolutions , lower precision values outperform higher ones in a Pareto sense . Second , we find that the Pareto optimal precision value depends on the number of input channels per output filter and provide theoretical insights for it . Last , we propose an algorithm to select the precision values of a given network which as a result outperforms 8 bit fixed-point and mixed-precision baselines . 3 METHODOLOGY . We conduct all of our experiments on image classification datasets including CIFAR-100 and ImageNet . All the experiments are trained from scratch to ensure different precision values are trained equally long . While we do not start from a pre-trained model , we note that our baseline fixedpoint models ( i.e. , 4 bit for CIFAR and 8 bit for ImageNet ) achieve iso-accurate results compared to their floating-point counterparts . For all the experiments on CIFAR , we run the experiments three times and report the mean and standard deviation . The training hyper-parameters are detailed in Appendix A . 3.1 QUANTIZATION . While our work focuses on weight quantization , we still quantize the activations since they are normally quantized for efficient deployment ( Jacob et al . ( 2018 ) ) . For activation quantization , we use the technique proposed in prior art ( Jacob et al . ( 2018 ) ) and use 4 bit for CIFAR-100 and 8 bit for ImageNet experiments . We note that the precision value is chosen such that iso-accurate results can be achieved when compared to the floating-point baselines . For weight quantization , we use a straight-through estimator ( Bengio et al . ( 2013 ) ) to conduct quantization-aware training . Specifically , for precision beyond 2 bit ( b > 2 ) , we use the following quantization function for weights during the forward pass : Q ( Wi , : ) = b clamp ( Wi , : , −ai , ai ) si e × si , si = ai 2b−1 − 1 , ( 1 ) where b·e denotes the round-to-nearest-neighbor function , W ∈ RCout×d , d = CinKwKh denotes the real-value weights for the ith output filter of a convolution layer that has Cin channels and Kw × Kh kernel size . a ∈ RCout denotes the vector of clipping factors which are selected to minimize ‖Q ( Wi , : ) −Wi , :‖22 by assuming Wi , : ∼ N ( 0 , σ2I ) . More details about the determination of ai is in Appendix B . For special cases such as 2 bit and 1 bit , we use schemes proposed in prior art . Specifically , let us first define : ¯|Wi , :| = 1 d d∑ j=1 |Wi , j | . ( 2 ) For 2 bit precision , we follow trained ternary networks ( Zhu et al . ( 2017 ) ) and define the quantization function as follows : Q ( Wi , : ) = ( sign ( Wi , : ) Mi , j ) × ( ¯|Wi , :| ) Mi , j = { 0 , Wi , j < 0.7 ¯|Wi , :| . 1 , otherwise . ( 3 ) For 1 bit precision , we follow DoReFaNets ( Zhou et al . ( 2016 ) ) and define the quantization function as follows : Q ( Wi , : ) = sign ( Wi , : ) × ( ¯|Wi , :| ) . ( 4 ) For the backward pass for all the precision values , we use a straight-through estimator as in prior art to make the training differentiable . That is , Q ( Wi , : ) ∂Wi , : = I . ( 5 ) In the sequel , we quantize the first and last layers to 8 bit . They are fixed throughout the experiments . We note that it is a common practice to leave the first and the last layer un-quantized ( Zhou et al . ( 2016 ) ) , however , we find that using 8 bit can achieve iso-accurate results .

Summary: This paper studies the accuracy vs model-size trade-off of quantized CNNs under different channel width multipliers. The authors demonstrated that while all-to-all convolution works well under low bit settings, depthwise conv needs a different sweet spot. The authors then proceed to use the insight to design quantized cnns that have two different schemes for depthwise and normal conv.