On the Pareto Efficiency of Quantized CNN

1 INTRODUCTION . Recent success of convolutional neural networks ( CNNs ) in computer vision applications such as image classification and semantic segmentation , have fueled many important applications in energyconstrained devices , e.g. , virtual reality headsets , drones , and robots . As a result , improving the energy-efficiency of CNNs while maintaining their attractive features ( e.g. , accuracy for a task ) has gained tremendous research momentum in recent years . Among the efforts of improving CNNs ’ efficiency , weight quantization was shown to be an effective technique ( Zhou et al . ( 2016 ; 2017 ) ; Hou & Kwok ( 2018 ) ; Ding et al . ( 2019 ) ) . The majority of research efforts in quantization have been devoted to develop better quantization algorithms such that an iso-figure-of-merit ( i.e. , accuracy ) is achieved with lowest possible weight precision value . Nevertheless , the iso-accurate precision value depends on the dataset , task , and network architecture of interest , which greatly increases the neural network implementation complexity from both hardware and software perspectives . For example , hardware and software implementations optimized for executing an 8 bit network are sub-optimal for executing a 4 bit network , and vice versa . The design optimization complexity further increases for recently proposed mixed-precision networks ( Wang et al . ( 2019 ) ; Wu et al . ( 2018b ) ; Dong et al . ( 2019 ) ) . The key observation we have is that most prior literature in this space studies quantization for fixed network architectures , which is reasonable for evaluating the effectiveness of quantization algorithms , but unnecessary when considering the Pareto efficiency ( accuracy vs. model size ) of neural networks . In this work , we relax the restriction of fixing the network architecture and allow the number of channels of the CNN under consideration to vary . More concretely , we use the widthmultiplier1 ( Howard et al . ( 2017 ) ) as a tool to compare the performance of different weight precision values under the same model size . Overall , we systematically analyze the model size and accuracy trade-offs considering both weight precision values and the number of channels for various modern networks architectures ( variants of 1Width-multiplier grows or shrinks the number of channels across the layers with identical proportion for a certain network , e.g. , grow the number of channels for all the layers by 2× . ResNet , VGG , and MobileNet ) and datasets ( CIFAR and ImageNet ) and have the following nontrivial and novel contributions : • We are the first to empirically show that when considering channel counts , lower precision weight values outperform higher precision weight values in a Pareto sense ( accuracy vs. model size ) for networks with standard convolutions . This is intriguing since it implies that scaling up ( in terms of model size ) along the channel count dimension is more effective for accuracy than the precision value dimension . • We are the first to show that the fan-in channel counts per output filter for a convolution layer determine the effectiveness of accuracy improvement when the model is scaled along the weight precision dimension and provide both theoretical and empirical reasoning for this . • We are the first to show that with a simple model scaling rule ( the proposed DualPrecision ) , one can achieve a more accurate model ( given the same model size ) even compared to mixed-precision prior art that uses deep reinforcement learning to search for layer-wise weight precision values . Moreover , the results are validated on the large-scale dataset , i.e. , ImageNet . This is a manifestation of our two previous findings . The remainder of the paper is organized as follows . Section 2 discusses related work . Section 3 discusses the methodology used to discover our findings . Section 4 shows lower precision values are preferable for networks with standard convolutions . Section 5 discusses how fan-in channel count per output filter affects precision scaling for convolution layers . Section 6 discusses DualPrecision , our simple yet effective model scaling rule . Section 7 concludes the paper . 2 RELATED WORK . Several techniques for improving the efficiency of CNNs have been recently proposed . For instances , pruning removes the redundant connections of a trained neural network ( Zhuang et al . ( 2018 ) ; Ye et al . ( 2018 ) ; Theis et al . ( 2018 ) ; Li et al . ( 2017 ) ; Frankle & Carbin ( 2019 ) ; Chin et al . ( 2019 ) ; Yu et al . ( 2018 ) ) , neural architecture search ( NAS ) tunes the number of channels , size of kernels , and depth of a network ( Tan et al . ( 2018 ) ; Stamoulis et al . ( 2019 ) ; Cai et al . ( 2018 ) ; Stamoulis et al . ( 2018 ) ) , and convolution operations can be made more efficient via depth-wise convolutions ( Howard et al . ( 2017 ) ) , group convolutions ( Huang et al . ( 2017 ) ; Zhao et al . ( 2019b ) ) , and shift-based convolutions ( He et al . ( 2019 ) ; Wu et al . ( 2018a ) ) . In addition to the aforementioned techniques , network quantization introduces an opportunity for hardware-software co-design to achieve better efficiency for CNNs . There are in general two directions for weight quantization in prior literature , post-training quantization ( Nagel et al . ( 2019 ) ; Meller et al . ( 2019 ) ; Zhao et al . ( 2019a ) ; Sheng et al . ( 2018 ) ) and quantization-aware training ( Rastegari et al . ( 2016 ) ; Zhu et al . ( 2017 ) ; Jacob et al . ( 2018 ) ; Jung et al . ( 2019 ) ; Yuan et al . ( 2019 ) ; Hou & Kwok ( 2018 ) ; Choi et al . ( 2018 ) ) . The former assumes training data is not available when quantization is applied . While being fast and training-data-free , its performance is worse compared to quantization-aware training . In contrast , our work falls under the category of quantization-aware training . In quantization-aware training , ( Rastegari et al . ( 2016 ) ) introduces binary neural networks , which lead to significant efficiency gain by replacing multiplications with XNOR operations , but at the expense of significant accuracy degradation . Later , ( Zhu et al . ( 2017 ) ) propose ternary quantization and ( Zhou et al . ( 2016 ) ; Jacob et al . ( 2018 ) ) bridge the gap between floating-point networks and binarized neural networks by introducing fixed-point quantization . Building upon prior art , the vast majority of existing work focuses on reducing the accuracy degradation by improving the training strategy ( Zhou et al . ( 2017 ) ; Yang et al . ( 2019 ) ; Louizos et al . ( 2019 ) ; Ding et al . ( 2019 ) ) and better quantization schemes ( Jung et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Yuan et al . ( 2019 ) ) . However , prior art studies quantization by fixing the network architecture , which may lead to sub-optimal precision decisions in terms of Pareto efficiency ( model size vs. accuracy ) . Related to our work , Mishra et al . ( 2018 ) have also considered the impact of channel count in quantization . In contrast , our work has the following novel features . First , we find that in CNNs with standard convolutions , lower precision values outperform higher ones in a Pareto sense . Second , we find that the Pareto optimal precision value depends on the number of input channels per output filter and provide theoretical insights for it . Last , we propose an algorithm to select the precision values of a given network which as a result outperforms 8 bit fixed-point and mixed-precision baselines . 3 METHODOLOGY . We conduct all of our experiments on image classification datasets including CIFAR-100 and ImageNet . All the experiments are trained from scratch to ensure different precision values are trained equally long . While we do not start from a pre-trained model , we note that our baseline fixedpoint models ( i.e. , 4 bit for CIFAR and 8 bit for ImageNet ) achieve iso-accurate results compared to their floating-point counterparts . For all the experiments on CIFAR , we run the experiments three times and report the mean and standard deviation . The training hyper-parameters are detailed in Appendix A . 3.1 QUANTIZATION . While our work focuses on weight quantization , we still quantize the activations since they are normally quantized for efficient deployment ( Jacob et al . ( 2018 ) ) . For activation quantization , we use the technique proposed in prior art ( Jacob et al . ( 2018 ) ) and use 4 bit for CIFAR-100 and 8 bit for ImageNet experiments . We note that the precision value is chosen such that iso-accurate results can be achieved when compared to the floating-point baselines . For weight quantization , we use a straight-through estimator ( Bengio et al . ( 2013 ) ) to conduct quantization-aware training . Specifically , for precision beyond 2 bit ( b > 2 ) , we use the following quantization function for weights during the forward pass : Q ( Wi , : ) = b clamp ( Wi , : , −ai , ai ) si e × si , si = ai 2b−1 − 1 , ( 1 ) where b·e denotes the round-to-nearest-neighbor function , W ∈ RCout×d , d = CinKwKh denotes the real-value weights for the ith output filter of a convolution layer that has Cin channels and Kw × Kh kernel size . a ∈ RCout denotes the vector of clipping factors which are selected to minimize ‖Q ( Wi , : ) −Wi , :‖22 by assuming Wi , : ∼ N ( 0 , σ2I ) . More details about the determination of ai is in Appendix B . For special cases such as 2 bit and 1 bit , we use schemes proposed in prior art . Specifically , let us first define : ¯|Wi , :| = 1 d d∑ j=1 |Wi , j | . ( 2 ) For 2 bit precision , we follow trained ternary networks ( Zhu et al . ( 2017 ) ) and define the quantization function as follows : Q ( Wi , : ) = ( sign ( Wi , : ) Mi , j ) × ( ¯|Wi , :| ) Mi , j = { 0 , Wi , j < 0.7 ¯|Wi , :| . 1 , otherwise . ( 3 ) For 1 bit precision , we follow DoReFaNets ( Zhou et al . ( 2016 ) ) and define the quantization function as follows : Q ( Wi , : ) = sign ( Wi , : ) × ( ¯|Wi , :| ) . ( 4 ) For the backward pass for all the precision values , we use a straight-through estimator as in prior art to make the training differentiable . That is , Q ( Wi , : ) ∂Wi , : = I . ( 5 ) In the sequel , we quantize the first and last layers to 8 bit . They are fixed throughout the experiments . We note that it is a common practice to leave the first and the last layer un-quantized ( Zhou et al . ( 2016 ) ) , however , we find that using 8 bit can achieve iso-accurate results .

Summary: The author studies the quantization strategy of CNNs in terms of Pareto Efficiency. Through a series of experiments with three standard CNN models (ResNet, VGG11, MobileNetV2), the authors demonstrated that lower precision value can be better than high precision values in term of Pareto efficiency under the iso-model size scenario. They also study cases with and without depth-wise convolution, and propose a new quantization method, DualPrecision. DualPrecision empirically outperformed 8-bit quantization and flexible quantization methods on ImageNet.