CURSOR-BASED ADAPTIVE QUANTIZATION FOR DEEP NEURAL NETWORK

1 INTRODUCTION . Deep learning ( DL ) has achieved great successes in varied fields such as gaming , natural language processing , speech recognition , computer vision and so on . However , its huge computational burden and large memory consumption still intimidate many potential applications , especially for mobile devices and embedded systems . A number of efforts have been devoted to compress the DL model size and accelerate its training and test speed . These efforts can be roughly categorized into four major classes : network pruning ( Han et al . ( 2015 ) ; Anwar et al . ( 2015 ) ; Peng et al . ( 2019 ) ; Zhuang et al . ( 2018 ) ) , low rank approximation ( Tai et al . ( 2015 ) ; Wang et al . ( 2018 ) ; Hayashi et al . ( 2019 ) ) , knowledge distillation ( Hinton et al . ( 2015 ) ; Zagoruyko & Komodakis ( 2016 ) ) , and network quantization ( Courbariaux & Bengio ( 2016 ) ; Lin et al . ( 2015 ) ; Wu et al . ( 2015 ) ; Polino et al . ( 2018 ) ; Zhang et al . ( 2018 ) ) . Among them , network quantization methods , jointly optimizing the whole network weights , activations or gradients with low bit ( such as 8 bits or even 1 bit ) , show great potential in compressing model size and accelerating inference time . In addition , quantization based approaches are preferable for mobile devices and embedded systems since these devices are gradually equipped by specifically designed low bit computing hardware . Although existing quantization based approaches , which mainly use one fixed bit to represent the whole DNN model , yields encouraging compression ratio while keeping the model ’ s performance , we argue that simply using only a fixed bit for quantization is not the optimal choice for the trade-off between a model size and its performance . For example , to run a model on chips with strict memory limitations , 1 bit or 2 bits ’ quantization suffers from severe accuracy loss ( Rastegari et al . ( 2016 ) ) while 16 bits ’ or 8 bits ’ quantization can not significantly reduce the model size . To address the above problem , we propose a cursor based adaptive quantization method to derive a different number of bits in different layers for DNN model compression , i.e. , we search for the best configuration of different bit quantization for different layers in a neural network model . Distinctive from most algorithms aforementioned , our approach is motivated by recent neural architecture search ( NAS ) that aims to find better performance neural architecture with less calculations or less size automatically . The key in our algorithm is using a continuous cursor that represents the bit quantization scheme for each layer . For different layers , many cursors will be adaptively searched during the NAS process . Since the cursor itself is continuous and the whole search procedure can be considered as a differentiable architecture search ( DAS ) process , which can be effectively solved based on an alternative optimization strategy . A novel cost function that considers the model compression and prediction accuracy is also proposed in the DAS process . In the searching process of the cursors , a quantization process is applied to compress the model size at the same time . To reduce the possible quantization noise and local convergence problem , we make use of the closest two integer bits to the cursor to quantize the weights for each layer in a DNN model . We validate our proposed method with image classification tasks on CIFAR10 , CIFAR100 and ImageNet . Comprehensive experiments on some backbone DNN models such as ResNet20 , ResNet18 , ResNet56 and MobileNetV2 show that the proposed cursor based quantization method achieves remarkably better performance of compression ratio with ignorable accuracy drop or even better accuracy . In summary , the contributions of this work are four-fold : • We cast the adaptive quantization of neural network as a problem of neural architecture search . A continuous cursor is proposed to represent the possible quantization bit , leading to a more efficient search space . • A novel regularization function is proposed to optimize model compression in the search process . Thus the search for the cursor position and weights can be efficiently solved in an alternative optimization manner . • Two nearest neighbor integers to the cursor are adopted with a carefully designed strategy to implement the quantization of the network to reduce the quantization noise and avoid possible local convergence . • We comprehensively evaluate the proposed adaptive quantization method on some benchmark datasets and achieve new state-of-the-art performance for different number of bits quantization of neural network . 2 RELATED WORK . Recently , a lot of new quantization approaches have been proposed , enabling the quantized compressed model to compete with their full precision counterparts . Clustering method is applied for the weight codebook representation ( Han et al . ( 2015 ) ) , and then the network is retrained to get better quantized centroids . In ( Zhang et al . ( 2018 ) ) , the authors jointly trained a DNN and its associated quantizes to reduce the noticeable predication accuracy gap between the quantized model and its full precision one . A direct differentiable quantization method was introduced in ( Louizos et al . ( 2018 ) ) with promising test accuracy . A new activation quantization method that takes an activation clipping parameter was proposed in ( Choi et al . ( 2018 ) ) to ensure the suitable quantization scale . Some efforts have been taken on quantization of the neural network with a different number of bits used for different layers . In ( Lin et al . ( 2015 ) ) , signal-to-quantization-noise ratio ( SQNR ) is applied on layer weight to evaluate the effects of quantization error . Based on SQNR , different bits were used for quantization of each layer , yielding about 20 % model size reduction without accuracy loss in their tests . The authors of ( Wang et al . ( 2018 ) ) proposed an automated mixed precision quantization scheme based on reinforcement learning ( RL ) to achieve better latency on different hardware platforms such as edge devices and cloud data center . They also claimed that their actorcritic model produced efficient actions that result in better latency and less energy consumption with negligible loss of accuracy . In the past few years , a new trend has been witnessed for network design , i.e. , neural architecture search ( NAS ) . RL based approaches are first utilized to generate network with high accuracy ( Zoph & Le ( 2016 ) ) , and they also build a strong basis for the following recent works such as ( Gao et al . ( 2019 ) ; Guo et al . ( 2018 ) ) . Then , evolution based approach ( Liang et al . ( 2018 ) is further applied to obtain the possible optimal solution in the large search space . Both of these two category approaches tend to yield large amount of computational burden because NAS is treated as a blackbox optimization problem in a discrete domain , yielding a large number of architecture evaluations , and thus run very slow even on the most advanced GPU machine . To alleviate this problem , in 2018 , the authors ( Liu et al . ( 2018 ) ) proposed a differentiable approach to accelerate the search of a good neural network by relaxation of the possible operation on the cell level structure . Wu et al . recently proposed a new approach to find the mixed bits for different layers by applying differentiable NAS ( DNAS ) method based on a model of super net ( Wu et al . ( 2018 ) ) , which is a kind of directed acyclic graph . They considered the quantization as a problem of sampling on a stochastic super net , where a Gumbel softmax function is applied to make the sampling process differentiable . We cast the different bit quantization for DNN as a cursor based adaptive architecture search problem , and it is different from the traditional direct quantization works and the learning based mixed bits ’ quantization approaches . Moreover , it is also distinctive from DARTs and DNAS in the methodology itself . 3 CURSOR-BASED ADAPTIVE QUANTIZATION . 3.1 NEURAL ARCHITECTURE SEARCH . It is well known that DNN model needs much time to design its structure . Neural architecture search ( NAS ) recently emerged as a new methodology to overcome this challenge . It designs the optimal architecture of a neural network by considering all possible factors such as number of layers , width of each layer , different operators in each layer and so on . Two key concepts are related to a NAS process , i.e. , search space and search strategy . All the possible combinations of the major factors that determine a network structure constitute the search space . Generally speaking , the search space of a DNN is very large , leading to a huge computational task . As such , the previous NAS works first devise normal and reduction cell ( Pham et al . ( 2018 ) ) . Such kind of motif is then repeated to build the final neural network . Another definition is about search strategy , that is , how to transverse in such a large search space . With each searched network structure , the performance of it will be evaluated . A typical search method is random search , however , its computational efficiency is not ideal . Therefore , most recent works ( Cai et al . ( 2018 ) ; Liu et al . ( 2018 ) ) have been proposed along this big direction to improve the search efficiency as much as possible . 3.2 SEARCH SPACE FOR QUANTIZATION PROBLEM . Quantization has also been a very hot research topic in the past few years . Rounding function , vector quantization or stochastic function are typically applied to implement quantization to compact the model size while maintaining equivalent performance or acceptable loss . Some other approaches also use stochastic or probabilistic methods to quantize the neural network . Most previous methods simply apply one kind of bit quantization to the whole network due to the simplicity of implementation . A few recent works begin to utilize different bit quantization scheme to further improve the compression ratio and prediction accuracy . If we consider quantization choice as a part of the neural architecture , we can estimate its corresponding search space . Let us take Resent20 as an example and if we decide to quantize the neural network with the possible bit width of 1 , 2 , 4 , 8 , 16 , 32 , then all the possible quantization choices for ResNet20 will be 620 . In the context of NAS , this is a very large number for the search space . Hence , evaluation of so many designs one by one seems infeasible right now . 3.3 DIFFERENTIABLE CURSOR SEARCH FOR ADAPTIVE QUANTIZATION . The discrete search space of the above quantization scheme is so large . If we further consider the possible bit choice for each layer as a virtual continuous cursor in the range of [ 1 , 32 ] , the cursors then become significant parts of the architecture for a neural network model . Here , we define the cursor as a position that is related to the quantization choice for each layer . Its value is a floatingpoint number within [ 1 , 32 ] . If we assume a DNN has N layers , and each layer has a different cursor , denoted by c1 , c2 , ... , cN , together with their weights of WC , our goal is to find a good combination of c1 , c2 , ... , cN to achieve better prediction accuracy and compression ratio . As such , for the whole neural network it can be described as an optimization problem that minimizes the loss on the validation data after training through the minimization of the loss on the training data as follows ( Liu et al . ( 2018 ) ) : minE ( x′ , y′ ) DT ( LossT ( C , WC ) ) s.t . WC∗ = argminE ( x , y ) DV ( LossV ( C , WC ) ) ( 1 ) where C represents the cursor vector , WC∗ is the weights corresponding to the optimal C , LossT ( C , WC ) and LossV ( C , WC ) is the respective training and validation loss function based on the cursors and weights with condition of C. DT and DV represents the training and validation dataset respectively , ( x , y ) and ( x′ , y′ ) means data from the training and validation dataset . It should be noted that using training and validation data is a tradition to derive the weight parameters and architecture parameters in the field of NAS , which is a little bit different from the other problems in deep learning . For simplicity and efficiency , in this paper , we let LossT ( C , WC ) equals to LossV ( C , WC ) and assume they share the same form of Loss ( C , WC ) . To consider both the prediction accuracy and model size , we design the loss function as a combination of cross entropy and model size as follows : Loss ( C , WC ) = CrossEntropy ( C , WC ) + λ× LossC ( 2 ) where CrossEntropy ( C , WC ) is the widely used cross entropy function , encoding the prediction accuracy of the model . The reason why we add a regularization item to the loss function is first because it may determines the model ’ s performance compromise between the accuracy and quantization , which directly determines the model size , and λ is a regularization coefficient that adjusts the trade-off of accuracy and compression . In addition , it may prevent overfitting to some extent . Concerning the loss related to LossC , we focus on the model size change with quantization . So we design it in the form of Eq . ( 3 ) , which will be introduced in details in the next subsection . The above process is a bi-level optimization problem , which requires to deduce higher order derivatives and is hard to obtain an exact solution . An approximated iterative solution can be applied instead , so we alternatively take the optimization strategy in weight and cursor space to update WC based on the training losses fromDT and renew C based on the validation losses fromDV . By solving this bi-level optimization problem using an alternative approximation approach , the cursors can be efficiently searched by gradient based optimization approach such as Adam . Our later experimental results also show that this alternative optimization method may yield a good solution with high compression ratio and accuracy . Compared to the original discrete search space , this search method is more efficient because the design of continuous cursor and the direct gradient based optimization approach . The whole differentiable cursor search for adaptive quantization based on the alternative optimization of WC and C is described in Algorithm 1 . With the training and validation set , initialized cursor value and a pretrained model as inputs , our algorithm first quantizes the weights in each layer of the network using the two most close integers to the cursor and calculates the loss based on the training data for forward process , and then it updates the original 32 bit weight by gradient descent algorithm . For the subsequent validation , it also first quantizes the network with the two most close integers to the cursor , and use them to obtain the validation error , then our algorithm utilizes this error to update the cursors with gradient descend algorithm . It should be emphasized that we utilize the original 32 bit weight in the above training and validation step , and use it for sharing when implementing quantization with the cursor ’ s two neighbor integers . The key step in the algorithm about quantization with the nearest two integers to the cursor will be elaborated in the subsequent section . The outputs of the whole algorithm are rounded cursor values for each layer and its quantized network model . It should be noted that our proposed cursor based differentiable search is different from DARTs ( Liu et al . ( 2018 ) ) in the following three aspects . First , DARTs method considers the possible operation in each layer as a mixture of primitive operations . Here , we directly make use of cursor to represent the quantization bit for each layer , no similar mixture operation exists in the whole search algorithm . Second , in DARTs , each primitive operation is assigned with a probability through a softmax function . Cursor based search is optimized directly without probability . Third , DARTs approach concentrates on the cell structure , but we apply the DAS directly on the whole network . Compared to DNAS ( Wu et al . ( 2018 ) ) , our approach is also distinctive . For DNAS , the authors build a stochastic super net first to describe all the possible quantization choices , then a sampling approach based on Gumbel softmax function that enables the discrete distribution to be continuous and differentiable is applied in each layer of the super net . Our cursor based differentiable search has no super net or sampling process in the pipeline . Hence , the subsequent solutions to the optimization problem is also completely different . In short , the proposed method requires no relaxation anymore as in both DARTs and DNAS approach . input : The training set DT , validation set DV , initialized cursors C , pretrained 32-bit weight W , and the batch size n while not reaching the target epochs or not converge do Sample data from training data DT ; Quantize the network using two integers that are closest to the cursor , calculate the loss L on training data with Eq . ( 2 ) ; Update the weight W by gradient descent W ′ =W −∇W L ( C , W ) ; Sample data from validation data DV ; Quantize the network using two integers that are closest to the cursor , calculate the loss L on validation data with Eq . ( 2 ) ; Update the cursor C by gradient descent C ′ = C −∇C L ( C , W ) ; end output : Rounded cursor values for each layer and quantized network Algorithm 1 : Differentiable Cursor Search for Adaptive Quantization

Summary: The authors developed a novel quantization technique that yields layer-wise different mixed-precision quantization. To do so, they alternatively update the pre-trained weights and the quantizer, which they call cursor. The following two features distinguish this paper: using two precision values around the cursor's value (instead of the closest one) and regularizing the parameter size using a new loss function. Because the whole process is differentiable, the appropriate precision to each layer can be found fast. Thanks to these efforts, this method balances the compression rate and accuracy well on CIFAR-10 and ImageNet.