Filter pre-pruning for improved fine-tuning of quantized deep neural networks

Deep Neural Networks ( DNNs ) have many parameters and activation data , and these both are expensive to implement . One method to reduce the size of the DNN is to quantize the pre-trained model by using a low-bit expression for weights and activations , using fine-tuning to recover the drop in accuracy . However , it is generally difficult to train neural networks which use low-bit expressions . One reason is that the weights in the middle layer of the DNN have a wide dynamic range and so when quantizing the wide dynamic range into a few bits , the step size becomes large , which leads to a large quantization error and finally a large degradation in accuracy . To solve this problem , this paper makes the following three contributions without using any additional learning parameters and hyperparameters . First , we analyze how batch normalization , which causes the aforementioned problem , disturbs the fine-tuning of the quantized DNN . Second , based on these results , we propose a new pruning method called Pruning for Quantization ( PfQ ) which removes the filters that disturb the fine-tuning of the DNN while not affecting the inferred result as far as possible . Third , we propose a workflow of fine-tuning for quantized DNNs using the proposed pruning method ( PfQ ) . Experiments using well-known models and datasets confirmed that the proposed method achieves higher performance with a similar model size than conventional quantization methods including fine-tuning . 1 INTRODUCTION . DNNs ( Deep Neural Networks ) greatly contribute to performance improvement in various tasks and their implementation in edge devices is required . On the other hand , a typical DNN ( He et al. , 2015 ; Simonyan & Zisserman , 2015 ) has the problem that the implementation cost is very large , and it is difficult to operate it on an edge device with limited resources . One approach to this problem is to reduce the implementation cost by quantizing the activations and weights in the DNN . Quantizing a DNN using extremely low bits , such as 1 or 2 bits has been studied by Courbariaux et al . ( 2015 ) and Gu et al . ( 2019 ) . However , it is known that while such a bit reduction has been performed for a large model such as ResNet ( He et al. , 2015 ) , it has not yet been performed for a small model such as MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) , and is very difficult to apply to this case . For models that are difficult to quantize , special processing for the DNN is required before quantization . On the other hand , although fine-tuning is essential for quantization with extremely low bit representation , few studies have been conducted on pre-processing for easy fine-tuning of quantized DNNs . In particular , it has been experimentally shown from previous works ( Lan et al. , 2019 ; Frankle & Carbin , 2018 ) that , regardless of quantization , some weights are unnecessary for learning or in fact disturb the learning process . Therefore , we focused on the possibility of the existence of weights that specially disturb the finetuning quantized DNN and to improve the performance of the quantized DNN after fine-tuning by removing those weights . 2 RELATED WORK AND PROBLEM . 2.1 CONVENTIONAL QUANTIZATION WORKS . In order to reduce the implementation cost of DNN in hardware , many methods of reduction by pruning ( Molchanov et al. , 2019 ; You et al. , 2019 ; He et al. , 2019 ) and quantization ( Gu et al. , 2019 ; Gong et al. , 2020 ; Fan et al. , 2020 ; Uhlich et al. , 2019 ; Sheng et al. , 2018 ; Nagel et al. , 2019 ) have been studied . Research is also underway in network architectures such as depthwise convolution and pointwise convolution ( Howard et al. , 2017 ) , which are arithmetic modules that maintain high performance even when both the number of operations and the model capacity are kept small ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) . Knowledge Transfer ( Hinton et al. , 2015 ; Zagoruyko & Komodakis , 2016 ; Radosavovic et al. , 2018 ; Chen et al. , 2019 ) is also being studied as a special learning method for achieving high performance in these compression technologies and compressed architectures . There have been various approaches to studies in DNN compression using quantization . DNN quantization is usually done for activations and weights in DNN . As a basic approach to quantization , there are some methods for a pre-trained model of which one uses only quantization ( Sheng et al. , 2018 ; Nagel et al. , 2019 ) and another uses fine-tuning ( Gong et al. , 2019 ; Gu et al. , 2019 ; Jung et al. , 2019 ; Uhlich et al. , 2019 ; Gong et al. , 2020 ; Fan et al. , 2020 ) . Cardinaux et al . ( 2020 ) and Han et al . ( 2015 ) propose a method for expressing values that are as precise as possible . However , the simplest expression method is easier to calculate and implement in hardware , so expressions to divide the range of quantized values by a linear value ( Jacob et al. , 2018 ; Sheng et al. , 2018 ; Nagel et al. , 2019 ; Gu et al. , 2019 ; Gong et al. , 2019 ; 2020 ; Jung et al. , 2019 ) or a power of 2 value ( Lee et al. , 2017 ; Li et al. , 2019 ) are widely used at present in quantization research . It is known that fine-tuning is essential to quantize DNN with low bits in order not to decrease accuracy as much as possible . Jacob et al . ( 2018 ) shows that an accuracy improvement can be expected by fine-tuning with quantization . Since the quantization function is usually non-differentiable , the gradient in the quantization function is often approximated by STE ( Bengio et al. , 2013 ) . However , in the case of quantization with low bits , the error is very large in the approximation by STE , and it is known that it is difficult to accurately propagate the gradient backward from the output side to the input side , making learning difficult . In order to solve this problem , Gong et al . ( 2019 ) , Fan et al . ( 2020 ) and Darabi et al . ( 2018 ) proposed the backward quantization function so that the gradient can be propagated to the input side as accurately as possible . Gu et al . ( 2019 ) and Banner et al . ( 2018 ) considered activations and weights in DNN as random variables , and proposed an approach to optimize the stochastic model behind DNN so that activations and weights in DNN become vectors which are easy to quantize . Jung et al . ( 2019 ) and Uhlich et al . ( 2019 ) considered that it is important for performance to determine the dynamic range of vectors to be quantized appropriately , and it is proposed that this dynamic range is also learned as a learning parameter . 2.2 PROBLEM OF WIDE DYNAMIC RANGE ON QUANTIZATION . It is important to set the dynamic range appropriately in the quantization . In modern network architectures , most of the computation is made up of a block of convolution and batch normalization ( BN ) . It is known that the above block can be replaced by the equivalent convolution , and quantization is performed after this replacement . Thus , it is possible to suppress the quantization error rather than quantizing the convolution and BN separately . On the other hand , it is also known that the dynamic range of the convolution may be increased by this replacement ( Sheng et al. , 2018 ; Nagel et al. , 2019 ) . In order to explain this in detail , the calculation formulas of convolution and BN in pre-training are described . First , the equation for convolution is described as follows C ( o , j , k ) = ∑ i ∑ ( u , v ) w ( o , i , u , v ) x ( i , j+u , k+v ) + b ( o ) , ( 1 ) where C ( o , j , k ) is the o-th convolution output feature at the coordinate ( j , k ) , w ( o , i , u , v ) is the weight of the kernel coordinates ( u , v ) in the i-th input map , and x ( i , j+u , k+v ) is the input data . For simplicity , C ( o , j , k ) is represented as vector C ( o ) = ( c ( o,1 ) , ... , c ( o , N ) ) 1 and w ( o , i , u , v ) is represented as vector w ( o , i ) if it is not necessary . Next , the calculation formula for BN is described as B ( o ) = C ( o ) − µ ( o ) √ ( σ ( o ) ) 2 + ǫ γ ( o ) + β ( o ) , ( 2 ) where B ( o ) is the o-th BN output feature 2. µ ( o ) , ( σ ( o ) ) 2 are respectively calculated by equation ( 3 ) , ( 4 ) during the training or by equation ( 7 ) , ( 8 ) during the inference . γ ( o ) , β ( o ) are respectively the scaling factor and shift amounts of BN . The mean and variance are calculated as µ ( o ) = 1 N N ∑ n c ( o , n ) ( 3 ) ( σ ( o ) ) 2 = 1 N N ∑ n ∣ ∣c ( o , n ) − µ ( o ) ∣ ∣ 2 . ( 4 ) At the time of inference or the fine-tuning of quantization for the pre-trained model , BN can be folded into convolution by inserting equation ( 1 ) into ( 2 ) . The new weight ( ŵ ( o , i , u , v ) ) and bias ( b̂ ( o ) ) of the convolution when BN is folded in the convolution are calculated by ŵ ( o , i ) = γ ( o ) √ V ( τ ) ( o ) + ǫ w ( o , i ) ( 5 ) b̂ ( o ) = γ ( o ) √ V ( τ ) ( o ) + ǫ b ( o ) + β ( o ) − γ ( o ) √ V ( τ ) ( o ) + ǫ M ( τ ) ( o ) , ( 6 ) where M ( τ ) ( o ) , V ( τ ) ( o ) are respectively the mean ( running mean ) and variance ( running variance ) of BN used at inference , and the values are calculated as M ( τ ) ( o ) = M ( τ−1 ) ( o ) ρ+ µ ( o ) ( 1− ρ ) ( 7 ) V ( τ ) ( o ) = V ( τ−1 ) ( o ) ρ+ ( σ ( o ) ) 2 ( 1− ρ ) N N − 1 , ( 8 ) where τ is an iteration index , and is omitted in this paper when not specifically needed , and ρ is a hyper-parameter of BN in pre-training so that 0 < ρ < 1 . The initial values of ( 7 ) and ( 8 ) are set before learning in the same way as weights such as convolution . Sheng et al . ( 2018 ) pointed out that the dynamic range of the weights of depthwise convolution is 1 N is the number of elements in a channel C ( o ) . 2Vector and scalar operations are performed by implicitly broadcasting scalars . large , which leads to the large quantization error . They also pointed out the relationship that the filters ( sets of weights ) with wide dynamic range correspond to the running variances of the BN with small magnitude . Nagel et al . ( 2019 ) attacked this problem by adjusting scales of weights of consecutive depthwise and pointwise convolutions without affecting the outputs of the pointwise convolution . However , they can not solve the problem in quantization completely . In this study , we theoretically analyze the weights with the running variances which have small magnitude and show that certain weights disturb the fine-tuning of the quantized DNN . Based on this analysis , we propose a new quantization training method which can solve the problem and improve the performance ( section 4 ) .

Summary: This paper studies the effect of quantization during training together with batch normalization in quantized deep neural networks. The compound effect of convolution and batch normalization on the dynamic range of activations has implications on the progress of training. The authors propose a protocol for training a quantized neural network combining filter pruning, fine tuning and bias correction. The experiments show that the model size is reduced significantly, while keeping or improving the accuracy.