Hybrid Weight Representation: A Quantization Method Represented with Ternary and Sparse-Large Weights

1 INTRODUCTION . Deep Neural Networks have made considerable progress in various tasks such as image classification ( LeCun et al . 1998 , Simonyan & Zisserman 2014 , Szegedy et al . 2015 ) , object detection ( Ren et al . 2015 , Liu et al . 2016 ) , and speech recognition ( Graves et al . 2013 , Amodei et al . 2016 ) . However , outstanding neural networks usually require deeper and/or wider layers , thus making them hard to deploy on mobile and embedded devices . In response to this problem , many studies set their sights on more efficient networks . Various methods such as pruning ( He et al . 2017 ) , light-weights ( Howard et al . 2017 ) , and quantization ( Courbariaux et al . 2015 ) have been carried out to reduce the model size and/or computation complexity effectively . In ternarization , the accuracy degradation is resulted from quantizing values in a limited range with only 2bits . For example , the ternary weights networks ( TWN , Li et al . 2016 ) yields only three quantized values , which prohibits the networks from utilizing high weight values . As known in Han et al . 2015b , large-valued weights tend to have an important role in the prediction . Therefore , the absence of large values can cause the accuracy degradation . To solve this problem , our paper proposes a hybrid weight representation ( HWR ) , expressing networks with both ternary weights ( TW ) and sparse-large weights ( SLW ) . By taking the advantages of both the TW and SLW , the proposed HWR method can preserve their model size compared to ternary weights , as well as avoiding the accuracy degradation in networks . To be specific , the large values of SLW help networks to improve their accuracy . Furthermore , SLW can be encoded with one remaining state which is not used to store TW in a 2 bits representation . It allows the networks to preserve their model size similarly to ternary weights . The compression rate of the encoding method is affected by the entropy of weight distributions . To train narrower distributions for the efficiency of HWR , we also introduce a centralized quantization ( CQ ) process and a weighted ridge ( WR ) regularizer . Figure 1 shows the differences between conventional quantization and HWR . As shown in Figure 1 , there is a small number of SLW and the indices of encoded SLW are allocated in storage , unlike TW . We conduct various experiments , showing that HWR obtains better classification accuracy with the similar model size compared to the trained ternary quantization ( TTQ , Zhu et al . 2016 ) , which is a baseline ternarization method . The experiments are carried out on CIFAR-100 ( Krizhevsky et al . 2009 ) and ImageNet ( Russakovsky et al . 2015 ) . We use AlexNet ( Krizhevsky et al . 2012 ) and ResNet-18 ( He et al . 2016 ) as baseline networks . Our proposed representation improves the AlexNet performance on CIFAR-100 by 4.15 % with only 1.13 % increase in the model size . The contributions of this paper are as follows : • We propose a hybrid weight representation ( HWR ) , including both values ( TW ) and indices of values ( SLW ) . SLW allows the networks to improve their accuracy . Besides , the model size can be preserved by encoding SLW with one remaining state of 2 bits for TW . • We propose a training process , namely centralized quantization ( CQ ) , to improve the efficiency of HWR . In CQ , we can sparsify almost large weights toward ternary weights . The low entropy of the centralized distribution improves the compression rate of encoding . • We propose a regularizer , namely weighted ridge ( WR ) , which gives more penalty to large weights . WR is utilized to centralize weights for narrower distributions and categorize the weights into TW and SLW . 2 RELATED WORK . Quantization In low precision training , one major difference from full precision training is that the conventional 32-bits weights ( w ) are discretized by a quantization funtion and represented with the finite number of elements . The discretized weights ( wq ) are multiplied by input matrices in the feed-forward pass . For example , the binarized neural networks ( BNN , Courbariaux & Bengio 2016 ) utilized a sign function to quantize weights and activations to { -1 , +1 } . The binarized weights ( wb ) are defined as : wb = sign ( w ) , sign ( x ) = { +1 if x ≥ 0 −1 otherwise In XNOR-Net ( Rastegari et al . 2016 ) , input data is even binarized . Furthermore , the multiplication and addition operations in convolution layers are replaced with XNOR and bit-count operations , respectively . In TWN ( Li et al . 2016 ) , the binarized weights are pruned by thresholding as : wt = { +WE if w > ∆ 0 if |w| ≤ ∆ −WE if w < −∆ , ∆ = 0.7 · E ( |w| ) WE = E i∈ { i|∆ < |w ( i ) | } ( |w ( i ) | ) In DoReFa-Net ( Zhou et al . 2016 ) , they carried out experiments in a wider bit-width and also quantized the gradients . The quantized weights in k bit-width ( wk ) by a quantization function ( Qk ( · ) ) are described as : wk = 2 ·Qk ( tanh ( w ) 2max ( |tanh ( w ) | ) + 0.5 ) − 1 , Qk ( x ) = round ( x· ( 2k−1 ) ) 2k−1 In back-propagation , their is a major concern caused by discretization of full precision weights ( w ) . The derivatives of the discretizing functions such as sign ( · ) and Qk ( · ) have zero values at almost input ranges . Therefore , the gradients of w , calculated from the discretized weights ( wb , wt , and wk ) , are also zero values and prevent w from optimizing . To solve this vanishing gradient problem , the straight-through estimator ( STE ) method ( Hinton et al . 2012 , Bengio et al . 2013 ) is proposed , where not calculating the gradients ofw . In other words , the gradients ofw ( ∂L∂w ) is replaced with the gradients of the discretized weights ( ∂L∂wb , ∂L ∂wt , and ∂L∂wk ) instead of back-propagating zero gradients from discretizing functions . ∂L ∂w = ∂L ∂wb = ∂L∂wt = ∂L ∂wk Above methods are linear quantization which have the same intervals between adjoining quantum . By quantizing with the same intervals , the float operations can be replaced to the integer operations . Their are also non-linear quantization methods that the weights have irregular intervals . For instance , the deep compression ( Han et al . 2015a ) clustered the weights to quantize and fine-tuned the quantized weights of each clustering group . In TTQ ( Zhu et al . 2016 ) , they quantized the weights to ternary values with two trainable scale coefficients for negative and positive weight values , i.e. , { −Wn , 0 , +Wp } . In these cases , the model size can be significantly reduced by replacing the weight values with the indices of them . However , the indices are needed to be transformed as weight values . And the irregular intervals make it difficult to utilize integer-based and bit-wise operations . Entropy coding The Huffman coding ( Van Leeuwen 1976 ) aims to compress bit-streams in a lossless manner by reducing bit-length of each element with an optimal prefix tree . The entropy of the weight distribution is one of the important factors that determine the compression rate in Huffman coding . Deep compression ( Han et al . 2015a ) pruned the weights to make lower entropy so saved more storage when applying the Huffman coding . Regularization The regularization is utilized to manipulate weights in artificial direction by imposing penalty . There are two conventional regularizers such as Ridge ( L2 ) and Lasso ( L1 ) ( Han et al . 2015b ) . The L2 weight decay usually prevents networks from being biased on the training dataset by restricting weights from growing . The penalty of Lasso makes the weights to be close to zero value which are unrelated to predict outputs during training-time . Moreover , the explicit loss of Zhou et al . 2018 makes it possible to quantize weights by controlling the strength of their regularizer . The penalty of regularization can be utilized to change the entropy of the weight distribution . From previous studies , we focus on that ternary weights , linearly quantized , only require three states when being stored in 2bits , thus the remaining state has a potential to be utilized as a prefix of more extended weights for improving efficiency in quantization . Furthermore , we draw a deduction that the regularization can help us to generate the lower entropy of the weight distribution , maximizing the compression efficiency when encoding the extended weights . 3 METHOD . In this section , we explain : i ) how centralized quantization ( CQ ) can centralize weights toward ternary values and categorize the weights into TW and SLW ; and ii ) how the quantized weights are encoded to be expressed as hybrid weight representation ( HWR ) . The detailed processes are illustrated in Figure 2 . 3.1 BASIC QUANTIZATION METHOD . The basic quantization ( BQ ) uses a round function to simply quantize the full precision weights ( w ) . The quantization function ( Qw ( · ) ) of BQ is fixed for each layer during training-time . Equation 1 shows how w is quantized to wq by Qw ( · ) . The rng in Eq 1 is the fixed range of w while low precision training , determined by the maximum absolute value of the pre-trained weights ( Mwp = max ( |wp| ) ) . Before being entered into Qw ( · ) , w is clipped to wc by rng to prevent misquantization of the round function . The Qw ( · ) also requires the number of quantization states ( s ) ascertained by the number of bits . To be specific aboutQw ( · ) , thewc is scaled by a float value to be discretized then it is restored by the reciprocal to the float value after the round function . Using STE , the derivative of wc ( ∂L∂wc ) is replaced with the derivative of wq ( ∂L ∂wq ) . By clipping w , we can take a saturation effect on w as in BNN ( Courbariaux & Bengio 2016 ) . But if some weights still over rng then w can be restated by clipping again after updating gradients . wq = Qw ( wc , rng , s ) = round ( wc · s−1 2 rng ) · rngs−1 2 , wc = clip ( w , −rng , rng ) ( 1 ) To quantize activated values , we restrict the range of activated values by using ReLU1 function as activation . The activated values with ReLU1 ( a ) can be quantized to aq in k bit-width by a round function in Equation 2 . As shown in TTQ ( Zhu et al . 2016 ) , initializing with the pre-trained model helps the networks to improve their quantization performance . To take this advantage , our training starts with a full precision model in which weights are pre-trained with ReLU1 activation . a = ReLU1 ( x ) = clip ( x , 0 , 1 ) = min ( max ( 0 , x ) , 1 ) aq = Qa ( a , k ) = round ( a · ( 2k − 1 ) ) / ( 2k − 1 ) ( 2 )

Summary: This paper is about quantization, and how to represent values as the finite number of states in a low bit width, using discretization. Particularly, they propose an approach to tackle the problems associated with previous ternarization which quantize weights to three values. Their approach is a hybrid weight representation method, which uses a network to output two weight types: ternary weight and sparse-large weights. For the ternary weight, they need 3 states to be stored with 2 bits. The one remaining state is used to indicate the sparse-large weight. They also propose an approach to centralize the weights towards ternary values. Their experiments show that their approach outperforms other compressed modeling approaches, and show an increase in AlexNet performance on the CIFAR-100 while increasing model size by 1.13%.