Hybrid Weight Representation: A Quantization Method Represented with Ternary and Sparse-Large Weights

1 INTRODUCTION . Deep Neural Networks have made considerable progress in various tasks such as image classification ( LeCun et al . 1998 , Simonyan & Zisserman 2014 , Szegedy et al . 2015 ) , object detection ( Ren et al . 2015 , Liu et al . 2016 ) , and speech recognition ( Graves et al . 2013 , Amodei et al . 2016 ) . However , outstanding neural networks usually require deeper and/or wider layers , thus making them hard to deploy on mobile and embedded devices . In response to this problem , many studies set their sights on more efficient networks . Various methods such as pruning ( He et al . 2017 ) , light-weights ( Howard et al . 2017 ) , and quantization ( Courbariaux et al . 2015 ) have been carried out to reduce the model size and/or computation complexity effectively . In ternarization , the accuracy degradation is resulted from quantizing values in a limited range with only 2bits . For example , the ternary weights networks ( TWN , Li et al . 2016 ) yields only three quantized values , which prohibits the networks from utilizing high weight values . As known in Han et al . 2015b , large-valued weights tend to have an important role in the prediction . Therefore , the absence of large values can cause the accuracy degradation . To solve this problem , our paper proposes a hybrid weight representation ( HWR ) , expressing networks with both ternary weights ( TW ) and sparse-large weights ( SLW ) . By taking the advantages of both the TW and SLW , the proposed HWR method can preserve their model size compared to ternary weights , as well as avoiding the accuracy degradation in networks . To be specific , the large values of SLW help networks to improve their accuracy . Furthermore , SLW can be encoded with one remaining state which is not used to store TW in a 2 bits representation . It allows the networks to preserve their model size similarly to ternary weights . The compression rate of the encoding method is affected by the entropy of weight distributions . To train narrower distributions for the efficiency of HWR , we also introduce a centralized quantization ( CQ ) process and a weighted ridge ( WR ) regularizer . Figure 1 shows the differences between conventional quantization and HWR . As shown in Figure 1 , there is a small number of SLW and the indices of encoded SLW are allocated in storage , unlike TW . We conduct various experiments , showing that HWR obtains better classification accuracy with the similar model size compared to the trained ternary quantization ( TTQ , Zhu et al . 2016 ) , which is a baseline ternarization method . The experiments are carried out on CIFAR-100 ( Krizhevsky et al . 2009 ) and ImageNet ( Russakovsky et al . 2015 ) . We use AlexNet ( Krizhevsky et al . 2012 ) and ResNet-18 ( He et al . 2016 ) as baseline networks . Our proposed representation improves the AlexNet performance on CIFAR-100 by 4.15 % with only 1.13 % increase in the model size . The contributions of this paper are as follows : • We propose a hybrid weight representation ( HWR ) , including both values ( TW ) and indices of values ( SLW ) . SLW allows the networks to improve their accuracy . Besides , the model size can be preserved by encoding SLW with one remaining state of 2 bits for TW . • We propose a training process , namely centralized quantization ( CQ ) , to improve the efficiency of HWR . In CQ , we can sparsify almost large weights toward ternary weights . The low entropy of the centralized distribution improves the compression rate of encoding . • We propose a regularizer , namely weighted ridge ( WR ) , which gives more penalty to large weights . WR is utilized to centralize weights for narrower distributions and categorize the weights into TW and SLW . 2 RELATED WORK . Quantization In low precision training , one major difference from full precision training is that the conventional 32-bits weights ( w ) are discretized by a quantization funtion and represented with the finite number of elements . The discretized weights ( wq ) are multiplied by input matrices in the feed-forward pass . For example , the binarized neural networks ( BNN , Courbariaux & Bengio 2016 ) utilized a sign function to quantize weights and activations to { -1 , +1 } . The binarized weights ( wb ) are defined as : wb = sign ( w ) , sign ( x ) = { +1 if x ≥ 0 −1 otherwise In XNOR-Net ( Rastegari et al . 2016 ) , input data is even binarized . Furthermore , the multiplication and addition operations in convolution layers are replaced with XNOR and bit-count operations , respectively . In TWN ( Li et al . 2016 ) , the binarized weights are pruned by thresholding as : wt = { +WE if w > ∆ 0 if |w| ≤ ∆ −WE if w < −∆ , ∆ = 0.7 · E ( |w| ) WE = E i∈ { i|∆ < |w ( i ) | } ( |w ( i ) | ) In DoReFa-Net ( Zhou et al . 2016 ) , they carried out experiments in a wider bit-width and also quantized the gradients . The quantized weights in k bit-width ( wk ) by a quantization function ( Qk ( · ) ) are described as : wk = 2 ·Qk ( tanh ( w ) 2max ( |tanh ( w ) | ) + 0.5 ) − 1 , Qk ( x ) = round ( x· ( 2k−1 ) ) 2k−1 In back-propagation , their is a major concern caused by discretization of full precision weights ( w ) . The derivatives of the discretizing functions such as sign ( · ) and Qk ( · ) have zero values at almost input ranges . Therefore , the gradients of w , calculated from the discretized weights ( wb , wt , and wk ) , are also zero values and prevent w from optimizing . To solve this vanishing gradient problem , the straight-through estimator ( STE ) method ( Hinton et al . 2012 , Bengio et al . 2013 ) is proposed , where not calculating the gradients ofw . In other words , the gradients ofw ( ∂L∂w ) is replaced with the gradients of the discretized weights ( ∂L∂wb , ∂L ∂wt , and ∂L∂wk ) instead of back-propagating zero gradients from discretizing functions . ∂L ∂w = ∂L ∂wb = ∂L∂wt = ∂L ∂wk Above methods are linear quantization which have the same intervals between adjoining quantum . By quantizing with the same intervals , the float operations can be replaced to the integer operations . Their are also non-linear quantization methods that the weights have irregular intervals . For instance , the deep compression ( Han et al . 2015a ) clustered the weights to quantize and fine-tuned the quantized weights of each clustering group . In TTQ ( Zhu et al . 2016 ) , they quantized the weights to ternary values with two trainable scale coefficients for negative and positive weight values , i.e. , { −Wn , 0 , +Wp } . In these cases , the model size can be significantly reduced by replacing the weight values with the indices of them . However , the indices are needed to be transformed as weight values . And the irregular intervals make it difficult to utilize integer-based and bit-wise operations . Entropy coding The Huffman coding ( Van Leeuwen 1976 ) aims to compress bit-streams in a lossless manner by reducing bit-length of each element with an optimal prefix tree . The entropy of the weight distribution is one of the important factors that determine the compression rate in Huffman coding . Deep compression ( Han et al . 2015a ) pruned the weights to make lower entropy so saved more storage when applying the Huffman coding . Regularization The regularization is utilized to manipulate weights in artificial direction by imposing penalty . There are two conventional regularizers such as Ridge ( L2 ) and Lasso ( L1 ) ( Han et al . 2015b ) . The L2 weight decay usually prevents networks from being biased on the training dataset by restricting weights from growing . The penalty of Lasso makes the weights to be close to zero value which are unrelated to predict outputs during training-time . Moreover , the explicit loss of Zhou et al . 2018 makes it possible to quantize weights by controlling the strength of their regularizer . The penalty of regularization can be utilized to change the entropy of the weight distribution . From previous studies , we focus on that ternary weights , linearly quantized , only require three states when being stored in 2bits , thus the remaining state has a potential to be utilized as a prefix of more extended weights for improving efficiency in quantization . Furthermore , we draw a deduction that the regularization can help us to generate the lower entropy of the weight distribution , maximizing the compression efficiency when encoding the extended weights . 3 METHOD . In this section , we explain : i ) how centralized quantization ( CQ ) can centralize weights toward ternary values and categorize the weights into TW and SLW ; and ii ) how the quantized weights are encoded to be expressed as hybrid weight representation ( HWR ) . The detailed processes are illustrated in Figure 2 . 3.1 BASIC QUANTIZATION METHOD . The basic quantization ( BQ ) uses a round function to simply quantize the full precision weights ( w ) . The quantization function ( Qw ( · ) ) of BQ is fixed for each layer during training-time . Equation 1 shows how w is quantized to wq by Qw ( · ) . The rng in Eq 1 is the fixed range of w while low precision training , determined by the maximum absolute value of the pre-trained weights ( Mwp = max ( |wp| ) ) . Before being entered into Qw ( · ) , w is clipped to wc by rng to prevent misquantization of the round function . The Qw ( · ) also requires the number of quantization states ( s ) ascertained by the number of bits . To be specific aboutQw ( · ) , thewc is scaled by a float value to be discretized then it is restored by the reciprocal to the float value after the round function . Using STE , the derivative of wc ( ∂L∂wc ) is replaced with the derivative of wq ( ∂L ∂wq ) . By clipping w , we can take a saturation effect on w as in BNN ( Courbariaux & Bengio 2016 ) . But if some weights still over rng then w can be restated by clipping again after updating gradients . wq = Qw ( wc , rng , s ) = round ( wc · s−1 2 rng ) · rngs−1 2 , wc = clip ( w , −rng , rng ) ( 1 ) To quantize activated values , we restrict the range of activated values by using ReLU1 function as activation . The activated values with ReLU1 ( a ) can be quantized to aq in k bit-width by a round function in Equation 2 . As shown in TTQ ( Zhu et al . 2016 ) , initializing with the pre-trained model helps the networks to improve their quantization performance . To take this advantage , our training starts with a full precision model in which weights are pre-trained with ReLU1 activation . a = ReLU1 ( x ) = clip ( x , 0 , 1 ) = min ( max ( 0 , x ) , 1 ) aq = Qa ( a , k ) = round ( a · ( 2k − 1 ) ) / ( 2k − 1 ) ( 2 )

Summary: The paper proposes a hybrid weights representation method where the weights of the neural network is split into two portions: a major portion of ternary weights and a minor portion of weights that are represented with different number of bits. The two portions of weights are differentiated by using the previous unused state of a typical ternary neural network since only three states are used out of the four states given 2-bit representation. The experiments are solid based on the selected baseline model on CIFAR-100 and Imagenet dataset.