Title: HYBRID WEIGHT REPRESENTATION: A QUANTIZA-TION METHOD REPRESENTED WITH TERNARY AND SPARSE-LARGE WEIGHTS

Abstract: Previous ternarizations such as the trained ternary quantization (TTQ), which quantized weights to three values (e.g., {-W n , 0, +W p }), achieved the small model size and efficient inference process. However, the extreme limit on the number of quantization steps causes some degradation in accuracy. To solve this problem, we propose a hybrid weight representation (HWR) method which produces a network consisting of two types of weights, i.e., ternary weights (TW) and sparse-large weights (SLW). The TW is similar to the TTQ's and requires three states to be stored in memory with 2 bits. We utilize the one remaining state to indicate the SLW which is referred to as very rare and greater than TW. In HWR, we represent TW with values while SLW with indices of values by encoding SLW. As a result, the networks can preserve their model size with improving their accuracy compared to ternary weights. To sparsify non-ternary weights in quantization, we also introduce a centralized quantization (CQ) process with a weighted ridge (WR) regularizer. They aim to reduce the entropy of weight distributions by centralizing weights toward ternary values. Our comprehensive experiments show the efficiency of HWR in terms of the trade-off between model size and accuracy.

1. INTRODUCTION:

Deep Neural Networks have made considerable progress in various tasks such as image classification (LeCun et al. 1998 , Simonyan & Zisserman 2014 , Szegedy et al. 2015) , object detection (Ren et al. 2015 , Liu et al. 2016) , and speech recognition (Graves et al. 2013 , Amodei et al. 2016 ). However, outstanding neural networks usually require deeper and/or wider layers, thus making them hard to deploy on mobile and embedded devices. In response to this problem, many studies set their sights on more efficient networks. Various methods such as pruning (He et al. 2017 ), light-weights (Howard et al. 2017) , and quantization (Courbariaux et al. 2015) have been carried out to reduce the model size and/or computation complexity effectively.

In ternarization, the accuracy degradation is resulted from quantizing values in a limited range with only 2bits. For example, the ternary weights networks (TWN, Li et al. 2016 ) yields only three quantized values, which prohibits the networks from utilizing high weight values. As known in Han et al. 2015b , large-valued weights tend to have an important role in the prediction. Therefore, the absence of large values can cause the accuracy degradation. To solve this problem, our paper proposes a hybrid weight representation (HWR), expressing networks with both ternary weights (TW) and sparse-large weights (SLW) . By taking the advantages of both the TW and SLW, the proposed HWR method can preserve their model size compared to ternary weights, as well as avoiding the accuracy degradation in networks.

To be specific, the large values of SLW help networks to improve their accuracy. Furthermore, SLW can be encoded with one remaining state which is not used to store TW in a 2 bits representation. It allows the networks to preserve their model size similarly to ternary weights. The compression rate of the encoding method is affected by the entropy of weight distributions. To train narrower distributions for the efficiency of HWR, we also introduce a centralized quantization (CQ) process and a weighted ridge (WR) regularizer. Figure 1 shows the differences between conventional quantization and HWR. As shown in Figure 1 , there is a small number of SLW and the indices of encoded SLW are allocated in storage, unlike TW.

We conduct various experiments, showing that HWR obtains better classification accuracy with the similar model size compared to the trained ternary quantization (TTQ, Zhu et al. 2016) , which is a baseline ternarization method. The experiments are carried out on CIFAR-100 (Krizhevsky et al. 2009) and ImageNet (Russakovsky et al. 2015) . We use AlexNet (Krizhevsky et al. 2012) and ResNet-18 (He et al. 2016 ) as baseline networks. Our proposed representation improves the AlexNet performance on CIFAR-100 by 4.15% with only 1.13% increase in the model size.

The contributions of this paper are as follows:

• We propose a hybrid weight representation (HWR), including both values (TW) and indices of values (SLW). SLW allows the networks to improve their accuracy. Besides, the model size can be preserved by encoding SLW with one remaining state of 2 bits for TW.

• We propose a training process, namely centralized quantization (CQ), to improve the efficiency of HWR. In CQ, we can sparsify almost large weights toward ternary weights. The low entropy of the centralized distribution improves the compression rate of encoding.

• We propose a regularizer, namely weighted ridge (WR), which gives more penalty to large weights. WR is utilized to centralize weights for narrower distributions and categorize the weights into TW and SLW.

2. RELATED WORK:

Quantization In low precision training, one major difference from full precision training is that the conventional 32-bits weights (w) are discretized by a quantization funtion and represented with the finite number of elements. The discretized weights (w q ) are multiplied by input matrices in the feed-forward pass. For example, the binarized neural networks (BNN, Courbariaux & Bengio 2016 ) utilized a sign function to quantize weights and activations to {-1, +1}. The binarized weights (w b ) are defined as:

w b = sign(w), sign(x) = +1 if x ≥ 0 -1 otherwise

In XNOR-Net (Rastegari et al. 2016) , input data is even binarized. Furthermore, the multiplication and addition operations in convolution layers are replaced with XNOR and bit-count operations, respectively. In TWN (Li et al. 2016) , the binarized weights are pruned by thresholding as:

w t = +W E if w > ∆ 0 if |w| ≤ ∆ -W E if w < -∆ , ∆ = 0.7 • E(|w|) W E = E i∈{i|∆<|w(i)|} (|w(i)|)

In DoReFa-Net (Zhou et al. 2016) , they carried out experiments in a wider bit-width and also quantized the gradients. The quantized weights in k bit-width (w k ) by a quantization function (Q k (•)) are described as: In back-propagation, their is a major concern caused by discretization of full precision weights (w).

w k = 2 • Q k ( tanh(w) 2max(|tanh(w)|) + 0.5) -1 , Q k (x) = round(x•(2 k -1)) 2 k -1

The derivatives of the discretizing functions such as sign(•) and Q k (•) have zero values at almost input ranges. Therefore, the gradients of w, calculated from the discretized weights (w b , w t , and w k ), are also zero values and prevent w from optimizing. To solve this vanishing gradient problem, the straight-through estimator (STE) method (Hinton et al. 2012 , Bengio et al. 2013 ) is proposed, where not calculating the gradients of w. In other words, the gradients of w ( ∂L ∂w ) is replaced with the gradients of the discretized weights ( ∂L ∂w b , ∂L ∂wt , and ∂L ∂w k ) instead of back-propagating zero gradients from discretizing functions.

∂L ∂w = ∂L ∂w b = ∂L ∂wt = ∂L ∂w k

Above methods are linear quantization which have the same intervals between adjoining quantum. By quantizing with the same intervals, the float operations can be replaced to the integer operations.

Their are also non-linear quantization methods that the weights have irregular intervals. For instance, the deep compression (Han et al. 2015a ) clustered the weights to quantize and fine-tuned the quantized weights of each clustering group. In TTQ (Zhu et al. 2016) , they quantized the weights to ternary values with two trainable scale coefficients for negative and positive weight values, i.e., {-W n , 0, +W p }. In these cases, the model size can be significantly reduced by replacing the weight values with the indices of them. However, the indices are needed to be transformed as weight values.

And the irregular intervals make it difficult to utilize integer-based and bit-wise operations.

Entropy coding The Huffman coding (Van Leeuwen 1976) aims to compress bit-streams in a lossless manner by reducing bit-length of each element with an optimal prefix tree. The entropy of the weight distribution is one of the important factors that determine the compression rate in Huffman coding. Deep compression (Han et al. 2015a ) pruned the weights to make lower entropy so saved more storage when applying the Huffman coding.

Regularization The regularization is utilized to manipulate weights in artificial direction by imposing penalty. There are two conventional regularizers such as Ridge (L2) and Lasso (L1) (Han et al. 2015b) . The L2 weight decay usually prevents networks from being biased on the training dataset by restricting weights from growing. The penalty of Lasso makes the weights to be close to zero value which are unrelated to predict outputs during training-time. Moreover, the explicit loss of Zhou et al. 2018 makes it possible to quantize weights by controlling the strength of their regularizer. The penalty of regularization can be utilized to change the entropy of the weight distribution.

From previous studies, we focus on that ternary weights, linearly quantized, only require three states when being stored in 2bits, thus the remaining state has a potential to be utilized as a prefix of more extended weights for improving efficiency in quantization. Furthermore, we draw a deduction that the regularization can help us to generate the lower entropy of the weight distribution, maximizing the compression efficiency when encoding the extended weights.

3. METHOD:

In this section, we explain: i) how centralized quantization (CQ) can centralize weights toward ternary values and categorize the weights into TW and SLW; and ii) how the quantized weights are encoded to be expressed as hybrid weight representation (HWR). The detailed processes are illustrated in Figure 2 .

3.1. BASIC QUANTIZATION METHOD:

The basic quantization (BQ) uses a round function to simply quantize the full precision weights (w).

The quantization function (Q w (•)) of BQ is fixed for each layer during training-time. Equation 1shows how w is quantized to w q by Q w (•). The rng in Eq 1 is the fixed range of w while low precision training, determined by the maximum absolute value of the pre-trained weights (M wp = max(|w p |)). Before being entered into Q w (•), w is clipped to w c by rng to prevent misquantization of the round function. The Q w (•) also requires the number of quantization states (s) ascertained by the number of bits. To be specific about Q w (•), the w c is scaled by a float value to be discretized then it is restored by the reciprocal to the float value after the round function. Using STE, the derivative of w c ( ∂L ∂wc ) is replaced with the derivative of w q ( ∂L ∂wq ). By clipping w, we can take a saturation effect on w as in BNN (Courbariaux & Bengio 2016) . But if some weights still over rng then w can be restated by clipping again after updating gradients.

EQUATION

To quantize activated values, we restrict the range of activated values by using ReLU1 function as activation. The activated values with ReLU1 (a) can be quantized to a q in k bit-width by a round function in Equation 2. As shown in TTQ (Zhu et al. 2016) , initializing with the pre-trained model helps the networks to improve their quantization performance. To take this advantage, our training starts with a full precision model in which weights are pre-trained with ReLU1 activation. a = ReLU 1(x) = clip(x, 0, 1) = min(max(0, x), 1)

a q = Q a (a, k) = round(a • (2 k -1)) / (2 k -1) (2) 3.2 WEIGHTED RIDGE

We introduce a new version of L2 weight decay, namely weighted ridge (WR), which aims to achieve two objectives: i) centralizing almost all weights toward below the threshold; and ii) categorizing the weights into TW and SLW by the threshold at the end of re-training. In this step, we quantize full precision weights (w) by the basic quantization (BQ) with WR as a regularizer. This step's quantization function is defined using Equation 1:

EQUATION

where the rng in Equation 1 is the maximum absolute value of the pre-trained weights (M wp = max(|w p |)), fixed during re-training. The s in Equation 1 is derived from the number of bits of SLW (b sl ). If b sl is 2, 3, or 4 bits then the number of quantization steps of SLW (s sl ) is calculated to 4, 8, or 16 while the number of bits and quantization steps of TW (b t and s t ) are fixed to 2 and 3. The loss of pL1 is only proportionate to the absolute value of the weight which is greater than the threshold (thresh). In other words, the large weights receive more penalty compared to the weights under thresh in back-propagation as illustrated in Figure 3 (c ). This penalty allows us to sparsify the weights that do not have the necessity to keep their largeness. WR, pL1, and thresh are defined:

EQUATION

There are two hyper-parameters (λ 1 and λ 2 ) which are tools to control the intensity of WR. By adjusting the greater λ 2 , more weights can be sparsified below thresh. The M wp is also a factor for applying different intensity to each layer since some layers have too small M wp than others.

At the end of applying WR, we classify the re-trained weights into TW and SLW. The thresh is utilized as a criterion for labeling them as Equation 6.

mask(w) = T W if |w| ≤ thresh SLW if |w| > thresh (6)

An observation is that WR tends to avoid overfitting as effectively as the L2 regularizer. By adjusting WR, we can obtain better accuracy than the L2 weight decay in some experiments as in Table 3 . 7, modified from Equation 3 (The explanations for M wp , thresh, s t , and s sl are in Section 3.2). At the end of fine-tuning, the fine-tuned weights are re-categorized by the same way as Equation 6. This finetuning step can be skipped depending on the result of SQ, if it yields only slight difference compared to the weighted ridge (WR).

3.3. SELECTIVE QUANTIZATION:

Q s (w) = Q w (clip(w, -thresh, thresh), thresh, s t + 1) if mask(w) = T W Q w (clip(w, -M wp , M wp ), M wp , s t + s sl ) if mask(w) = SLW (7)

Specifically, we use the stochastic gradient descent optimizer (SGD, Bottou 2010) in this fine-tuning step. The momentum optimizer (Sutskever et al. 2013 ) can misrepresent gradients since the weight values are clipped after updating gradients. 

3.4. ENCODING SPARSE-LARGE WEIGHTS:

The distributions of centralized-quantized weights have low entropy since there are only small number of SLW. It allows the networks to achieve a higher compression rate by encoding SLW. We utilize one remaining state of TW as a prefix to encode SLW as shown in Figure 5 . As a result, the hybrid weight representation (HWR) method includes both the values (TW) and the indices (SLW) when the weights are allocated in memory.

There are some advantages of using HWR. The model size can be maintained similar to TTQ (Zhu et al. 2016) since SLW is very rare. The restoration of encoded SLW is not that difficult. The value of SLW can be restored by taking the first sign bit of b sl and adding 2 to the rest of b sl . However, there is a concern that HWR causes more complex and extra burden to memory access and inference time due to the SLW. To minimize the negative effect of SLW, we suggest to utilize the sparsity of SLW by gating them or applying a variant of sparse matrix-vector multiplication. Then the TW takes up most of the weights and still has the potential to keep some benefits of ternary weights such as replacing multiplication operations to sign assignment operations (Rastegari et al. 2016) and utilizing gates for skipping zero weights (Deng et al. 2018) . More details about our proposed inference methods are in Appendices C.

4. EXPERIMENT RESULTS:



Model architecture:

We utilize AlexNet (Krizhevsky et al. 2012 ) and pre-activated ResNet-18 (He et al. 2016) as baseline models, implemented on Tensorflow (Abadi et al. 2016) framework. In AlexNet, we use a variant of it by adding batch normalization (Ioffe & Szegedy 2015) layers and removing dropout layers. In ResNet, we change two things: i) not using skip-connection for the first block of ResNet-18, since the first layer is not quantized and the not quantized weights cannot be entered into the add operations to keep integer-precision stream while inference-time; and ii) mediating of two quantum intervals from convolution layers. In add operations, each input from two convolution layers has different quantum interval due to layer-wise quantization functions. The different intervals make it hard to keep integer-precision stream after add operations. More details of ResNet-18 are explained in Appendices A.

4.1. EXPERIMENTS ON CIFAR-100:

Training setup We set the learning rate of AlexNet to 0.01 with decay 0.1 after every 100 epochs until 300 epochs and the L2 weight decay (λ 1 ) to 0.02. We also set the learning rate of ResNet-18 to 0.1 with the same decay of AlexNet and the λ 1 to 0.0002. For the dataset, we select CIFAR-100 Krizhevsky et al. 2009 ) and set the minibatch size to 128 to estimate the performance of networks. We also use Nesterov Accelerated Gradients (Sutskever et al. 2013 ) as an optimizer. To prevent the fluctuation of validation accuracy, we take a simple moving averages (sma) in 5 epochs during training-time and average the sma over four repetitions. All experiments are our implementation. Implementation Our experiments show the comparison between our proposed methods (BQ, BQ with WR, and SQ in Section 3) and TTQ (Zhu et al. 2016 ) as a baseline model. In weight quantization, the number of bits of TW (b t ) is fixed to 2 while the number of bits of SLW (b sl ) can be changed to 2, 3, or 4. In activation quantization, we select 4 bits to quantize the activations. For the original TTQ paper, they did not quantize activated values. In our implementation of TTQ (TTQ*), we quantize activations with the same quantization function in Equation 2 to make similar condition. Also, our ResNet-18 architecture for TTQ* can be different from the origian TTQ as written above. We quantize all the weights and activations, even the last activation before the global average pooling, except the weights of first and last layers and the outputs of soft-max layer.

In Table 1 , the effect of the weighted ridge (WR) regularizer can be found out from the comparison of the basic quantization (BQ) and BQ with WR. The results of BQ+WR show more centralized distribution and less accuracy than only BQ. Furthermore, we observe that the results of the selective quantization (SQ) have more centralized distribution than BQ+WR and more improved accuracy than BQ, removing the part of L1 in WR and fine-tuning the restricted weights by WR. After the encoding step, the difference of the average bit length between SQ and TTQ* becomes similar. When b sl is 3 for AlexNet, SQ reaches top-1 accuracy of 74.72%, which is 4.15% higher than TTQ*, with only 1.25% increase in the model size. In ResNet-18, when b sl is 3, the top-1 accuracy of SQ is 1.88% higher than TTQ* with only 0.5% loss in the model size.

4.2. EXPERIMENTS ON IMAGENET:

Training setup We performed the experiments of ResNet-18 on ImageNet (Russakovsky et al. 2015) dataset. We set the learning rate to 0.1 with decay 0.1 after 25, 50, 75, and 80 epochs until 85 epochs and the minibatch size to 256. We use the momentum optimizer (Sutskever et al. 2013) . We take a simple moving average of validation error in 5 epochs during training-time.

In Table 2 , the results of BQ, BQ+WR, and SQ on ImageNet show a similar tendency of the results on CIFAR-100. The SQ (b sl = 4) reaches top-1 accuracy of 59%, which is 6.41% higher than TTQ*, In this study, we try to observe the effect of the weighted ridge (WR) regularizer on full precision models as shown in Table 3 . When λ 2 is set to 1, the penalty of the part of L1 (pL1) of WR is calculated to the maximum absolute value of the pre-trained model (M wp ) by the derivative of Equation 4, which is sufficiently large because the penalty M wp is same with the maximum penalty of L2 (M wp ). The results of AlexNet and ResNet-18 with WR obtain 0.26% and 0.81% higher accuracy than the base-line models. The pL1 prevents the networks from overfitting as effectively as L2 regularizer. We conjecture that the biased large weights toward training dataset have more negatively effect on generalization than small weights. Therefore, the restriction of large weights enables us to make more generalized models. More ablation studies are in Appendices B.

5. DISCUSSION:



5.1. TRADE-OFF BETWEEN ACCURACY AND MODEL SIZE:

We perform more extended experiments in various λ 2 of WR to monitor the correlation between the percentage of SLW and the accuracy of ResNet-18 on CIFAR-100. The complete results are listed in Appendices E. Figure 6 shows that the intensity of pL1 (λ 2 ) in Equation 4has less relation with the classification accuracy. In spite of the greater λ 2 , some results show even better accuracy than λ 2 (2, 1, 1  2 , 1  4 , 1  8 , 1  16 ).

lower λ 2 . Therefore, we can perceive that many large weights in networks can be centralized with comparable accuracy, accompanied of generalization. As illustrated in Figure 7 , the blue and red bars refer to the percentage of TW and SLW respectively. In the SQ step, the forcibly restricted weights by WR can grow during fine-tuning, resulting in less percentage of SLW than BQ+WR as in Table 1 . From that, we assume that when a weight starting to grow in fine-tuning begins to contribute more importantly for prediction than some other weights which have made a similar contribution with the growing weight, the other weights become smaller in contrast. The most central weights of SLW in BQ+WR are wide-spread after SQ step. This shows that the penalty of pL1 cannot be proper to make more naturally centralized distributions. To make more flatten distributions after centralization, we suggest using another version of WR which utilize a part of exponential lasso (pEL1, Breheny 2015) instead of pL1 as written in Appendices D.

5.3. ANALYSIS OF QUANTIZATION METHOD.:

Figure 8 : Layer-wise comparison of the percentage of SLW and the maximum absolute value.

Figure 8 displays the layer-wise maximum absolute value and percentage of SLW. If one layer has many SLW after BQ then the layer tends to have a relatively higher number of SLW after SQ. The maximum absolute values (max-abs) of SQ tend to be larger than them of BQ since the restriction of the number of SLW causes more strong gradients for large weights. The max-abs of BQ are remarkably lower than the full precision model, used to fix the range of quantization functions at initialization. That shows a problem that the basic quantization (BQ) method is not an effective way because some layers do not use the whole quantization states. For this reason, we need to apply more effective quantization methods instead of BQ. The following Table 4 lists other methods. The state-of-the-art quantizations Table 4 presents that our method HWR has less accuracy than other state-of-the-art quantization methods. There are two main reasons: i) more complex quantization methods. For example, PACT (Choi et al. 2018 ) use parameterized clipping activation and QIR (Jung et al. 2019 ) use a transformer before discretizing weights; ii) differences of detail of ResNet-18 architecture. As explained in Section 4, we try to keep the integer-precision stream during inference-time. Especially, the skip-connections and add operations of ResNet can disturb the stream to maintain integer-precision since two outputs from different convolution layers, having different quantization intervals, should be added.

To deal with that BQ has low performance, the state-of-the-art quantization methods can be harmonized with the centralized quantization (CQ) process. The full precision model for initialization can be replaced with the pre-quantized model trained by those methods. The quantization function of BQ can be replaced with the quantization function of the pre-quantized model.

6. CONCLUSION:

We propose the hybrid weight representation (HWR), consisting of two types of weights such as the ternary weights (TW) and sparse-large weights(SLW). In HWR, we represent TW with values while SLW with indices of values. To represent SLW as indices, we encode SLW by utilizing one usable state as a prex, which is not used to store TW in 2 bits. To maximize the effect of encoding, we introduce the centralized quantization (CQ) process: i) applying the basic quantization (BQ) with the weighted ridge (WR) as a regularizer to centralize and categorize the weights; and ii) applying the selective quantization (SQ) to fine-tune the weights. After encoding process, the model size of HWR is similar with the TTQ even non-ternary weights exist. Additionally, we propose a method to mediate two different quantum intervals of add operations in ResNet to keep the integer-precision stream while inference-time. As a result, SLW helps the networks to improve their accuracy while preserving their model size by encoding SLW compared to the 2-bit representation. We can achieve more efficient networks in terms of the trade-off between accuracy and model size.

7. FUTURE WORK:

We consider three ways to extend our research: i) using more accurate quantization method instead of BQ. The pre-quantized weights and quantization functions from state-of-the-art methods can be utilized for HWR; ii) using more bits for central weights. In mid-tread and symmetrical quantization, there is still one remaining state similarly with the ternary weights. Therefore, more experiments can be carried out to find the optimal number of bits for center and edge weights; and iii) verifying the performance of suggested inference methods. As written in Appendices C, we suggest efficient inference methods utilizing the sparsity of SLW to minimize the extra burden caused by SLW. In our quantization method, the weights and activations are quantized in the same intervals (w q = i w • I w , a q = i a • I a ). The I w and I a are fixed float intervals while the i w and i a are integer variables. Therefore, convolution and fully-connected layers can be inferred by integer operations as Equation 8. Other operations such as batch normalization (BN, Ioffe & Szegedy 2015), ReLU1 activation, and the Q a (•) also can be integrated and compressed by integer comparators (Lahoud et al. 2019 , Lin et al. 2017 ) since the coefficients of BN and boundaries of ReLU1 and Q a (•) are fixed so can be prearranged.

o q = layer(a q , w q ) = a q w q = I a I w • i a i w (8) However, ResNet (He et al. 2016 ) has identity mappings and add operations that prevent the data streams from keeping the same quantum intervals due to the layer-wise intervals. To deal with this problem, we scaled one input of add operations by a scale factor r as defined in Equation 9. The r is derived from the maximum absolute values of the shortcut (m short ) and last convolution (m conv b ) layers from each block as in Figure 9 . Since we use a log and floor function, r is always in the range (0.5, 1] and it helps to keep the effect of identity mapping of ResNet. If r is zero then nothing is added in the add operation or if r is larger than 1 then the shortcut can be perturbed by a large residual.

EQUATION

By adjusting r, two different intervals from jth (shortcut) and kth (conv b ) convolution layers can be mediated and it will allow us to add two inputs with only shift operations (multiplication of multiple of 2) as Equation 10.

EQUATION

A.2 THE FIRST RESIDUAL BLOCK OF RESNET

In our quantization, the first convolution layer is not quantized. Therefore, to prevent ResNet from having mixed precision output after add operations, our first residual block do not have a skipconnection even though the input shape of the block is same with the output shape of the first layer. To observe the effect of the fixed range of the basic quantization (BQ) method, we perform a particular experiment. The results of our implementation of TTQ (TTQ*) is performed with the quantization function of which the range is not fixed. On the other hand, a version of implementation (TTQ**) have two different conditions compared to TTQ*: i) fixing the range of the quantization function at initialization; and ii) using one coefficient as {-W s , 0, +W s }. When applying them, the accuracy of AlexNet and ResNet-18 are rather improved by 1.47% and 0.76%. The results of TTQ** shows that the fixed range also can be utilized for quantization. Additionally, TTQ set a constant t to 0.05 for quantizing weights under t • max(|w|) as zero values. In our method, however, if b sl is 2, 3, or 4 bits then the t is derived as 1 6 , 1 10 , or 1 18 . All our t are higher than 0.05 of TTQ and it can cause worse results since L2 regularizer gives more penalty to the larger weights. In Table 6 , the experiments are conducted to observe the effect of the coefficient r, introduced in Appendices A.1, on the accuracy of ResNet-18. We compare three coefficients such as zero, r in Equation 9, and m short /m conv b . We observe that r performs as similar as non-scaled models. As mentioned in Appendices A.1, the m short /m conv b , greater than 1, can perturb the shortcut connections. We apply r scaling for all our quantization experiments on ResNet-18.

B.2 MEDIATION OF QUANTUM INTERVALS:



C EFFICIENT INFERENCE METHODS:

In this section, we suggest two methods for efficient inference of HWR. Those methods utilize the sparsity of SLW to minimize the extra burden caused by SLW and the benefit of ternary weights that the multiplication of ternary weights can be replaced with the sign assignment operation (Rastegari et al. 2016) .

C.1 GATED MULTIPLICATION Figure 10 : Computing architecture for HWR in inference time. According to the TW of each weight (W 1 ), the operation for input (X 1 ) is determined. There are two operations such as sign assignment (sign asgmt) and integer multiplication (MUTT) for each type, TW and SLW.

In GXNOR-Net (Deng et al. 2018) , they proposed gated XNOR architecture to take an advantage of binary weights when using ternary weights. If the input or weight are zero then the multiplication can be skipped while if they are not zero then the XNOR operation can be applied similarly with the XNOR-Net (Rastegari et al. 2016) . In HWR case, we use enable pins as gates to divide two cases of TW and SLW. As shown in Figure 10 , if the TW is 01 2 or 11 2 and input is not zero then only sign assignment operation (sign asgmt) is done while if the TW is 10 2 then only integer multiplication (MUTT) is done. The sign-assigned or multiplied values are added at the end to be an output of one neuron. In conventional layer, there is only one weight matrix. On the other hand, we can divide the one quantized matrix (W q ) to two matrices such as a dense matrix W T W and a sparse matrix W SLW as in Equation 11.

C.2 SEPARATED MULTIPLICATION:

EQUATION

The W T W and W SLW can be described as:

W T W = W q if mask(w) = T W 0 otherwise , W SLW = 0 otherwise W q if mask(w) = SLW (12)

In inference time, the multiplication of quantized activations (A q ) and quantized weights (W q ) also can be divided as:

A q • W q = A q • W T W + A q • W SLW (13)

For multiplication of A q and W T W , the only sign assignment is required. For multiplication of A q and W SLW , we can utilize sparse convolution operation (Park et al. 2016) to reduce computational complexity. After applying each operation, two calculated results are added to be an output of a layer.

D A PART OF EXPONENTIAL LASSO 

E FULL EXPERIMENTS OF Table 7 shows the full version of our experiment results on CIFAR-100 dataset with ResNet-18 model in various λ 2 .

:



Figures and Tables:

Figure 1: Comparison between conventional quantization and hybrid weight representation (HWR).

Figure 2: Overall processes of the hybrid weight representation (HWR). The weights are initialized from a full precision model trained with ReLU1 activation. First of the centralized quantization (CQ), the weights are quantized with the weight ridge (WR) as a regularizer. By WR, the quantized weights are also centralized under threshold and categorized into ternary weights (TW) and sparselarge weights (SLW) by threshold. Second, selective quantization (SQ) is applied to fine-tune the weights of the previous step. Finally, SLW is encoded by one usable state of TW as a prefix.

Figure 3: Specifications of weighted ridge (WR). (a) plots both a normal L2 and a part of L1 (pL1). The WR is an addition of them as in (b). (c) is the derivative of WR.

Figure 3 (b) denotes WR which is a mixture of both a normal L2 and a part of L1 (pL1) in (a).The loss of pL1 is only proportionate to the absolute value of the weight which is greater than the threshold (thresh). In other words, the large weights receive more penalty compared to the weights under thresh in back-propagation as illustrated in Figure3 (c). This penalty allows us to sparsify the weights that do not have the necessity to keep their largeness. WR, pL1, and thresh are defined:

Figure 4: In the selective quantization (SQ), two different quantization functions are applied for each category (TW and SLW).

Figure 5: Encoding trees of hybrid weight representation (HWR). In HWR, we represent TW with values while SLW with indices of values. The number of bits and quantization steps of TW (b t , s t ) are fixed to 2, 3. In contrast, them of SLW (b sl , s sl ) can be changed to (2, 4), (3, 8), or (4, 16).

Figure 6: The trade-off between accuracy and model size in variousλ 2 (2, 1, 1 2 , 1 4 , 1 8 , 1 16).

Figure 7: The weight distributions of each method.

Figure 9: Before add operations in ResNet, the outputs of last convolution layers from each block are scaled by a scale factor r, derived from the maximum absolute values of convolution layers.

Figure 11: Computing architecture for HWR in inference time. There are two weight matrices such as a dense matrix W T W and a sparse matrix W SLW . The original weight matrix can be restored by sum of W T W and W SLW . In inference time, sign assignment operation is carried out with W T W while sparse convolution is carried out with W SLW . The output of a layer is sum of outputs from two operations.

Figure 12: Another version of WR, using the part of exponential lasso (pEL1) instead of pL1. (a) denotes both L2 and pEL1. The version of WR is addition of them as in (b). (c) is the derivative of the version of WR.

The results of both the top-1 accuracy and the average bit length in multiple bit-width. The full experiments of ResNet-18 are listed in Appendices E

Table content: <table><tr><td>Model Name</td><td>Bit-W (b t /b sl )</td><td colspan="2">Bit-A λ 2</td><td>Percentage of SLW</td><td>Average Bit Length</td><td>Top-1 Accuracy</td><td>Remarks</td></tr><tr><td>AlexNet</td><td>32</td><td>32</td><td>-</td><td>-</td><td>-</td><td>75.06%</td><td>FP</td></tr><tr><td>AlexNet</td><td>32</td><td>32</td><td>-</td><td>-</td><td>-</td><td>74.78%</td><td>FP(ReLU1)</td></tr><tr><td>AlexNet</td><td>2 / -</td><td>4</td><td>-</td><td>0%</td><td>2</td><td>70.57%</td><td>TTQ*</td></tr><tr><td>AlexNet</td><td>2 / 3</td><td>4</td><td>-</td><td>12.05%</td><td>2.362</td><td>73.54%</td><td>BQ</td></tr><tr><td>AlexNet</td><td>2 / 3</td><td>4</td><td>1.2</td><td>1.39%</td><td>2.042</td><td>73.39%</td><td>BQ+WR</td></tr><tr><td>AlexNet</td><td>2 / 3</td><td>4</td><td>-</td><td>0.83%</td><td>2.025</td><td>74.72%</td><td>SQ</td></tr><tr><td>ResNet-18</td><td>32</td><td>32</td><td>-</td><td>-</td><td>-</td><td>76.07%</td><td>FP</td></tr><tr><td>ResNet-18</td><td>32</td><td>32</td><td>-</td><td>-</td><td>-</td><td>74.54%</td><td>FP(ReLU1)</td></tr><tr><td>ResNet-18</td><td>2 / -</td><td>4</td><td>-</td><td>0%</td><td>2</td><td>72.79%</td><td>TTQ*</td></tr><tr><td>ResNet-18</td><td>2 / 2</td><td>4</td><td>-</td><td>0.59%</td><td>2.0118</td><td>73.92%</td><td>BQ</td></tr><tr><td>ResNet-18</td><td>2 / 2</td><td>4</td><td>1.0</td><td>0.014%</td><td>2.0003</td><td>73.92%</td><td>BQ+WR</td></tr><tr><td>ResNet-18</td><td>2 / 2</td><td>4</td><td>-</td><td>0.0015%</td><td>2.00003</td><td>74.22%</td><td>SQ</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>4</td><td>-</td><td>7.34%</td><td>2.2203</td><td>74.07%</td><td>BQ</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>4</td><td>1.0</td><td>0.434%</td><td>2.013</td><td>74.05%</td><td>BQ+WR</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>4</td><td>-</td><td>0.321%</td><td>2.0096</td><td>74.67%</td><td>SQ</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>4</td><td>-</td><td>28.80%</td><td>3.1521</td><td>74.54%</td><td>BQ</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>4</td><td>1.0</td><td>4.995%</td><td>2.2</td><td>74.32%</td><td>BQ+WR</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>4</td><td>-</td><td>4.145%</td><td>2.166</td><td>74.84%</td><td>SQ</td></tr></table>

The results of both the top-1 accuracy and the average bit length on ImageNet dataset.

Table content: <table><tr><td>Model Name</td><td>Bit-W (b t /b sl )</td><td>Bit-A</td><td>λ 2</td><td>Percentage of SLW</td><td>Average Bit Length</td><td>Top-1 Accuracy</td><td>Remarks</td></tr><tr><td>ResNet-18</td><td>32</td><td>32</td><td>-</td><td>-</td><td>-</td><td>70.41%</td><td>FP</td></tr><tr><td>ResNet-18</td><td>32</td><td>32</td><td>-</td><td>-</td><td>-</td><td>62.51%</td><td>FP(ReLU1)</td></tr><tr><td>ResNet-18</td><td>2</td><td>32</td><td>-</td><td>0%</td><td>2</td><td>66.6%</td><td>TTQ</td></tr><tr><td>ResNet-18</td><td>2</td><td>4</td><td>-</td><td>0%</td><td>2</td><td>52.59%</td><td>TTQ*</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>4</td><td>-</td><td>1.11%</td><td>2.033</td><td>55.51%</td><td>BQ</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>4</td><td>0.25</td><td>0.84%</td><td>2.025</td><td>55.20%</td><td>BQ+WR</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>4</td><td>-</td><td>0.54%</td><td>2.016</td><td>57.67%</td><td>SQ</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>4</td><td>-</td><td>4.012%</td><td>2.161</td><td>58.43%</td><td>BQ</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>4</td><td>1</td><td>0.293%</td><td>2.012</td><td>56.70%</td><td>BQ+WR</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>4</td><td>-</td><td>0.269%</td><td>2.011</td><td>59.00%</td><td>SQ</td></tr><tr><td colspan="8">with only 0.55% loss in the average bit length. One notable point is that using ReLU1 activation</td></tr><tr><td colspan="8">causes large accuracy degradation, even the accuracy of TTQ (Zhu et al. 2016) is higher than the</td></tr><tr><td colspan="3">full precision model with ReLU1.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">4.3 ABLATION STUDY</td><td/><td/><td/><td/><td/><td/></tr></table>

The results of the weighted ridge (WR) regularizer on full precision models.

Table content: <table><tr><td>Model Name</td><td>Quantizaton Plan (b t /b sl )</td><td>λ 2</td><td>Top-1 Accuracy</td><td>Remarks</td></tr><tr><td>AlexNet</td><td>-</td><td>-</td><td>75.06%</td><td>FP</td></tr><tr><td>AlexNet</td><td>2 / 3</td><td>1.2</td><td>75.32%</td><td>FP+WR</td></tr><tr><td>ResNet-18</td><td>-</td><td>-</td><td>76.07%</td><td>FP</td></tr><tr><td>ResNet-18</td><td>2 / 2</td><td>1</td><td>76.49%</td><td>FP+WR</td></tr><tr><td>ResNet-18</td><td>2 / 3</td><td>1</td><td>76.28%</td><td>FP+WR</td></tr><tr><td>ResNet-18</td><td>2 / 4</td><td>1</td><td>76.88%</td><td>FP+WR</td></tr></table>

Classification accuracy of the state-of-the-art quantization methods trained on ImageNet with Resnet-18 model. Bit-W and Bit-A refer to the weights and activations bit-width respectively.

Table content: <table><tr><td>Model</td><td colspan="2">Bit-W Bit-A</td><td>Top-1 Accuracy</td></tr><tr><td>HWR (ours, b t /b sl = 2/3)</td><td>2/3</td><td>4</td><td>57.67%</td></tr><tr><td>HWR (ours, b t /b sl = 2/4)</td><td>2/4</td><td>4</td><td>59.00%</td></tr><tr><td>ABC-Net (Lin et al. 2017)</td><td>3</td><td>3</td><td>61.0%</td></tr><tr><td>PACT (Choi et al. 2018)</td><td>4</td><td>4</td><td>69.2%</td></tr><tr><td>QIL (Jung et al. 2019)</td><td>4</td><td>4</td><td>70.1%</td></tr></table>

Comparison of non-fixing the range of the quantization function (TTQ*) and fixing them (TTQ**).

Table content: <table><tr><td>Model Name</td><td colspan="2">Bit-W Bit-A</td><td>Top-1 Accuracy</td><td>Remarks</td></tr><tr><td>AlexNet</td><td>32</td><td>32</td><td>75.06%</td><td>FP</td></tr><tr><td>AlexNet</td><td>2</td><td>4</td><td>70.57%</td><td>TTQ*</td></tr><tr><td>AlexNet</td><td>2</td><td>4</td><td>72.04%</td><td>TTQ**</td></tr><tr><td>ResNet-18</td><td>32</td><td>32</td><td>76.07%</td><td>FP</td></tr><tr><td>ResNet-18</td><td>2</td><td>4</td><td>72.79%</td><td>TTQ*</td></tr><tr><td>ResNet-18</td><td>2</td><td>4</td><td>73.55%</td><td>TTQ**</td></tr></table>

The results of applying the scaling factor r, derived from the maximum absolute values of two inputs of add operations, to mediate two different intervals on ResNet-18 with full precision.

Table content: <table><tr><td>Model Name</td><td>Scaling</td><td>Activation function</td><td>λ 1</td><td>Top-1 Accuracy</td></tr><tr><td>ResNet-18</td><td>-</td><td>ReLU</td><td>0.0002</td><td>76.07%</td></tr><tr><td>ResNet-18 ResNet-18</td><td>m short mconv b r</td><td>ReLU ReLU</td><td>0.0002 0.0002</td><td>75.80% 76.15%</td></tr><tr><td>ResNet-18</td><td>-</td><td>ReLU1</td><td>0.0002</td><td>74.54%</td></tr><tr><td>ResNet-18 ResNet-18</td><td>m short mconv b r</td><td>ReLU1 ReLU1</td><td>0.0002 0.0002</td><td>74.16% 74.54%</td></tr></table>

Peer review (iclr2020):
This paper works on weight quantization in deep networks. The authors propose to utilize the extra state in 2-bit ternary representation to encode large weight values. The authors also propose to use a weighted ridge regularizer which contains a *part of L1* term to make the weights with large values sparse. The idea is simple and straightforward. However, the paper is not written very well with some typos and some terms defined unclearly. For instance, in the basic quantization method in Section 3.1, 1. What does RELU1 in the first paragraph mean? Clarification in the experiment section can be further improved. How are the activations for TTQ in section 4.2 quantized? The original TTQ paper also has results for ImageNet, how does the proposed method perform when compared with TTQ on ImageNet? One major concern is that some popular recent quantization methods are not compared. For instance, [1] also quantized both weights and activations. Can the proposed method outperform it? More comparison with these methods can better illustrate the efficacy of the proposed method. Another concern is that, though the proposed method has accuracy gain compared with the full-precision baseline and TTQ, the quantization becomes much more complex due to the usage of SLW, does the proposed quantization method cause extra burden to memory access and inference time? Others: 1. In Tables 3 and 4, *Top-1 Error* => *Top-1 Accuracy*? [1]. Choi, Jungwook, et al. *Pact: Parameterized clipping activation for quantized neural networks.* arXiv preprint arXiv:1805.06085 (2018). ---------- post-rebuttal comments----------- I really appreciate the authors for their detailed response and additional experiments. However, from the revised manuscript, the proposed method performs significantly worse than some recent quantization methods like PACT and QIL. Thus I keep my rating unchanged. --------------------------------------------------------