HYBRID WEIGHT REPRESENTATION: A QUANTIZA-
Year: 2019

Abstract:
Previous ternarizations such as the trained ternary quantization (TTQ), which quantized weights to three values (e.g., {−Wn, 0, +Wp}), achieved the small model size and efficient inference process. However, the extreme limit on the number of quantization steps causes some degradation in accuracy. To solve this problem, we propose a hybrid weight representation (HWR) method which produces a network consisting of two types of weights, i.e., ternary weights (TW) and sparse-large weights (SLW). The TW is similar to the TTQ’s and requires three states to be stored in memory with 2 bits. We utilize the one remaining state to indicate the SLW which is referred to as very rare and greater than TW. In HWR, we represent TW with values while SLW with indices of values by encoding SLW. As a result, the networks can preserve their model size with improving their accuracy compared to ternary weights. To sparsify non-ternary weights in quantization, we also introduce a centralized quantization (CQ) process with a weighted ridge (WR) regularizer. They aim to reduce the entropy of weight distributions by centralizing weights toward ternary values. Our comprehensive experiments show the efficiency of HWR in terms of the trade-off between model size and accuracy.

heading: 1 INTRODUCTION
text: Deep Neural Networks have made considerable progress in various tasks such as image classification (LeCun et al. 1998, Simonyan & Zisserman 2014, Szegedy et al. 2015), object detection (Ren et al. 2015, Liu et al. 2016), and speech recognition (Graves et al. 2013, Amodei et al. 2016). However, outstanding neural networks usually require deeper and/or wider layers, thus making them hard to deploy on mobile and embedded devices. In response to this problem, many studies set their sights on more efficient networks. Various methods such as pruning (He et al. 2017), light-weights (Howard et al. 2017), and quantization (Courbariaux et al. 2015) have been carried out to reduce the model size and/or computation complexity effectively.
In ternarization, the accuracy degradation is resulted from quantizing values in a limited range with only 2bits. For example, the ternary weights networks (TWN, Li et al. 2016) yields only three quantized values, which prohibits the networks from utilizing high weight values. As known in Han et al. 2015b, large-valued weights tend to have an important role in the prediction. Therefore, the
absence of large values can cause the accuracy degradation. To solve this problem, our paper proposes a hybrid weight representation (HWR), expressing networks with both ternary weights (TW) and sparse-large weights (SLW). By taking the advantages of both the TW and SLW, the proposed HWR method can preserve their model size compared to ternary weights, as well as avoiding the accuracy degradation in networks.
To be specific, the large values of SLW help networks to improve their accuracy. Furthermore, SLW can be encoded with one remaining state which is not used to store TW in a 2 bits representation. It allows the networks to preserve their model size similarly to ternary weights. The compression rate of the encoding method is affected by the entropy of weight distributions. To train narrower distributions for the efficiency of HWR, we also introduce a centralized quantization (CQ) process and a weighted ridge (WR) regularizer. Figure 1 shows the differences between conventional quantization and HWR. As shown in Figure 1, there is a small number of SLW and the indices of encoded SLW are allocated in storage, unlike TW.
We conduct various experiments, showing that HWR obtains better classification accuracy with the similar model size compared to the trained ternary quantization (TTQ, Zhu et al. 2016), which is a baseline ternarization method. The experiments are carried out on CIFAR-100 (Krizhevsky et al. 2009) and ImageNet (Russakovsky et al. 2015). We use AlexNet (Krizhevsky et al. 2012) and ResNet-18 (He et al. 2016) as baseline networks. Our proposed representation improves the AlexNet performance on CIFAR-100 by 4.15% with only 1.13% increase in the model size.
The contributions of this paper are as follows:
• We propose a hybrid weight representation (HWR), including both values (TW) and indices of values (SLW). SLW allows the networks to improve their accuracy. Besides, the model size can be preserved by encoding SLW with one remaining state of 2 bits for TW.
• We propose a training process, namely centralized quantization (CQ), to improve the efficiency of HWR. In CQ, we can sparsify almost large weights toward ternary weights. The low entropy of the centralized distribution improves the compression rate of encoding.
• We propose a regularizer, namely weighted ridge (WR), which gives more penalty to large weights. WR is utilized to centralize weights for narrower distributions and categorize the weights into TW and SLW.

heading: 2 RELATED WORK
text: Quantization In low precision training, one major difference from full precision training is that the conventional 32-bits weights (w) are discretized by a quantization funtion and represented with the finite number of elements. The discretized weights (wq) are multiplied by input matrices in the feed-forward pass. For example, the binarized neural networks (BNN, Courbariaux & Bengio 2016) utilized a sign function to quantize weights and activations to {-1, +1}. The binarized weights (wb) are defined as:
wb = sign(w), sign(x) = { +1 if x ≥ 0 −1 otherwise
In XNOR-Net (Rastegari et al. 2016), input data is even binarized. Furthermore, the multiplication and addition operations in convolution layers are replaced with XNOR and bit-count operations, respectively. In TWN (Li et al. 2016), the binarized weights are pruned by thresholding as:
wt =
{ +WE if w > ∆
0 if |w| ≤ ∆ −WE if w < −∆
, ∆ = 0.7 · E(|w|)
WE = E i∈{i|∆<|w(i)|}
(|w(i)|)
In DoReFa-Net (Zhou et al. 2016), they carried out experiments in a wider bit-width and also quantized the gradients. The quantized weights in k bit-width (wk) by a quantization function (Qk(·)) are described as:
wk = 2 ·Qk( tanh(w)2max(|tanh(w)|) + 0.5)− 1 , Qk(x) = round(x·(2k−1)) 2k−1
In back-propagation, their is a major concern caused by discretization of full precision weights (w). The derivatives of the discretizing functions such as sign(·) and Qk(·) have zero values at almost input ranges. Therefore, the gradients of w, calculated from the discretized weights (wb, wt, and wk), are also zero values and prevent w from optimizing. To solve this vanishing gradient problem, the straight-through estimator (STE) method (Hinton et al. 2012, Bengio et al. 2013) is proposed, where not calculating the gradients ofw. In other words, the gradients ofw ( ∂L∂w ) is replaced with the gradients of the discretized weights ( ∂L∂wb , ∂L ∂wt
, and ∂L∂wk ) instead of back-propagating zero gradients from discretizing functions.
∂L ∂w = ∂L ∂wb = ∂L∂wt = ∂L ∂wk
Above methods are linear quantization which have the same intervals between adjoining quantum. By quantizing with the same intervals, the float operations can be replaced to the integer operations.
Their are also non-linear quantization methods that the weights have irregular intervals. For instance, the deep compression (Han et al. 2015a) clustered the weights to quantize and fine-tuned the quantized weights of each clustering group. In TTQ (Zhu et al. 2016), they quantized the weights to ternary values with two trainable scale coefficients for negative and positive weight values, i.e., {−Wn, 0, +Wp}. In these cases, the model size can be significantly reduced by replacing the weight values with the indices of them. However, the indices are needed to be transformed as weight values. And the irregular intervals make it difficult to utilize integer-based and bit-wise operations.
Entropy coding The Huffman coding (Van Leeuwen 1976) aims to compress bit-streams in a lossless manner by reducing bit-length of each element with an optimal prefix tree. The entropy of the weight distribution is one of the important factors that determine the compression rate in Huffman coding. Deep compression (Han et al. 2015a) pruned the weights to make lower entropy so saved more storage when applying the Huffman coding.
Regularization The regularization is utilized to manipulate weights in artificial direction by imposing penalty. There are two conventional regularizers such as Ridge (L2) and Lasso (L1) (Han et al. 2015b). The L2 weight decay usually prevents networks from being biased on the training dataset by restricting weights from growing. The penalty of Lasso makes the weights to be close to zero value which are unrelated to predict outputs during training-time. Moreover, the explicit loss of Zhou et al. 2018 makes it possible to quantize weights by controlling the strength of their regularizer. The penalty of regularization can be utilized to change the entropy of the weight distribution.
From previous studies, we focus on that ternary weights, linearly quantized, only require three states when being stored in 2bits, thus the remaining state has a potential to be utilized as a prefix of more extended weights for improving efficiency in quantization. Furthermore, we draw a deduction that the regularization can help us to generate the lower entropy of the weight distribution, maximizing the compression efficiency when encoding the extended weights.

heading: 3 METHOD
text: In this section, we explain: i) how centralized quantization (CQ) can centralize weights toward ternary values and categorize the weights into TW and SLW; and ii) how the quantized weights are encoded to be expressed as hybrid weight representation (HWR). The detailed processes are illustrated in Figure 2.

heading: 3.1 BASIC QUANTIZATION METHOD
text: The basic quantization (BQ) uses a round function to simply quantize the full precision weights (w). The quantization function (Qw(·)) of BQ is fixed for each layer during training-time. Equation 1 shows how w is quantized to wq by Qw(·). The rng in Eq 1 is the fixed range of w while low precision training, determined by the maximum absolute value of the pre-trained weights (Mwp = max(|wp|)). Before being entered into Qw(·), w is clipped to wc by rng to prevent misquantization of the round function. The Qw(·) also requires the number of quantization states (s) ascertained by the number of bits. To be specific aboutQw(·), thewc is scaled by a float value to be discretized then it is restored by the reciprocal to the float value after the round function. Using STE, the derivative of wc ( ∂L∂wc ) is replaced with the derivative of wq ( ∂L ∂wq
). By clipping w, we can take a saturation effect on w as in BNN (Courbariaux & Bengio 2016). But if some weights still over rng then w can be restated by clipping again after updating gradients.
wq = Qw(wc, rng, s) = round(wc · s−1 2
rng ) · rngs−1
2
, wc = clip(w, −rng, rng) (1)
To quantize activated values, we restrict the range of activated values by using ReLU1 function as activation. The activated values with ReLU1 (a) can be quantized to aq in k bit-width by a round function in Equation 2. As shown in TTQ (Zhu et al. 2016), initializing with the pre-trained model helps the networks to improve their quantization performance. To take this advantage, our training starts with a full precision model in which weights are pre-trained with ReLU1 activation.
a = ReLU1(x) = clip(x, 0, 1) = min(max(0, x), 1)
aq = Qa(a, k) = round(a · (2k − 1)) / (2k − 1) (2)

heading: 3.2 WEIGHTED RIDGE
text: We introduce a new version of L2 weight decay, namely weighted ridge (WR), which aims to achieve two objectives: i) centralizing almost all weights toward below the threshold; and ii) categorizing the weights into TW and SLW by the threshold at the end of re-training.
In this step, we quantize full precision weights (w) by the basic quantization (BQ) with WR as a regularizer. This step’s quantization function is defined using Equation 1:
wq = Qw(clip(w, −Mwp , Mwp), Mwp , st + ssl) (3) where the rng in Equation 1 is the maximum absolute value of the pre-trained weights (Mwp = max(|wp|)), fixed during re-training. The s in Equation 1 is derived from the number of bits of SLW (bsl). If bsl is 2, 3, or 4 bits then the number of quantization steps of SLW (ssl) is calculated to 4, 8, or 16 while the number of bits and quantization steps of TW (bt and st) are fixed to 2 and 3.
Figure 3 (b) denotes WR which is a mixture of both a normal L2 and a part of L1 (pL1) in (a). The loss of pL1 is only proportionate to the absolute value of the weight which is greater than the threshold (thresh). In other words, the large weights receive more penalty compared to the weights under thresh in back-propagation as illustrated in Figure 3 (c). This penalty allows us to sparsify the weights that do not have the necessity to keep their largeness. WR, pL1, and thresh are defined:
WR(w, λ1, λ2) = λ1(L2(w) + λ2 ·Mwp · pL1(w)) (4)
pL1(w) =
{ |w| − thresh if |w| > thresh
0 if |w| ≤ thresh , (thresh = Mwp · st
st + ssl − 1 ) (5)
There are two hyper-parameters (λ1 and λ2) which are tools to control the intensity of WR. By adjusting the greater λ2, more weights can be sparsified below thresh. The Mwp is also a factor for applying different intensity to each layer since some layers have too small Mwp than others.
At the end of applying WR, we classify the re-trained weights into TW and SLW. The thresh is utilized as a criterion for labeling them as Equation 6.
mask(w) = { TW if |w| ≤ thresh SLW if |w| > thresh (6)
An observation is that WR tends to avoid overfitting as effectively as the L2 regularizer. By adjusting WR, we can obtain better accuracy than the L2 weight decay in some experiments as in Table 3.

heading: 3.3 SELECTIVE QUANTIZATION
text: The selective quantization (SQ), the second step of CQ, aims to fine-tune the previous step’s weights which are restricted by the additional penalty of the part of L1 (pL1). By removing pL1, the restricted weights can grow again and be more optimized for classification. The key of SQ is that two different quantization functions are applied for each category. As shown in Figure 4 (a), the weights in TW category are always ternarized, thus the percentage of TW is guaranteed during fine-tuning. Otherwise, the weights in SLW category can be quantized to either ternary values or large values as Figure 4 (b). The quantization function of SQ (Qs(·)) can be expressed as Equation 7, modified from Equation 3 (The explanations for Mwp , thresh, st, and ssl are in Section 3.2). At the end of fine-tuning, the fine-tuned weights are re-categorized by the same way as Equation 6. This finetuning step can be skipped depending on the result of SQ, if it yields only slight difference compared to the weighted ridge (WR).
Qs(w) = { Qw(clip(w, −thresh, thresh), thresh, st + 1) if mask(w) = TW Qw(clip(w, −Mwp , Mwp), Mwp , st + ssl) if mask(w) = SLW
(7)
Specifically, we use the stochastic gradient descent optimizer (SGD, Bottou 2010) in this fine-tuning step. The momentum optimizer (Sutskever et al. 2013) can misrepresent gradients since the weight values are clipped after updating gradients.

heading: 3.4 ENCODING SPARSE-LARGE WEIGHTS
text: The distributions of centralized-quantized weights have low entropy since there are only small number of SLW. It allows the networks to achieve a higher compression rate by encoding SLW. We utilize one remaining state of TW as a prefix to encode SLW as shown in Figure 5. As a result, the hybrid weight representation (HWR) method includes both the values (TW) and the indices (SLW) when the weights are allocated in memory.
There are some advantages of using HWR. The model size can be maintained similar to TTQ (Zhu et al. 2016) since SLW is very rare. The restoration of encoded SLW is not that difficult. The value of SLW can be restored by taking the first sign bit of bsl and adding 2 to the rest of bsl. However, there is a concern that HWR causes more complex and extra burden to memory access and inference time due to the SLW. To minimize the negative effect of SLW, we suggest to utilize the sparsity of SLW by gating them or applying a variant of sparse matrix-vector multiplication. Then the TW takes up most of the weights and still has the potential to keep some benefits of ternary weights such as replacing multiplication operations to sign assignment operations (Rastegari et al. 2016) and utilizing gates for skipping zero weights (Deng et al. 2018). More details about our proposed inference methods are in Appendices C.

heading: 4 EXPERIMENT RESULTS
text: Model architecture We utilize AlexNet (Krizhevsky et al. 2012) and pre-activated ResNet-18 (He et al. 2016) as baseline models, implemented on Tensorflow (Abadi et al. 2016) framework. In AlexNet, we use a variant of it by adding batch normalization (Ioffe & Szegedy 2015) layers and removing dropout layers. In ResNet, we change two things: i) not using skip-connection for the first block of ResNet-18, since the first layer is not quantized and the not quantized weights cannot be entered into the add operations to keep integer-precision stream while inference-time; and ii) mediating of two quantum intervals from convolution layers. In add operations, each input from two convolution layers has different quantum interval due to layer-wise quantization functions. The different intervals make it hard to keep integer-precision stream after add operations. More details of ResNet-18 are explained in Appendices A.

heading: 4.1 EXPERIMENTS ON CIFAR-100
text: Training setup We set the learning rate of AlexNet to 0.01 with decay 0.1 after every 100 epochs until 300 epochs and the L2 weight decay (λ1) to 0.02. We also set the learning rate of ResNet-18 to 0.1 with the same decay of AlexNet and the λ1 to 0.0002. For the dataset, we select CIFAR-100 Krizhevsky et al. 2009) and set the minibatch size to 128 to estimate the performance of networks. We also use Nesterov Accelerated Gradients (Sutskever et al. 2013) as an optimizer. To prevent the fluctuation of validation accuracy, we take a simple moving averages (sma) in 5 epochs during training-time and average the sma over four repetitions. All experiments are our implementation.
Implementation Our experiments show the comparison between our proposed methods (BQ, BQ with WR, and SQ in Section 3) and TTQ (Zhu et al. 2016) as a baseline model. In weight quantization, the number of bits of TW (bt) is fixed to 2 while the number of bits of SLW (bsl) can be changed to 2, 3, or 4. In activation quantization, we select 4 bits to quantize the activations. For the original TTQ paper, they did not quantize activated values. In our implementation of TTQ (TTQ*), we quantize activations with the same quantization function in Equation 2 to make similar condition. Also, our ResNet-18 architecture for TTQ* can be different from the origian TTQ as written above. We quantize all the weights and activations, even the last activation before the global average pooling, except the weights of first and last layers and the outputs of soft-max layer.
In Table 1, the effect of the weighted ridge (WR) regularizer can be found out from the comparison of the basic quantization (BQ) and BQ with WR. The results of BQ+WR show more centralized distribution and less accuracy than only BQ. Furthermore, we observe that the results of the selective quantization (SQ) have more centralized distribution than BQ+WR and more improved accuracy than BQ, removing the part of L1 in WR and fine-tuning the restricted weights by WR. After the encoding step, the difference of the average bit length between SQ and TTQ* becomes similar. When bsl is 3 for AlexNet, SQ reaches top-1 accuracy of 74.72%, which is 4.15% higher than TTQ*, with only 1.25% increase in the model size. In ResNet-18, when bsl is 3, the top-1 accuracy of SQ is 1.88% higher than TTQ* with only 0.5% loss in the model size.

heading: 4.2 EXPERIMENTS ON IMAGENET
text: Training setup We performed the experiments of ResNet-18 on ImageNet (Russakovsky et al. 2015) dataset. We set the learning rate to 0.1 with decay 0.1 after 25, 50, 75, and 80 epochs until 85 epochs and the minibatch size to 256. We use the momentum optimizer (Sutskever et al. 2013). We take a simple moving average of validation error in 5 epochs during training-time.
In Table 2, the results of BQ, BQ+WR, and SQ on ImageNet show a similar tendency of the results on CIFAR-100. The SQ (bsl = 4) reaches top-1 accuracy of 59%, which is 6.41% higher than TTQ*,
with only 0.55% loss in the average bit length. One notable point is that using ReLU1 activation causes large accuracy degradation, even the accuracy of TTQ (Zhu et al. 2016) is higher than the full precision model with ReLU1.

heading: 4.3 ABLATION STUDY
text: In this study, we try to observe the effect of the weighted ridge (WR) regularizer on full precision models as shown in Table 3. When λ2 is set to 1, the penalty of the part of L1 (pL1) of WR is calculated to the maximum absolute value of the pre-trained model (Mwp ) by the derivative of Equation 4, which is sufficiently large because the penalty Mwp is same with the maximum penalty of L2 (Mwp ). The results of AlexNet and ResNet-18 with WR obtain 0.26% and 0.81% higher accuracy than the base-line models. The pL1 prevents the networks from overfitting as effectively as L2 regularizer. We conjecture that the biased large weights toward training dataset have more negatively effect on generalization than small weights. Therefore, the restriction of large weights enables us to make more generalized models. More ablation studies are in Appendices B.

heading: 5 DISCUSSION

heading: 5.1 TRADE-OFF BETWEEN ACCURACY AND MODEL SIZE
text: We perform more extended experiments in various λ2 of WR to monitor the correlation between the percentage of SLW and the accuracy of ResNet-18 on CIFAR-100. The complete results are listed in Appendices E. Figure 6 shows that the intensity of pL1 (λ2) in Equation 4 has less relation with the classification accuracy. In spite of the greater λ2, some results show even better accuracy than
lower λ2. Therefore, we can perceive that many large weights in networks can be centralized with comparable accuracy, accompanied of generalization.

heading: 5.2 THE WEIGHT DISTRIBUTIONS
text: As illustrated in Figure 7, the blue and red bars refer to the percentage of TW and SLW respectively. In the SQ step, the forcibly restricted weights by WR can grow during fine-tuning, resulting in less percentage of SLW than BQ+WR as in Table 1. From that, we assume that when a weight starting to grow in fine-tuning begins to contribute more importantly for prediction than some other weights which have made a similar contribution with the growing weight, the other weights become smaller in contrast. The most central weights of SLW in BQ+WR are wide-spread after SQ step. This shows that the penalty of pL1 cannot be proper to make more naturally centralized distributions. To make more flatten distributions after centralization, we suggest using another version of WR which utilize a part of exponential lasso (pEL1, Breheny 2015) instead of pL1 as written in Appendices D.

heading: 5.3 ANALYSIS OF QUANTIZATION METHOD.
text: Figure 8 displays the layer-wise maximum absolute value and percentage of SLW. If one layer has many SLW after BQ then the layer tends to have a relatively higher number of SLW after SQ. The maximum absolute values (max-abs) of SQ tend to be larger than them of BQ since the restriction of the number of SLW causes more strong gradients for large weights. The max-abs of BQ are remarkably lower than the full precision model, used to fix the range of quantization functions at initialization. That shows a problem that the basic quantization (BQ) method is not an effective way because some layers do not use the whole quantization states. For this reason, we need to apply more effective quantization methods instead of BQ. The following Table 4 lists other methods.
The state-of-the-art quantizations Table 4 presents that our method HWR has less accuracy than other state-of-the-art quantization methods. There are two main reasons: i) more complex quantization methods. For example, PACT (Choi et al. 2018) use parameterized clipping activation and QIR (Jung et al. 2019) use a transformer before discretizing weights; ii) differences of detail of ResNet-18 architecture. As explained in Section 4, we try to keep the integer-precision stream during inference-time. Especially, the skip-connections and add operations of ResNet can disturb the stream to maintain integer-precision since two outputs from different convolution layers, having different quantization intervals, should be added.
To deal with that BQ has low performance, the state-of-the-art quantization methods can be harmonized with the centralized quantization (CQ) process. The full precision model for initialization can be replaced with the pre-quantized model trained by those methods. The quantization function of BQ can be replaced with the quantization function of the pre-quantized model.

heading: 6 CONCLUSION
text: We propose the hybrid weight representation (HWR), consisting of two types of weights such as the ternary weights (TW) and sparse-large weights(SLW). In HWR, we represent TW with values while SLW with indices of values. To represent SLW as indices, we encode SLW by utilizing one usable state as a prex, which is not used to store TW in 2 bits. To maximize the effect of encoding, we introduce the centralized quantization (CQ) process: i) applying the basic quantization (BQ) with the weighted ridge (WR) as a regularizer to centralize and categorize the weights; and ii) applying the selective quantization (SQ) to fine-tune the weights. After encoding process, the model size of HWR is similar with the TTQ even non-ternary weights exist. Additionally, we propose a method to mediate two different quantum intervals of add operations in ResNet to keep the integer-precision stream while inference-time. As a result, SLW helps the networks to improve their accuracy while preserving their model size by encoding SLW compared to the 2-bit representation. We can achieve more efficient networks in terms of the trade-off between accuracy and model size.

heading: 7 FUTURE WORK
text: We consider three ways to extend our research: i) using more accurate quantization method instead of BQ. The pre-quantized weights and quantization functions from state-of-the-art methods can be utilized for HWR; ii) using more bits for central weights. In mid-tread and symmetrical quantization, there is still one remaining state similarly with the ternary weights. Therefore, more experiments can be carried out to find the optimal number of bits for center and edge weights; and iii) verifying the performance of suggested inference methods. As written in Appendices C, we suggest efficient inference methods utilizing the sparsity of SLW to minimize the extra burden caused by SLW.

heading: A DETAILS OF RESNET
text: A.1 THE MEDIATION OF QUANTUM INTERVALS
In our quantization method, the weights and activations are quantized in the same intervals (wq = iw ·Iw, aq = ia ·Ia). The Iw and Ia are fixed float intervals while the iw and ia are integer variables. Therefore, convolution and fully-connected layers can be inferred by integer operations as Equation 8. Other operations such as batch normalization (BN, Ioffe & Szegedy 2015), ReLU1 activation, and the Qa(·) also can be integrated and compressed by integer comparators (Lahoud et al. 2019, Lin et al. 2017) since the coefficients of BN and boundaries of ReLU1 and Qa(·) are fixed so can be prearranged.
oq = layer(aq, wq) = ∑∑ aqwq = IaIw · ∑∑ iaiw (8)
However, ResNet (He et al. 2016) has identity mappings and add operations that prevent the data streams from keeping the same quantum intervals due to the layer-wise intervals. To deal with this problem, we scaled one input of add operations by a scale factor r as defined in Equation 9. The r is derived from the maximum absolute values of the shortcut (mshort) and last convolution (mconvb ) layers from each block as in Figure 9. Since we use a log and floor function, r is always in the range (0.5, 1] and it helps to keep the effect of identity mapping of ResNet. If r is zero then nothing is added in the add operation or if r is larger than 1 then the shortcut can be perturbed by a large residual.
r = f(mshort,mconvb) = mshort mconvb · 2
⌊ log2
mconvb mshort ⌋ , (0.5 < r ≤ 1) (9)
By adjusting r, two different intervals from jth (shortcut) and kth (convb) convolution layers can be mediated and it will allow us to add two inputs with only shift operations (multiplication of multiple of 2) as Equation 10.
add(ojq, o k q ) = mshort · (2
− log2 Ika Ikw IjaI j w  · ∑∑
ijai j w + ∑∑ ikai k w) (10)
A.2 THE FIRST RESIDUAL BLOCK OF RESNET
In our quantization, the first convolution layer is not quantized. Therefore, to prevent ResNet from having mixed precision output after add operations, our first residual block do not have a skipconnection even though the input shape of the block is same with the output shape of the first layer.

heading: B ABLATION STUDY
text: B.1 FIXING THE RANGE OF THE QUANTIZATION FUNCTION
To observe the effect of the fixed range of the basic quantization (BQ) method, we perform a particular experiment. The results of our implementation of TTQ (TTQ*) is performed with the quantization function of which the range is not fixed. On the other hand, a version of implementation (TTQ**) have two different conditions compared to TTQ*: i) fixing the range of the quantization function at initialization; and ii) using one coefficient as {-Ws, 0, +Ws}. When applying them, the accuracy of AlexNet and ResNet-18 are rather improved by 1.47% and 0.76%. The results of TTQ** shows that the fixed range also can be utilized for quantization. Additionally, TTQ set a constant t to 0.05 for quantizing weights under t ·max(|w|) as zero values. In our method, however, if bsl is 2, 3, or 4 bits then the t is derived as 16 , 1 10 , or 1 18 . All our t are higher than 0.05 of TTQ and it can cause worse results since L2 regularizer gives more penalty to the larger weights.
B.2 MEDIATION OF QUANTUM INTERVALS
In Table 6, the experiments are conducted to observe the effect of the coefficient r, introduced in Appendices A.1, on the accuracy of ResNet-18. We compare three coefficients such as zero, r in Equation 9, and mshort/mconvb . We observe that r performs as similar as non-scaled models. As mentioned in Appendices A.1, the mshort/mconvb , greater than 1, can perturb the shortcut connections. We apply r scaling for all our quantization experiments on ResNet-18.

heading: C EFFICIENT INFERENCE METHODS
text: In this section, we suggest two methods for efficient inference of HWR. Those methods utilize the sparsity of SLW to minimize the extra burden caused by SLW and the benefit of ternary weights that
the multiplication of ternary weights can be replaced with the sign assignment operation (Rastegari et al. 2016).
C.1 GATED MULTIPLICATION
In GXNOR-Net (Deng et al. 2018), they proposed gated XNOR architecture to take an advantage of binary weights when using ternary weights. If the input or weight are zero then the multiplication can be skipped while if they are not zero then the XNOR operation can be applied similarly with the XNOR-Net (Rastegari et al. 2016). In HWR case, we use enable pins as gates to divide two cases of TW and SLW. As shown in Figure 10, if the TW is 012 or 112 and input is not zero then only sign assignment operation (sign asgmt) is done while if the TW is 102 then only integer multiplication (MUTT) is done. The sign-assigned or multiplied values are added at the end to be an output of one neuron.
C.2 SEPARATED MULTIPLICATION
In conventional layer, there is only one weight matrix. On the other hand, we can divide the one quantized matrix (Wq) to two matrices such as a dense matrix WTW and a sparse matrix WSLW as in Equation 11.
Wq = WTW +WSLW (11)
The WTW and WSLW can be described as:
WTW = { Wq if mask(w) = TW 0 otherwise , WSLW = { 0 otherwise Wq if mask(w) = SLW
(12)
In inference time, the multiplication of quantized activations (Aq) and quantized weights (Wq) also can be divided as:
Aq ·Wq = Aq ·WTW +Aq ·WSLW (13)
For multiplication of Aq and WTW , the only sign assignment is required. For multiplication of Aq and WSLW , we can utilize sparse convolution operation (Park et al. 2016) to reduce computational complexity. After applying each operation, two calculated results are added to be an output of a layer.

heading: D A PART OF EXPONENTIAL LASSO
text: In Figure 7 (b), the most central weights in SLW has a particularly high probability than other weights in SLW. This is because the penalty of pL1 prevents the weights from being large. If the penalty of pL1 for the weights, large enough for thresh, is reduced then the particularly centralized weights would grow well. For that, we suggest a version of WR which utilize a part of exponential L1 (pEL1, Breheny 2015) instead of pL1. The version of WR (WR*) and pEL1 can be defined as Equation 14, 15. The τ is an additional tuning parameter.
WR∗(w, λ1, λ2) = λ1(L2(w) + λ2 ·Mwp · pEL1(w)) (14)
pEL1(w) =
{ λ2 τ { 1− exp ( − τλ2 · (|w| − thresh) )} if |w| > thresh
0 if |w| ≤ thresh (15)

heading: E FULL EXPERIMENTS OF RESNET-18
text: Table 7 shows the full version of our experiment results on CIFAR-100 dataset with ResNet-18 model in various λ2.

Review summaries:
The paper proposes a hybrid weights representation method where the weights of the neural network is split into two portions: a major portion of ternary weights and a minor portion of weights that are represented with different number of bits. The two portions of weights are differentiated by using the previous unused state of a typical ternary neural network since only three states are used out of the four states given 2-bit representation. The experiments are solid based on the selected baseline model on CIFAR-100 and Imagenet dataset.

This paper is about quantization, and how to represent values as the finite number of states in a low bit width, using discretization. Particularly, they propose an approach to tackle the problems associated with previous ternarization which quantize weights to three values. Their approach is a hybrid weight representation method, which uses a network to output two weight types: ternary weight and sparse-large weights. For the ternary weight, they need 3 states to be stored with 2 bits. The one remaining state is used to indicate the sparse-large weight. They also propose an approach to centralize the weights towards ternary values. Their experiments show that their approach outperforms other compressed modeling approaches, and show an increase in AlexNet performance on the CIFAR-100 while increasing model size by 1.13%.