Hybrid Weight Representation: A Quantization Method Represented with Ternary and Sparse-Large Weights
Authors: Jinbae Park
Sung-Ho Bae
Venue:  (2020)
Keywords: quantized neural networks
centralized quantization
hybrid weight representation
weighted ridge
ternary weight

Abstract: Previous ternarizations such as the trained ternary quantization (TTQ), which quantized weights to three values (e.g., {−Wn,  0,+Wp}), achieved the small model size and efficient inference process. However, the extreme limit on the number of quantization steps causes some degradation in accuracy.  To solve this problem, we propose a hybrid weight representation (HWR) method which produces a network consisting of two types of weights, i.e., ternary weights (TW) and sparse-large weights (SLW). The TW is similar to the TTQ’s and requires three states to be stored in memory with 2 bits.  We utilize the one remaining state to indicate the SLW which is referred to as very rare and greater than TW. In HWR, we represent TW with values while SLW with indices of values.   By encoding SLW, the networks can preserve their model size with improving their accuracy. To fully utilize HWR, we also introduce a centralized quantization (CQ) process with a weighted ridge (WR) regularizer. They aim to reduce the entropy of weight distributions by centralizing weights toward ternary values.  Our comprehensive experiments show that HWR outperforms the state-of-the-art compressed models in terms of the trade-off between model size and accuracy.  Our proposed representation increased the AlexNet performance on CIFAR-100 by 4.15% with only1.13% increase in model size.

HYBRID WEIGHT REPRESENTATION: A QUANTIZA- TION METHOD REPRESENTED WITH TERNARY AND SPARSE-LARGE WEIGHTS
Anonymous authors Paper under double-blind review
ABSTRACT
Previous ternarizations such as the trained terarr quantization (TTQ), which quantized weights three values (e.g.. {-W,, 0, +W, achieved the small model size and efficient inference process. However, the extreme limit on the number of quantization steps causes some degradation in accuracy. To solve this problem, propose hybrid weight representation (HWR) method which pro- duces a network consisting two types of weights, i.c., ternary weights (TW) and sparse-large weights (SLW). The TW is similar the TTQ's and requires three states to be stored in memory with bits. We utilize the one remaining state to in- dicate the SLW which is referred to very rare and greater than TW. In HWR, we represent TW with values while SLW with indices of values by encoding SLW. As result, the networks can preserve their model size with improving their accuracy compared to ternary weights. To sparsify non-ternary weights in quantization, we also introduce centralized quantization (CQ) process with a weighted ridge (WR) regularizer. They aim to reduce the entropy of weight distributions by cen- tralizing weights toward ternary values. Our comprehensive experiments show the efficiency of HWR in terms of the trade-off between model size and accuracy.
INTRODUCTION
Deep Neural Networks have made considerable progress in various tasks such as image classification (LeCun et al. 1998 Simonyan & Zisserman 2014 Szegedy et al. 2015), object detection (Ren et al. 2015
Liu et al. 2016), and speech recognition (Graves al. 2013 Amodei al. 2016). However, outstanding neural networks usually require deeper and/or wider layers, thus making them hard deploy mobile and embedded devices. In response this problem, many studies their sights on more efficient networks. Various methods such as pruning (He et al. (217), light-weights (Howard al. 2017). and quantization (Courbariaux al. (0135 have been carried out to reduce the model size and/or computation complexity effectively.
In ternarization, the accuracy degradation is resulted from quantizing values in limited range with only 2bits. For example, the ternary weights networks (TWN, Li et al. 2016) yields only three quan- tized values, which prohibits the networks from utilizing high weight values. As known Han et al. 2015b large-valued weights tend to have an important role in the prediction. Therefore, the
Quantized Weights
Hybrid Weight Representation
Distribution
Components
Distribution
Bit Allocation
Centralized Quantization
Ternary Weights (2 bits)
+3:10
00
Sparso Lange
-1: 11
Weights (-2 hits
10
Figure 1: Comparison between conventional quantization and hybrid weight representation (HWR).

---

absence of large values poses a hybrid weight representation (HWR), expressing networks with both temary weights (TW) and sparse-lange weights (SLW). ). By taking the advantages of both the TW and SLW, the proposed HWR method can preserve their model size compared to ternary weights, as well as avoiding the accuracy degradation in networks.
the accuracy degradation. To solve this problem, our paper pro-
To be specific, the large values of SLW help networks to improve their accuracy. Furthermore, SLW can be encoded with one remaining state which is used to store TW in 2 bits representation. allows the networks to preserve their model size similarly to ternary weights. The compression rate of the encoding method is affected by the entropy of weight distributions. To train narrower distri- butions for the efficiency of HWR, we also introduce centralized quantization (CQ) process and a the differences between conventional quantization and HWR. As shown in Figure [] there is small number of SLW and the indices of encoded SLW are allocated in storage, unlike TW.
weighted ridge (WR) regularizer. Figure[]shove
We conduct various experiments, showing that HWR obtains better classification accuracy with the similar model size compared to the trained ternary quantization (TTQ, Zhu et al. 2016). which is baseline ternarization method. The experiments are carried out CIFAR-TOK 100 (Krizhevsky et al. (2009) and ImageNet (Russakovsky et al. (015).. We use AlexNet (Krizhevsky et al. 2012) and R8sNet-18 (He et al. (0166) baseline networks. Our proposed representation improves the AlexNet performance on CIFAR-100 by 4. 15% with only 1.13% increase in the model size.
The contributions of this paper are as follows:
We propose hybrid weight representation (HWR), including both values (TW) and indices of values (SLW). SLW allows the networks to improve their accuracy. Besides, the model size can be preserved by encoding SLW with one remaining state of 2 bits for TW.
We propose a training process, namely centralized quantization (CQ), improve the effi ciency of HWR. In CQ, we can sparsify almost large weights toward ternayy weights. The low entropy of the centralized distribution improves the compression rate of encoding.
We propose a regularizer, namely weighted ridge (WR), which gives more penalty to large weights. WR is utilized to centralize weights for narrower distributions and categorize the weights into TW and SLW.
2
RELATED WORK
Quantization In low precision training, one major difference from full precision training is that the conventional 32-bits weights (w) are discretized by a quantization funtion and represented with the finite number of elements. The discretized weights (w,) are multiplied by input matrices in the feed-forward pass. For example, the binarized neural networks (BNN, Courbariaux & Bengio[2016 utilized sign function quantize weights and activations to {-1, +1}. The binarized weights (w)) are defined as:
sign(x) -{
+1
if x
wb sign(w),
(-1 otherwise
In XNOR-Net (Rastegari et al. 2016). input data is even binarized. Furthermore, the multiplication and addition operations in convolution layers are replaced with XNOR and bit-count operations, respectively. In TWN (Li et al. 2016), the binarized weights pruned by thresholding
+WE
if A
A 0.7 E(ww) WE
Wt
if w S
SE(1]A<| (e)) (lw(1))
WE
if w wK<-A
In DoReFa-Net (Zhou et al. 2016), they carried out experiments in wider bit-width and also quan- tized the gradients. The quantized weights bit-width (wx) by a quantization function (Qx(.) are described as:
wg 2 Q+( Lanh(w) (maz([tank(r))] 0.5)
Qx(x)
round x- (2 1))
2

---

Full-precision model with ReLUI
iniialize
Applying weighted ridge
re-train
Centralized
Quantization
Applying
fine-ture
selective quantization
skip
Encoding
spurse-lurge weights
post-proecss
Figure 2: Overall processes of the hybrid weight representation (HWR). The weights are initialized from full precision model trained with ReLUI activation. First of the centralized quantization (CQ), the weights are quantized with the weight ridge (WR) as a regularizer. By WR, the quantized weights are also centralized under threshold and categorized into ternary weights (TW) and sparse- large weights (SLW) by threshold, Second, selective quantization (SQ) is applied to fine-tune the weights of the previous step. Finally, SLW is encoded by one usable state of TW prefix.
In back propagation, their is major concern caused by discretization of full precision weights (w). The derivatives of the discretizing functions such as sign(.) and Qx(-) have zero values at almost input ranges. Therefore, the gradients of w, calculated from the discretized weights (ws, Wr, and wg)), are also zero values and prevent w from optimizing. To solve this vanishing gradient problem, the serrigh--throuh estimator (STE) method (Hinton et al. 2012 Bengio et al. 2013) is proposed, where not calculating the gradients of w. other words, the gradients w &w replaced with the
(z2)
gradients of the discretized weights ( OL OL and aL iistead of back prok-propggating zero gradients from discretizing functions.
Swy Ow, 8ws.
0L
OL
8L
aL Sws
8h
Above methods are linear quantization which have the same intervals between adjoining quantum. By quantizing with the same intervals, the float operations can replaced to the integer operations.
Their are also non-linear quantization methods that the weights have irregular intervals. For in- stance, the deep compression (Han et al. 2015a) clustered the weights to quantize and fine-tuned the quantized weights of cach clustering group. In TTQ (Zhu et al. (2016). they quantized the weights to ternary values with two trainable scale coefficients for negative and positive weight values, i.c., {-Wn, 0, + Wpp) In these cases, the model size be significantly reduced by replacing the weight values with the indices them. However, the indices are needed to be transformed as weight values. And the irregular intervals make it difficult utilize integer-based and bit-wise operations.
Entropy coding The Huffman coding (Van Leeuwen 1976) aims to compress bit-streams in a loss- less manner by reducing bit-length of each element with an optimal prefix tree. The entropy of the weight distribution is one of the important factors that determine the compression rate in Huffman coding. Deep compression (Han et 2015a) pruned the weights to make lower entropy so saved more storage when applying the Huffman coding.
Regularization The regularization is utilized manipulate weights in artificial direction by impos- ing penalty. There are two conventional regularizers such as Ridge (L2) and Lasso (L1) (Han et al. 2015b). The L. weight decay usually prevents networks from being biased on the training dataset by restricting weights from growing. The penalty of Lasso makes the weights to be close to zero value which are unrelated to predict outputs during training-time. Moreover, the explicit loss of [Zhou al. 2018 makes it possible to quantize weights by controlling the strength of their regularizer. The penalty of regularization can be utilized to change the entropy of the weight distribution.
From previous studies, we focus on that ternary weights, linearly quantized, only require three states when being stored in 2bits, thus the remaining state has a potential to be utilized as prefix of more extended weights for improving efficiency in quantization. Furthermore, the regularization can help us to generate the lower entropy of the weight distribution, maximizing the compression efficiency when encoding the extended weights.
draw deduction that
3

---

3 METHOD
In this section, we explain: i) how centralized quantization (CQ) can centralize weights toward ternary values and categorize the weights into TW and SLW; and ii) how the quantized weights are encoded to be expressed as hybrid weight representation (HWR). The detailed processes are illustrated in Figure 2
3.1 BASIC QUANTIZATION METHOD
The basic quantization (BQ) uses round function to simply quantize the full precision weights (w)) The quantization function (Q (-)) of BQ fixed for each layer during training-time. Equation shows how w is quantized to Wy by (-).). The rng in Eq [ is the fixed range of w while low precision training, determined by the maximum absolute value of the pre-trained weights (Mw, mar((u, ((pp)))))) Before being entered into Q.,(),, is clipped to w. by rng to prevent misquantization of the round function. The (.) also requires the number quantization states (s) ascertained by the number of bits. To be specific about (-). the we is scaled by a float value to be discretized then it is restored by the reciprocal to the float value after the round function. Using STE, the derivative of we (dm.
AL is replaced with the derivative of Wy ( (am,)) By clipping wc,
0L
take saturation
effect on as in BNN (Courbariaux & Bengio 2016). But if some weights still over rng then w can be restated by clipping again after updating gradients.
W/g = w (We, rng, s) = round(w. 2
rgg e-1
We = ccip(w, -rng, rng) (1)
rng
To quantize activated values, restrict the range of activated values by using ReLU1 function as activation. The activated values with ReLUI (a) can be quantized to ag in k bit-width by a round function in Equation As shown in TTQ (Zhu et al. 2016), initializing with the pre-trained model helps the networks to improve their quantization performance. To take this advantage, our training starts with full precision model in which weights are pre-trained with ReLU1 activation.
ReLUI (x)) clip(x, 0, 1) = min(maax x), 1) 0g Qala, A)) = round(a a (2* 1)) / (2* -1) 1)
3.2 WEIGHTED RIDGE
We introduce a new version L2 weight decay, namely weighted ridge (WR), which aims to achieve two objectives: i) centralizing almost all weights toward below the threshold; and ii) categorizing the weights into TW and SLW by the threshold at the end of re-training.
(a) L2 and pL1
(b) WR
(c) the derivative of WR
Figure 3: Specifications weighted ridge (WR). (a) plots both a normal L2 and a part of L1 (pL1).
The WR is addition of them in (b). (c) is the derivative of WR.
In this step, we quantize full precision weights (w) by the basic quantization (BQ) with WR as a regularizer. This step's quantization function is defined using Equation [
Wig = Qu (clip(w, - Mu,)), M St + +81)
where the rng in Equation is the maximum absolute value of the pre- pre-trained weights (Mu, mar(ww,) ([wp))) fixed during re-trnnning. The in Equation D is derived from the number of bits of
SLW (ba)). If but is 2, 3, or bits then the number of quantization steps of SLW (sa)) is calculated to 4, 8, 16 while the number of bits and quantization steps of TW (b, and 8,) are fixed to 2 and 3.
4

---

Figure 3 (b) denotes WR which is mixture of both normal L2 and part of L1 (pL1) in (a). The loss of pL1 is only proportionate to the absolute value of the weight which is greater than the threshold (thresh). In other words, the large weights receive more penalty compared to the weights under thresh in pack-propagation as illustrated in Figure 3 the weights that do not have the necessity to keep their largeness. WR, pI.1, and thresh are defined:
(c). This penalty allows us to sparsify
W R(w, A1 A2) = X1 (12((t)) + X2 Ma, (ll((w)
(4)
pw]
|ll thresh if |w| > thresh if |w S thresh"
(thresh = Ma," 8t + Sal
8t
(5)
There are two hyper-parameters (A1 and A₂) which are tools control the intensity of WR. By adjusting the greater X2, more weights can be sparsified below thresh. The Me, is also a factor for applying different intensity to each layer since some layers have too small Ma, than others.
At the end of applying WR, we classify the re- trairid weights into TW and SLW. The thresh is utilized as a criterion for labeling them as Equation [
{
TW if |w| < thresh [ > thresh
mask(w)=
(6)
SLW
if
An observation is that WR tends to avoid overfitting as effectively as the L2 regularizer. By adjusting WR, we can obtain better accuracy than the L2 weight decay in some experiments as in Table 3
3.3 SELECTIVE QUANTIZATION
(clipped weightsl
We [clisped weignes) Eiquant weights)
Iquertiodd weights)
(a) mask(w) = TW
(b) mask(w) SLW
Figure 4: In the selective quantization (SQ), two different quantization functions are applied for each category (TW and SLW).
The selective quantization (SQ), the second step of CQ, aims to fine-tune the previous step's weights which are restricted by the additional penalty of the part of LI (pL1). By removing pL1, the re- stricted weights can grow again and be more optimized for classification. The key of SQ is that two different quantization functions are applied for each category. As shown in Figure (a), the weights in TW category are always ternarized, thus the percentage of TW is guaranteed during fine-tuning. Otherwise, the weights in SLW category can be quantized to either ternary values large values as Figure 4 (b). The quantization function of SQ (.) can be expressed as Equation 7 modified
from Equation 3 (The explanations for m. thresh, 8t, and 8ai are in Section 3.2). At the end
of fine-tuning, the fine-tuned weights are re-categorized by the same way as Equation 6 This fine-
tuning step can be skipped depending the result of SQ, if it yields only slight difference compared to the weighted ridge (WR).
(w) {
{
Qu [ml(eippw, (clip(w, -thresh, thresh), thresh, + 1) Qu (clip(w,, - -M,, Mw,), Muy. St + Sal) if massk(w= = SLW
if mask(w) TW
(7)
Specifically, we the stochastic gradient descent optimizer (SGD. Bottou|2100 in this fine-tuning step. The momentum optimizer (Sutskever et al. 2013) can misrepresent gradients since the weight values are clipped after updating gradients.
5

---

10011
10010
1001
10001
1000
10000
1010
10100
10101
10110
10111
b. 2, but = 2 9t 3, 8al 4
(b) b. 0at 3
8t 3. 8e1 8
Figure 5: Encoding trees of hybrid weight representation (HWR). In HWR, we represent TW with values while SLW with indices of values. The number of bits and quantization steps of TW (b., 8:) are fixed to 2, 3. contrast, them of SLW (bat, 8st) can be changed (2, 4), (3, 8), or (4, 16).
3.4 ENCODING SPARSE-LARGE WEIGHTS
The distributions centralizedd-quantied weights have low entropy since there are only small num- ber of SLW. It allows the networks to achieve higher compression rate by encoding SLW. We utilize one remaining state of TW as a prefix to encode SLW as shown in Figure As a result, the hybrid weight representation (HWR) method includes both the values (TW) and the indices (SLW) when the weights are allocated in memory.
There some advantages of using HWR. The model size can be maintained similar to TTQ ZZhu et al. 2016) since SLW is very rare. The restoration of encoded SLW is not that difficult. The value of SLW can be restored by taking the first sign bit of bat and adding 2 to the of bat However, there is a concern that HWR causes more complex and extra burden to memory access and inference time due the SLW. To minimize the negative effect of SLW, we suggest utilize the sparsity of SLW by gating them or applying variant of sparse matrix-vector multiplication. Then the TW takes up most of the weights and still has the potential to keep some benefits of ternary weights such as replacing multiplication operations to sign assignment operations (Rastegari et al. 2016) and utilizing gates for skipping zero weights (Deng et al. (018). More details about our proposed inference methods are in Appendices C
EXPERIMENT RESULTS
Model architecture We utilize AlexNet (Krizheskyy et al, 2012) and pre-activated ResNet-18 (He 2016) as bascline models, implemented on Tensorflow (Abadi et al. 2016) framework. In AlexNet, we use a variant of it by adding batch normalization (loffe & Szegedy 2015) layers and removing dropout layers. ResNet, we change two things: not using skip-connection for the first block of ResNet-18, since the first layer is not quantized and the not quantized weights cannot be entered into the add operations to keep integer- peegerr-recision stream while inference-time, and ii) mediating of two quantum intervals from convolution layers. In add operations, each input from two convolution layers has different quantum interval due to layer-wise quantization functions. The different intervals make it hard to keep integer-precision stream after add operations. More details of ResNet- 18 are explained in Appendices A
4.1 EXPERIMENTS ON CIFAR-100
Training setup We set the learning rate of AlexNet to 0.01 with decay 0.1 after every 100 epochs until 300 epochs and the L2 weight decay (A₁) to 0.02. We also set the learning rate ResNet-18 to 0.1 with the same decay of AlexNet and the A to 0.0002. For the dataset, we select CIFAR-100 Krizhevsky et al. 2009) and set the minibatch size to 128 to estimate the performance of networks. al. 2013)
We also use Nesterov Accelerated Gradients (Sutskever the fluctuation validation accuracy, we take a simple moving averages (sma) in 5 epochs during implementation.
optimizer. To prevent
training-time and average the over four repetitions. All experiments

---

Table 1: The results of both the top-1 accuracy and the average bit length in multiple bit-width. The full experiments of ResNet- 18 are listed in Appendices
Model
Bit-W (br/bat)
Percentage of SLW
Average Bit Length
Top-1 Accuracy
Bit-A
X2
Remarks
Name
AlexNet
32
32
75.06%
FP
AlexNet
32
32
74.78%
FP(ReLU1)
AlexNet
27-
0%
70.57%
TTQ
AlexNet
2/3
12.05%
2.362
73.54%
BQ
AlexNet
2/3
1.2
1.39%
2.042
73.39%
BQ+WR
AlexNet
2/3
0.83%
2.025
74.72%
SQ
ResNet-18
32
76.07%
FP
ResNet-18 18
32
32
74.54%
FP(ReLU1)
ResNet-18
2/-
4
0%
2
72.79%
TTQ
ResNet-18
2/2
0.59%
2.0118
73.92%
BQ
ResNet-18
2/2
1.0
0.014%
2.0003
73.92%
BQ+WR
ResNet-18 18
2/2
0.0015%
2.00003
74.22%
SQ
ResNet-18
273
4
7.34%
2.2203
74.07%
BQ
ResNet-18
2/3
1.0
0.434%
2.013
74.05%
BQ+WR
ResNet-18
2/3
0.321%
2.0096
74.67%
SQ
ResNet-18
274
28.80%
3.1521
74.54%
BQ
ResNet-18
2/4
1.0
4.995%
2.2
74.32%
BQ+WR
ResNet-18
2/4
4
4.145%
2.166
74.84%
SQ
Implementation Our experiments show the comparison between proposed methods (BQ, BQ with WR, and SQ in Section 3) and TTQ (Zhu et al. 2016) as baseline model. In weight quan- tization, the number of bits of TW (bx) is fixed to 2 while the number of bits of SLW (baz) can be changed to 2, 3, or 4. In activation quantization, we select 4 bits to quantize the activations. For the original TTQ paper, they did not quantize activated values. our implementation of TTQ (TTQ*), we quantize activations with the same quantization function in Equation 2 to make similar condi- tion. Also, our ResNet-18 18 architecture for TTQ* can be different from origian TTQ as written above. We quantize all the weights and activations, even the last activation before the global average pooling, except the weights of first and last layers and the outputs of soft-max layer.
In T the effect of the weighted ridge (WR) regularizer can be found out from the comparison of the basic quantization (BQ) and BQ with WR. The results of BQ+WR show more centralized distribution and less accuracy than only BQ. Furthermore, quantization (SQ) have more centralized distribution than BQ+WR and more improved accuracy than BQ, removing the part of I.1 in WR and fine-tuning the restricted weights by WR. After the encoding step, the difference When bat is for AlexNet, SQ reaches top-1 accuracy of 74.72%, which 4.15% higher than TTQ*, with only 1. 1.25% increase in the model size. In 188 when bat is 3, the top-1 accuracy of SQ is 1.88% higher than TTQ* with only 0.5% loss the model size.
observe that the results of the selective
the average bit length between SQ and TTQ* becomes similar.
4.2 EXPERIMENTS ON IMAGENET
Training setup We performed the experiments of ResNet--18 18 on ImageNet (Russakovsky et al. 20155) dataset. We set the learning rate to 0.1 with decay 0.1 after 25, 50, 75, and 80 epochs until 85 epochs and the minibatch size to 256. We use the momentum optimizer (Sutskever et al. 2013). We take a
simple moving average of validation error in epochs during training- time..
In Table the results BQ, BQ+WR, and SQ on ImageNet show a similar tendency of the results on CIFAR-100. The SQ (bat 4) reaches top-1 accuracy of 59%, which is 6.41% 41% higher than TTQ*,

---

Table The results of both the top-1 accuracy and the average bit length on ImageNet dataset.
Model Name
Bit-W (hr/bat)
Percentage of SLW
Average Bit Length
Top-1 Accuracy
Bit-A
A2
Remarks
ResNet-18
32
32
70.41%
HP
ResNet-18
32
32
62.51%
FP(RLLU1)
ResNer-18
2
32
0%
2
66.6%
TTQ
ResNet-18
2
0%
2
52.59%
TTQ*
ResNet-18
2/3
1.11%
2.033
55.51%
BQ
ResNet-18
18
2/3
4
0.25
0.84%
2.025
55.20%
BQ+WR
ResNet-18
2/3
4
0.54%
2.016
57.67 %
SQ
ResNet-18
2/4
4.012%
2.161
58.43%
BQ
ResNet-18
2/4
0.293%
2.012
56.70%
BQ+WR
ResNet-18
2/4
0.269%
2.011
59.00%
SQ
with only 0.55% loss in the average bit length. One notable point is that using ReLUU activation causes large accuracy degradation, even the accuracy TTQ (Zhu et al. 2016) higher than the full precision model with ReLUI.
4.3
ABLATION STUDY
Table 3: The results of the weighted ridge (WR) regularizer on full precision models.
Model Name
Quantizaton
Top-1 Accuracy
X2
Remarks
Plan (b./baa)
AlexNet
75.06%
FP
AlexNet
2/3
1.2
75.32% %
FP+WR
ResNet-18
76.07%
FP
ResNet-18
2/2
76.49% %
FP+WR
ResNet-18
2/3
1
76.28 %
FP+WR
ResNet-18
2/4
1
76.88% %
FP+WR
In this study, try to observe the effect of the weighted ridge (WR) regularizer full precision models as shown in Table 3 When A2 is set to 1, the penalty of the part of L1 (pL1) of WR calculated to the maximum absolute value of the pre-trained model (Mw,)) by the derivative of Equation 4 which is sufficiently large because the penalty M is same with the maximum penalty of L2 (Mw,)) The results of AlexNet and ResNet-18 with WR obtain 0.26% and 0.81% higher accuracy than the base-line models. The pL1 prevents the networks from overfitting as effectively as L2 regularizer. We conjecture that the biased large weights toward training dataset have more negatively effect on generalization than small weights. Therefore, the restriction of large weights enables us to make more generalized models. More ablation studies Appendices
DISCUSSION
5.1 TRADE-OFF BETWEEN ACCURACY AND MODEL SIZE
We perform more extended experiments in various Ay of WR to monitor the correlation between the percentage of SLW and the accuracy of ResNet-18 on CIFAR-100 The complete results listed in Appendices E, Figure shows that the intensity of pL1 (A2) in Equation 4 has less relation with the classification accuracy. In spite of the greater X2 some results show even better accuracy than
8

---

bybbg= states)
Accuracy
bybb== (11 states)
14.0
0y0by states)
W Biti states)
W Bit= (7 states)
Average Bit Length
Figure 6: The trade-off between accuracy and model size in various A2 1, 17 16)
lower 22. Therefore, we can perceive that many large weights in networks can be centralized with comparable accuracy, accompanied of generalization.
5.2 THE WEIGHT DISTRIBUTIONS
24
(a) BQ
(b) BQ+WR
(c) SQ
Figure 7: The weight distributions of each method.
As illustrated in Figure 7 the blue and red bars refer to the percentage of TW and SLW respectively. the SQ step, the forcibly restricted weights by WR can grow during fine-tuning, resulting less percentage of SLW than BQ+WR in Table From that, we assume that when weight starting to grow in fine-tuning begins to contribute more importantly for prediction than some other weights which have made similar contribution with the growing weight, the other weights become smaller in contrast. The most central weights of SLW BQ+WR are wide-spread after SQ step. This shows naturally centralized distributions. To make more flatten distributions after centralization, we suggest using another version of WR which utilize part of exponential lasso (pEL1, Breheny 2015) instead of pL1 written in Appendices D
that the penalty of pL1 cannot be proper make
5.3 ANALYSIS OF QUANTIZATION METHOD.
Maximum absolute value
0.4
FP(ReLU1)
BQ
12.5
0.3
SQ
7.5
0.2
0.1
2.5
shortcut1
shortcut2
shortcut3 Layer name
shortcut4
Figure Layer-wise comparison of the percentage of SLW and the maximum absolute value.
Figure displays the layer-wise maximum absolute value and percentage of SLW. one layer has many SLW after BQ then the layer tends to have relatively higher number of SLW after SQ. The maximum absolute values (max-abs) SQ tend to be larger than them of BQ since the restriction of the number of SLW causes more strong gradients for large weights. The max-abs of BQ are remarkably lower than the full precision model, used to fix the range of quantization functions at initialization. That shows problem that the basic quantization (BQ) method is not an effective way because some layers do not use the whole quantization states. For this reason, we need to apply more effective quantization methods instead of BQ. The following Table 4 lists other methods.

---

Table 4: Classification accuracy of the state-of-the-ar quantization methods trained on ImageNet with Resnet-18 18 model. Bit-W and Bit-A refer to the weights and activations bit-width respectively.
Top-1
Model
Bit-W
Bit-A
Accuracy
HWR (ours, be/bat 2/3)
2/3
4
57.67%
HWR (ours, b,/ /ba 2/4)
2/4
4
59.00%
ABC-Net (in et al.
2017)
3
61.0%
PACT Choi et al.
2018)
4
4
69.2%
QIL
Jung ct al.
2019)
4
4
70.1%
The state-of-the-art quantizations Table presents that our method HWR has less accuracy than other state-of-the-art quantization methods. There are two main reasons: i) more complex quanti- zation methods. For example, PACT (Choi et al. (018)) use parameterized clipping activation and QIR (Jung et al. 2019) use transformer before discretizing weights; ii) differences of detail of ResNet-18 architecture. As explained in Section we try to keep the
stream dur- ing inference-time. Especially, the skip-connections and add operations of ResNet disturb the stream to maintain integer-precision since two outputs from different convolution layers, having different quantization intervals, should be added.
To deal with that BQ has low performance, the state-of-the-art quantization methods can be harmo- nized with the centralized quantization (CQ) process. The full precision model for initialization can be replaced with the pre-quantized model trained by those methods. The quantization function of BQ can be replaced with the quantization function of the pre-quantized model.
6 CONCLUSION
We propose the hybrid weight representation (HWR), consisting of two types of weights such as the ternary weights (TW) and sparse-large weights(SLW), In HWR, we represent TW with values while SLW with indices of values. To represent SLW as indices, we encode SLW by utilizing one usable state as a prex, which is not used to store TW in 2 bits. To maximize the effect of encoding, we introduce the centralized quantization (CQ) process: i) applying the basic quantization (BQ) with the weighted ridge (WR) as a regularizer to centralize and categorize the weights; and ii) applying the selective quantization (SQ) to fine-tune the weights. After encoding process, the model size of HWR is similar with the TTQ even non non-ternary weights exist. Additionally, we propose a method to mediate two different quantum intervals of add operations in ResNet to keep the integer-precision stream while inference-tin time. As a result, SLW helps the networks to improve their accuracy while preserving their model size by encoding SLW compared to the 2- representation. We can achieve more efficient networks in terms of the trade-off between accuracy and model size.
7 FUTURE WORK
We consider three ways to extend our research: i) using more accurate quantization method instead of BQ. The pre-quantized weights and quantization functions from statc-of-the-art methods can be utilized for HWR; ii) using more bits for central weights. In mid-tread and symmetrical quantization, there is still one remaining state similarly with the ternary weights. Therefore, more experiments can be carried out to find the optimal number of bits for center and edge weights; and iii) verifying the performance of suggested inference methods. As written in Appendices C we suggest efficient inference methods utilizing the sparsity of SLW to minimize the extra burden caused by SLW.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, al. Tensorflow: Large-scale machine
10

---

learning heterogeneous distributed systems. arXiv preprint arXiv: 2016.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182, 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint urXiv:/308.344 2013.
Léon Bottou. scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'010,, pp. 177-186. Springer, 2010.
Patrick Breheny. The group exponential lasso for bi-level variable selection. Biometrics, 71(3): 731-740, 2015.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint
2018.
Matthieu Courbariaux and Yoshua Bengio. Binarynet:
Training deep neural networks with weights and activations constrained to -1. CoRR, abs/1602.02830, 2016. Ittt::///aa
org/ abd// 02830
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. CoRR, S.dx/551..00353 2015. URL http:
//arxiv.orq//////...ll....../tla abs 1511. 00363
Lei Deng, Peng Jiao, Jing Pei, Zhenzhi Wu, and Guoqi Li. Gxnor-net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework. Neural Networks, 100:49-58, 2018.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649, IEEE, 2013.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmman coding, arXiv preprint 1rXiv:1510.00149 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 143,, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645 Springer. 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14:8, 2012.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Mareo Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: arXiv:1704.04861, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1.02.003 150.0033.... 167, 2015.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4350 4359, 2019.
11

---

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. https://www: cs. toronto. edu/krizccifar html, 6, 2009.
Cifar- 10 and cifar- 1ffar-100 datasets.
URI:
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Fayez Lahoud, Radhakrishna Achanta, Pablo Márquez-Neila, and Sabine Süssttrukk. Self-binarizing networks. arXiv preprint arXiv:1902.007730 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (2:))22-2222 2324, 1998.
Fengfu Li, Bo Zhang, and Bin Liu. Ternryy weight networks. arXiv preprint arXiv: arXiv:1605.0471) 2016.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. arXiv preprint arXiv: 1608.01409, 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):2112255 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pp- 1139-1147, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Jan Van Leeuwen. On the construction of huffman trees. In ICALP, pp. 382-410, 1976.
Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit loss-error-aware quantization for low-bit deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9426-9435, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 16X1v:1606.06110, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.00066 2016.
12

---

A DETAILS OF RESNET
A.1 THE MEDIATION OF QUANTUM INTERVALS
200 activated input
BN&RRLU1
BNRRGLUI
I/shreer
BNERA.U
BNRRALU
blld:bcay-bb
bll2/eeavbb
Emky ]
(mh))
(m,)
mm) 22 1o
Foms
Tyi2 ((my.ma)
1922
Figure 9: Before add operations ResNet, the outputs of last convolution layers from each block scaled by scale factor r, derived from the maximum absolute values of convolution layers.
In our quantization method, the weights and activations are quantized in the same intervals (W, = tw Lw. ag in -.). The Lw and La are fixed float intervals while the ivr and ia are integer variables. Therefore, convolution and fully-connected layers can be inferred by integer operations as Equation Other operations such batch normalization (BN, loffe & Szegedy (2015). ReLU1 activation,
g
and the Q.(-) also can be integrated and compressed by integer comparators (Lahoud et al. 2019
Lin et al. 2017) since the coefficients of BN and boundaries of ReLUI and (.(+) are fixed so be prearranged.
09 layer(ay. Wg) = EΣ ug2 IIa ΣΣ iatu
However, ResNet (He et al, 2016) has identity mappings and add operations that prevent the data streams from keeping the same quantum intervals due to the layer-wise intervals. To deal with this problem, we scaled one input of add operations by scale factor r as defined Equation D The
derived from the maximum absolute values of the shortcut (Msshrrt) and last convolution (Mlconyy layers from each block as in Figure [ Since we use a log and floor function, r always in the range (0.5, 1] and it helps to keep the effect of identity mapping of ResNet. If is zero then nothing added the add operation or r is larger than then the shortcut can perturbed by a large residual.
Meones
loga
r f(mshorts Mconey)
Mshort
Mahort
(0.5 < r
Mconvs
By adjusting r, two different intervals from jth (shorteul) and kth (cones) convolution layers can be mediated and will allow us add two inputs with only shift operations (multiplication of multiple of 2) as Equation I0
I I
1oR2 P
add( d gr ot) Mohort (2
ΣΣ Hh +
ΣΣ
(10)
A.2 THE FIRST RESIDUAL BLOCK OF RESNET
In quantization, the first convolution layer is not quantized. Therefore, to prevent ResNet from having mixed precision output after add operations, our first residual block do not have a skip- connection even though the input shape of the block is same with the output shape of the first layer.
13

---

B ABLATION STUDY
B. FIXING THE RANGE OF THE QUANTIZATION FUNCTION
Table 5: Comparison of non-fixing the range of the quantization function (TTQ*) and fixing them (TTQ**).
Model Name
Top-1 Accuracy
Bit-W
Bit-A
Remarks
AlexNet
32
32
75.06%
FP
AlexNet
2
4
70.57%
TTQ*
AlexNet
2
4
72.04%
TTQ**
ResNet-18
32
32
76.07%
ResNet-18
2
72.79%
TTQ
ResNet-18
2
73.55%
TTQ**
To observe the effect of the fixed range of the basic quantization (BQ) method, we perform par- ticular experiment. The results of our implementation of TTQ (TTQ*) is performed with the quan- tization function of which the range not fixed. On the other hand, a version of implementation (TTQ**) have two different conditions compared to TTQ": i) fixing the range of the quantization function initialization; and ii) using one coefficient as {-W,. O, +W,). When applying them, the accuracy of AlexNet and ResNet-18 are rather improved by 1.47% and 0.76%. The results of TTQ** shows that the fixed range also can be utilized for quantization. Additionally, TTQ set constant to 0.05 for quantizing weights under t- maz(Jwl) as zero values. In our method, however, ba is 2, 3, or 4 bits then the is derived as 1 6. [ō or T8. All our are higher than 0.05 TTQ and it can cause worse results since 1.2 regularizer gives more penalty to the larger weights.
B.2 MEDIATION OF QUANTUM INTERVALS
Table The results applying the scaling factor r, derived from the maximum absolute values of two inputs of add operations, to mediate two different intervals on ResNet- 18 with full precision.
Model Name
Activation
Top-1 Accuracy
Scaling
A
function
ResNet-18
ReLU
0.0002
76.07%
ResNet-18
Mahort
ReLU
0.0002
75.80%
Mooney
ResNet-18
ReLU
0.0002
76.15%
ResNet-18
ReLU1
0.0002
74.54%
ResNet-18
Maharr
ReLUI
0.0002
74.16%
Meoney
ResNet-18
ReLU1
0.0002
74.54%
In Table 6 the experiments are conducted to observe the effect of the coefficient r, introduced in Appendices A. on the accuracy of ResNet-18. We compare three coefficients such zero, r in Equation and Mahor//MMonmn We observe that performs as similar as non-scaled models. As mentioned in Appendices A.I the Mshort/Mooomy greater than 1, can perturb the shortcut connec- tions. We apply r scaling for all our quantization experiments on ResNet-18.
C EFFICIENT INFERENCE METHODS
In this section, we suggest two methods for efficient inference HWR. Those methods utilize the sparsity of SLW to minimize the extra burden caused by SL.W and the benefit of ternary weights that
14

---

the multiplication of temnary weights can be replaced with the sign assignment operation (Rastegari et al. 2016).
C.I
GATED MULTIPLICATION
Sign sagat
Noc 07
TW
W
SLW
10s
Switch
MUTT
Σ
Decnder
Figure 10:
Computing architecture for HWR in inference time. According to the TW of each weight (W1), the operation for input (X₂) ) determined. There are two operations such sign assignment (sign asgmt) and integer multiplication (MUTT) for each type, TW and SLW.
In GXNOR-Net (Deng et 2018), they proposed gated XNOR architecture to take an advantage of binary weights when using ternary weights. If the input weight can be skipped while if they are not zero then the XNOR operation can be applied similarly with the XNOR-Net (Rastegari et 2016). In HWR case, we use enable pins as gates divide two cases of TW and SLW. As shown in Figure TO if the TW is 01, or 1l2 and input is not zero then only sign assignment operation (sign asgmt) is done while if the TW 10g then only integer multiplication (MUTT) is done. The sign-assigned or multiplied values are added at the end to be an output of one neuron.
then the multiplication
C.2 SEPARATED MULTTPPLICATION
X1
SpMV convolution
Sign asgint
Wetw
(Spera
W
Figure 11: Computing architecture for HWR in inference time. There are two weight matrices such as a dense matrix Wrw and sparse matrix Wstw. The original weight matrix can be restored by sum of Wrw and WSLw. In inference time, sign assignment operation is carried out with Wrw while sparse convolution is carried out with WeLw. The output of layer is sum of outputs from two operations.
In conventional layer, there is only weight matrix. On the other hand, divide the one quantized matrix (W₂) to two matrices such as a dense matrix Wrw and a sparse matrix W WSLw as in Equation
II
Wg Wrw + WSLW
(11)
The Wrw and WSLw can be described as:
Wrw = W if mask(w) = TW 0 otherwise
WSLW = { 0
otherwise
(12)
W
if mask(w) = SLW
15

---

In inference time, the multiplication of quantized activations (Ag) and quantized weights (W₂) also can be divided as:
A W, = A, Wrw + Ag WSLW
(13)
For multiplication of Ag and Wrw, the only sign assignment is required. For multiplication of Ag and W SLW, we can utilize sparse convolution operation (Park et al. 20T6) to reduce computational complexity. After applying each operation, two calculated results are added to be an output of layer.
D A PART OF EXPONENTIAL LASSO
(a) L2 and the pEL1
(b) the version of WR
(c) derivative of the version of WR
Figure 12: Another version WR, using the part of exponential lasso (pEL1) instead of pL1. (a) denotes both L.2 and PELI. The version of WR addition of them in (b). (c) the derivative of the version of WR.
In Figure 7 (b), the most central weights in SLW has particularly high probability than other weights in SLW. This is because the penalty of pl.1 prevents the weights from being large. the penalty of pl.1 for the weights, large enough for thresh, is reduced then the particularly centralized weights would grow well. For that, we suggest a version of WR which utilize part of exponential LI (pEL1, Breheny 2015) instead of pL1. The version of WR (WR*) and pEL1 can be defined as 15 The is an additional tuning parameter.
Equation [4
WR'(w, Å1 22) = A1(L2(w) + d2 Mw, pELL(w))
(14)
da (1 erp (- (lwl ((ulssthrssh))
if |w| > thresh
pELI(W)
(15)
if lo S thresh
E FULL EXPERIMENTS OF RESNET-18
Table shows the full version of our experiment results CIFAR-100 dataset with ResNet-18 model various X2.
16

---

Table 7: The experimental results of HWR and CQ in multiple bit-width and various A2.
Model
Bit-W (bz/ba)
Percentage of SLW
Average Bit Length
Top-1 Accuracy
Bit-A
X2
Remarks
Name
ResNet-18
32
32
76.07%
FP
ResNet-18 18
32
74.54%
FP(ReLU1)
ResNet-18
2/-
0%
2
72.79%
TTQ*
ResNet-18
2/2
0.59%
2.0118
73.92%
BQ
ResNet-18
2/2
0.0625
0.373%
2.0075
73.57%
BQ+WR(1)
ResNet-18
2/2
0.092%
2.0018
73.95%
SQ(1)
ResNet-18
2/2
0.125
0.198%
2.004
73.93%
BQ+WR(2)
ResNet-18
2/2
0.079%
2.0016
74.07% %
SQ(2)
ResNet-18
2/2
0.25
0.095%
2.002
73.73%
BQ+WR(3)
ResNet-18 18
2/2
0.049%
2.001
73.73%
SQ(3)
ResNet-18
212
0.5
0.023%
2.0004
74.11%
BQ+WR(4)
ResNet- 18
2/2
0.0068%
2.00013
73.95%
SQ(4)
ResNet-18
2/2
1.0
0.014%
2.0003
73.92%
BQ+WR(5)
ResNet-18 18
2/2
0.0015%
2.00003
74.22%
SQ(5)
ResNet- 18
2/2
4
2.0
0.0002%
2.000004
73.93%
BQ+WR(6)
ResNet-18
2/2
0.0004%
2.0+8e-7
74.08%
SQ(6)
ResNet-18
2/3
7.34%
2.2203
74.07%
BQ
ResNet-18
2/3
0.0625
4.955%
2.149
74.19%
BQ+WR(1)
ResNet-18 18
2/3
1.904%
2.057
74.23%
SQ(1)
ResNet-18
2/3
0.125
3.393%
2.102
74.05%
BQ+WR(2)
ResNet-18 18
2/3
1.563%
2.047
74.06%
SQ(2)
ResNet-18
2/3
0.25
2.11%
2.063
74.28%
BQ+WR(3)
ResNet-18
2/3
1.147%
2.034
74.52%
SQ (3)
ResNet-18
2/3
0.5
0.967%
2.029
74.1%
BQ+WR(4)
ResNet-18
2/3
0.7%
2.021
74.54%
SQ(4)
R8sNet-18
2/3
1.0
0.434%
2.013
74.05%
BQ+WR(5)
ResNet-18
2/3
0.321% %
2.0096
74.67%
SQ(5)
ResNer-
2/3
2.0
0.018%
2.0006
74.3%
BQ+WR(6)
ResNet-18
2/3
0.016%
2,0005
74.11%
SQ(6)
R8sNet-18
2/4
28.80%
3.1521
74.54%
BQ
ResNet-18
2/4
0.0625
20.67%
2.827
74.56%
BQ+WR(1)
ResNet-18
2/4
11.749%
2.47
74.67%
SQ(1)
ResNet-18
2/4
0.125
16.608%
2.664
74.28%
BQ+WR(2)
ResNet-18
2/4
10.395%
2,416
74.57 %
SQ(2)
ResNet- 18
2/4
0.25
12.38%
2,495
74.53%
BQ+WR(3)
R8
2/4
8.462%
2.338
74.78%
SQ(3)
ResNet-18
2/4
0.5
8.108%
2.324
74.26%
BQ+WR(4)
ResNer-18 18
2/4
5.485%
2,219
74.45%
SQ(4)
ResNet-18 18
2/4
1.0
4.995%
2.2
74.32%
BQ+WR(S)
ResNet-18 18
2/4
4.145%
2.166
74.84%
SQ(5)
ResNet-18
2/4
2.0
1.611%
2.064
74.06%
BQ+WR(6)
ResNet- 18
2/4
1.381%
2.055
74.73%
SQ(6)
17


Reviews:
[{"review_id": "SJlv2o_Jqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about quantization, and how to represent values as the finite number of states in a low bit width, using discretization. Particularly, they propose an approach to tackle the problems associated with previous ternarization which quantize weights to three values. Their approach is a hybrid weight representation method, which uses a network to output two weight types: ternary weight and sparse-large weights. For the ternary weight, they need 3 states to be stored with 2 bits. The one remaining state is used to indicate the sparse-large weight. They also propose an approach to centralize the weights towards ternary values. Their experiments show that their approach outperforms other compressed modeling approaches, and show an increase in AlexNet performance on the CIFAR-100 while increasing model size by 1.13%. \n\n- Overall, this is an interesting paper, offering a novel solution to tackle the degradation in accuracy occuring in ternary quantization techniques because of the number of quantization steps.  Their method seems technically sound, however I am not familiar with this area, so I would trust more the opinion of other reviewers - subject experts in the matter.\n- Their results on AlexNet and ResNet do show an improvement in terms of model accuracy, with only a slight increase in model size. They have also provided extensive experiments studying the benefits of their quantization method, the tradeoff among accuracy and model size. \n- I find the abbreviations, being used too often, to be confusing\n- The paper is targeted for a very focused audience, and does not give enough background for readers not familiar with \"ternarizations\".  Even the abstract could benefit from a motivation/ problem statement sentence, as well as less abbreviations being used. \n\nI would vote for acceptance of this paper, although it does seem a too-targeted paper on a specific audience. I would urge the authors to revise writing to make it broader accessible.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byxk-x1AFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper works on weight quantization in deep networks. The authors propose to utilize the extra state in 2-bit ternary representation to encode large weight values. The authors also propose to use a weighted ridge regularizer which contains a \"part of L1\" term to make the weights with large values sparse.\n\nThe idea is simple and straightforward. However, the paper is not written very well with some typos and some terms defined unclearly. For instance, in the basic quantization method in Section 3.1, 1. What does RELU1 in the first paragraph mean? \n\nClarification in the experiment section can be further improved. How are the activations for TTQ in section 4.2 quantized? The original TTQ paper also has results for ImageNet, how does the proposed method perform when compared with TTQ on ImageNet?\n\nOne major concern is that some popular recent quantization methods are not compared. For instance, [1] also quantized both weights and activations. Can the proposed method outperform it? More comparison with these methods can better illustrate the efficacy of the proposed method. \n\nAnother concern is that, though the proposed method has accuracy gain compared with the full-precision baseline and TTQ, the quantization becomes much more complex due to the usage of SLW, does the proposed quantization method cause extra burden to memory access and inference time?\n\nOthers:\n1. In Tables 3 and 4, \"Top-1 Error\" => \"Top-1 Accuracy\"?\n\n[1]. Choi, Jungwook, et al. \"Pact: Parameterized clipping activation for quantized neural networks.\" arXiv preprint arXiv:1805.06085 (2018).\n\n---------- post-rebuttal comments-----------\nI really appreciate the authors for their detailed response and additional experiments. However, from the revised manuscript, the proposed method performs significantly worse than some recent quantization methods like PACT and QIL. Thus I keep my rating unchanged.\n--------------------------------------------------------\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g3htmatr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a hybrid weights representation method where the weights of the neural network is split into two portions: a major portion of ternary weights and a minor portion of weights that are represented with different number of bits. The two portions of weights are differentiated by using the previous unused state of a typical ternary neural network since only three states are used out of the four states given 2-bit representation. The experiments are solid based on the selected baseline model on CIFAR-100 and Imagenet dataset.\n\nPros:\n\n•\tThe idea of using the previous unused state in ternary neural network is interesting\n•\tOverall, the paper is well written. The proposed method is presented clearly with proper graph illustration.\nCons:\n•\tThe idea of using mixed bit width for neural network quantization is not new. However, the experiments in the paper only compare with basic quantization method which makes the comparison not fair enough. For example, in ABC-net[1], a few full precision coefficients are used to binarize the network. With 3 bit for both weights and activations, it achieves 61% top 1 classification on ImageNet dataset with ResNet-18 as backbone model. This is around 3% higher than the paper’s proposed method with 2/4 bits for weights and 4 bits for activations. \n•\tIn the paper, it claims that the proposed weight ridge method “can obtain better accuracy than L2 weights decay”. However, there are no experiments or any theoretical supports for it.\n•\tAfter utilizing the forth state of a ternary neural network, it implies that all four states provided by 2 bit representation are used. Hence, the comparison with a quantized neural network of 2 bits should be given in the experiments also.\n\n[1] Lin, Xiaofan, Cong Zhao, and Wei Pan. \"Towards accurate binary convolutional neural network.\" Advances in Neural Information Processing Systems. 2017.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hybrid weighs representation method in deep networks. The authors propose to utilize the extra state in 2-bit ternary representation to encode large weight values. The idea is simple and straightforward. The main concern is on the experimental results. The use of mixed bit width for neural network quantization is not new, but the authors only compare with basic quantization method in the original submission. In the revised version of the paper, the proposed method performs significantly worse than recent quantization methods such as PACT and QIL. Moreover, writing can be improved, and parts of the paper need to be clarified.
Decision: Reject