Post-Training Weighted Quantization of Neural Networks for Language Models
Authors: Se Jung Kwon
Dongsoo Lee
Yongkweon Jeon
Byeongwook Kim
Bae Seong Park
Yeonju Ro
Venue:  (2021)
Keywords: Model Compression
Non-uniform Quantization
Post-training Quantization
Language Model

Abstract: As a practical model compression technique, parameter quantization is effective especially for language models associated with a large memory footprint. Neural network quantization is usually performed to reduce quantization loss assuming that quantization error of each parameter equally contributes to the overall training loss. The importance of each parameter, however, may highly differ such that for the same number of quantization bits, certain parameters lead to higher training loss than the others after quantization. In this paper, we consider a non-uniform quantization scheme, specifically binary-coding-based quantization, for high compression ratio and efficient computations while avoiding large accuracy degradation by uniform quantization (e.g., INT8). Then, we derive quantization optimization methods to take into account the importance of each parameter. We demonstrate that for post-training quantization, weight magnitude can represent importance and improve model accuracy significantly compared to the previous schemes lacking importance considerations. For various language models including BERT, DistilBERT, AWD-LSTM, and Transformer, we achieve 2-4 bits per weight by our proposed post-training quantization with reasonable accuracy degradation.

Under review as a conference paper at ICLR 2021
POST-TRAINING WEIGHTED QUANTIZATION OF NEURAL NETWORKS FOR LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
As practical model compression technique, parameter quantization is effective especially for language models associated with large memory footprint. Neural network quantization is usually performed to reduce quantization loss assuming that quantization error of each parameter equally contributes to the overall training loss. The importance of each parameter, however, may highly differ such that for the same number of quantization bits, certain parameters lead to higher training loss than the others after quantization. In this paper, consider non-uniform quantization scheme, specifically binary-coding-bast basedd quantization, for high com- pression ratio and efficient computations while avoiding large accuracy degradation by uniform quantization (e.g.. INT8). Then, we derive quantization optimization methods to take into account the importance of each parameter. We demonstrate that for post-training quantization, weight magnitude can represent importance and improve model accuracy significantly compared the previous schemes lacking importance considerations. For various language models including BERT, Dis- tilBERT, AWD-LSTM, and Transformer, we achieve 2-4 bits per weight by our proposed post-training quantization with reasonable accuracy degradation.
INTRODUCTION
Training techniques for deep neural networks (DNNs) have been developed in ways to incur a lot of
parameter redundancy to expedite seeking local minima (Denil al. [013) Jonathan Frankle (019))
As result, various model compression techniques including parameter pruning (Han et 2015 He
al. 2017). quantization (Courbariaux et al. 2015 Rastegari et al. 2016). low-rank approximation
(N. Sainath et al.] 2013 Prrabhavakkar et al.) 2016), and knowledge distillation (Hinton et al.) 2015
Polino et al. 2018) are proposed to lower storage requirements and improve inference performance.
Several compression techniques can be combined in synergistic way to enhance compression ratio (Han et al. 2016 Zhu et al. 2017). In this work, we consider parameter quantization that maintains structured model formats and presents high compression ratio. Note that due to limited hardware resources, quantization is an essential method for any inference systems. In general, quantization is classified into uniform quantization based fixed-point parameter representations (Jacob et al.
2018
Han et al., (2016) and non- non-uniform quantization associated with the binary codes (Zhou et al. Rastegari al.) 2016) or codebooks (Choi et al. 2017 Stock et al.) 2020).
2017
Most DNN quantization methods are performed based on the principle of minimizing the mean squared error (MSE) of quantized parameters (Rastegari et al., 2016; Xu et al., 2018 Zhou al. 2017). Optimizing the MSE is also an underlying principle of low-rank approximation techniques such the singular value decomposition (SVD) (Prabhavalkar al., 2016 N. Sainath et al.) 2013). Note that, however, minimizing the MSE implies that each parameter is equally important (i.e., squared errors from parameters are accumulated without considering importance of each weight). In practice, the impact of each parameter perturbation from quantization training loss can be vastly different and such impact needs to be analyzed through a sensitivity study of each parameter toward change in training loss value. In other words, minimizing the MSE (or the Euclidean distance between original parameters and quantized parameters) may not correspond to minimizing training loss function after quantization.
Robustness to quantization error of each parameter can be expressed as sensitivity. Sensitivity of i-th parameter w; is the amount of change in the loss function when w; is perturbed. A parameter

---

Under review as a conference paper at ICLR 2021
associated with high sensitivity would require relatively smaller quantization error if quantization is performed in a group manner. Several previous works acknowledge distinct sensitivity of each param- eter to improve quantization quality. Note that because exact sensitivity estimation of each parameter toward loss function is highly complicated, various heuristic techniques have been introduced. For example, Hessian-weighted k-means clustering is used for rodebook-hased implementations (Choi) ict al., (017)) or Taylor series expansion to bound loss function difference is conducted to decide the optimal quantization bits of each weight (Khoram & Li 2018). The Hessian matrix can be used to assign different numbers of quantization bits for each layer (Dong et al., 2019; Shhn et al. 2019p. Minimizing the reconstruction error on the output activations after each layer quantization is performed in (Stock et al. 2020
In this paper, we propose a weighted quantization framework where quantized parameters follow the structure of the binary codes so as to achieve high compression ratio and high computational efficiency (Rastegari et al. 2016; Jeon et al. 2020). Specifically, given that an importance of each parameter represented real number between 0 and 1, extract an optimal quantization solution modified from the previous binary-coding-bart
quantization methods that employ equal parameter importance. Similar to previous attempts, also find that calculating accurate importance of each parameter challenging. As a successful approximation of importance, we suggest that magnitude based importance estimation is especially effective for post-training non-uniform quantization.
2 POST-TRAINING PARAMETER QUANTIZATION FOR LANGUAGE MODELS
The number of parameters for language models dramatically increasing (e.g., GPT-3 (Brown) et al. 2020) requires 175 billion parameters). Correspondingly, model compression for language models is becoming a mandatory process to reduce response time and inference energy. We devise a compression method considering the followings:
Recent language models are usually memory-bound because of small batch size and lacking layers of high reuse (e.g.. conv layers). Thus, reducing memory footprint is critical.
Compression algorithms should be supported by dedicated kernels, designed specifically for language models if possible.
Compression training is challenging and expensive if hyper-paramenn parameters are added to already huge language models (hence, we choose post-training method.)
Fixed-point inference using uniform quantization is not desirable for language models because of noticeable accuracy degradation (Shen et al. 2019 Jeon et al. 2020) while the advantage of small computational units (e.g., INT8 MAC) is insignificant for memory-bound applications. Thus, we adopt floatbbsed parameter quantization (i.e., expected values of quantized parameters remains to be of full precision) that induce lot smaller number of quantization bits compared to fixed-poin quantization (uu et al. 2018 Stock et al. 2020).
Recently, kernel library, called B QGEMM (Jeon et al. 2020), was introduced support binary- coding-based quantization techniques to accelerate quantized neural networks. Using lookup tables, BiCGEMM enables byte-level memory accesses and achieves 8.3 run-time memory footprints and 3.5x speed up with a mobile CPU for Transformer (Chung et al. 2020). As result, binary-coding- based quantization has become a practical approach to quantizing language models. As such, we restrict our interests to binary-coding-basee quantization technique in this paper.
Quantization aware training an active research area to improve model accuracy (Courbariaux ct al,, 2015) Lee et al,, 2018). We note that in the case of language models, however, there are numerous occasions when retraining for quantization not available. For example, quantization- aware training requires in-depth knowledge on model compression while model designers may not have such expertise. On the other hand, the original training code or the entire training data may not be shared with model compression engineers. Also, modifying the original DNN models to be aware of quantization would increase model design efforts and training time significantly. Since language models already demand significant training time and cost, adding additional training complexity by quantization-awarn training would not be a practical option. As such, post- test-training quantization without retraining is gaining increasing attention (Zhao al. 2019 Nagel et al. 2019).
2

---

Under review as a conference paper at ICLR 2021
3 WEIGHTED QUANTIZATION BASED ON THE BINARY CODES
As discussed, we choose post-training dinary-coding-base quantization as our strategy to compress language DNN models efficiently. Following the Binary- Binary-Weight-Be Weight--etwort (Rastegari et al. 2016) introducing the binary codes as a quantization format, weight vector w is approximated to be ab by using a scaling factor a IR and binary vector b (E (=1,)++11)) where n is the vector size. A real number scaling factor is shared by multiple weights such that binary vector b occupies most of the weight storage requirements. Binary codes climinate the need for dequantization for inference, leading to reduced on-chip memory size for weights. In this section, we study general weighted quantization methodologies when quantization follows the format of the binary codes and and quantization error recognizes sensitivity information.
3.1 GREEDY METHOD AND ALTERNATING METHOD WITHOUT IMPORTANCE CONSIDERATIONS
In general. non-uniform weight quantization methods (in the form of the binary codes) strive to minimize || - ablla. In the case of I-bit quantization, we obtain the following analytical solution:
sign(w) (w), a"
On the other hand, in the case of bit quantization, there is no analytical solution (Rastegari et al. 2016 Xu al. 2018). result, various approximated methods exist for multi-bit quantization.
Greedy Method As computationally simple method, 1-bit quantization shown in Eq. ① can be extended to multi-bit (q-bit) quantization (Guo et al. 2017). Specifically, :hhbbit (i > 1) quantization performed by minimizing the residue of ( 1)th bit quantization as following:
min ||m:-1-;,bbb] 100-1-0,,b a,bill? where r w Σ ==1
Labbi 1 i<isa.
(2)
The optimal solution of Eq. then given as
b = sign(r:))) of = r rTib b
(3)
Alternating Method Greedy method described above is non-iterative. In order to reduce ||w Σ agb;ll' further than Greedy method, iterative methods would be necessary while increasing the number of iterations tends lower quantization error. Once initial a and b values are calculated by Greedy method, one can notice that {ar) can be refined (Guo et al. 2017) as
[a1...,,yy] = (BB B.) B3 wTw) )/w) when B, [b ..... b,l E (-1,
(4)
Then, B, can be refined as well by binary search given a new refined {a; [/=] As result, )/ and B, are refined alternatively. Alternating refinements of {a,) and B₃ are repeated until there is no noticeable improvement in quantization error. Such iterative quantization procedure is introduced as Alternating multi-bit method (Xu al., 2018).
3.2 IMPORTANCE-AWARE WEIGHTED QUANTIZATION
Let us assume that the importance of the i-th parameter is normalized and given as m. (0 S m, S 1). Then, we minimize the weighted quantization loss > (w; u; )) where w; is quantized to be Φ; = (aybb)) Before studying how to estimate importance values, we are interested in finding modified versions of Greedy method and Alternating method when importance values are given. For 1-bit quantization, weighted quantization also has the following analytical solution:
El= (m, (m,
ba = sign(w) w) a
(5)
E m
3

---

Under review as a conference paper at ICLR 2021
Note that if all importance values are equal (e.g.. m, 1 for all i), then Eq. becomes the same
as Eq. (T). Correspondingly. Eq. [ can be regarded as a special case of Eq. (5). Compared to the conventional Greedy method, proposed importance-awart Greedy method demands modifications to o; calculations as of = - (malri-al)/ Ei= m,.
For the importance-awart Alternating method, first conduct the importance-awarr Greedy method. Then, Eq. ( is to be transformed to employ importance. Let us define an n-by-n diagonal matrix M diag( 1 m,). where each diagonal element is importance value m,. By solving linear least squares, values are refined as
[a]. (a1.....a.) agl. 1a1=((B,)) (BB,,B.) MB,) B,,
when By = [b_..... byl byle +-1,+
(6)
while refining B, is still performed by binary search using refined scaling factors. Accordingly, Eq. (4) is a particular case of Eq. 66) when M is an identity matrix.
Overall, our proposed importance aware quantization scheme is comprehensive to include previous methods as a subset. In the rest of this paper, we investigate simple and efficient schemes to estimate importance metrics applicable post-training non-uniform binary-coding-bart based quantization.
IMPORTANCE ESTIMATION USING WEIGHT MAGNITUDE
Sensitivity or importance of each parameter can be estimated by evaluating the loss function change induced by parameter perturbation. Such estimation, however, is computationally demanding since each parameter's ' perturbation requires computations of entire feedforward paths while the number of parameters is ever increasing in recent DNN designs. Moreover, it difficult decide an appropriate amount of perturbation of parameters. In this section, we propose an efficient importance estimation method based on weight magnitude for fast and high-accel high-accurate post- post-training weight quantization.
4.1 WEIGIT MAGNITUDE AS IMPORTANCE
As an effort to study a factor affecting importance, we introduce Optimal Brain Damage (OBD) (LeCun et al. 1990). OBD approximates the loss function by the Taylor series and perturbation of the loss function oL. by weight perturbations presented as
aL
8L =
Sw;
+
hu ? +
hi + O(1ówl|s))
ow;
2
where v is the number of weights and hig is an element the Hessian matrix. Note that at local minimum, the first term is eliminated while all the h; ; non-negative. Using "diagonal" approximation and "quadratic" approximation (LeCun et al. 1990). Eq. (7) is simplified as
8L
(8)
2
As described in Eq. g diagonal elements of the Hessian matrix can be used as effective importance metries for loss-aware training algorithms for quantization (Hou et al. 2017 Hou & Kwok 2018).
Note that for post-traing test-training quantization, unfortunately, the first or the second partial derivatives (i.e., the gradient the Hessian) are not available. Thus, for post-training quantization, importance is required to be given as function of weight magnitude. In other words, we are interested whether Hessian-based importance can be replaced with magnitude-basee importance. The goal of this work is then to empirically show that magnitude-bant importance is indeed practical for post-training binary-coding based quantization.
To verify our basic assumption that large weights would present higher importance for binary-coding based quantization, we controlled scaling factors (i.e., a values of the minimum MSE are multiplied by 'Scaling Factor Multiplier') of a layer in LSTM model using PTB dataset (see details of models and dataset in Appendix) as shown in Figure I Indeed, when scaling factors become larger than the ones obtained by minimizing MSE present, test perplexity is enhanced for both layers.
4

---

Under review as a conference paper at ICLR 2021
150
Plrplosity
0.09
Pappesill
0.030
MSE
140
0.08
140
0.025
Conventional
(Muimizing MSE)
-30
Popples
0.07
Conventional (Minimixing MSE)
Paplexits
130
MSE
MSE
0.020
Optimal PPL
120
0.06
Optinal PPL
120
110
0.05
0.015
0.04
110
0.010
90
0.03
100
0.50
1.00
1.50
2,00
0.50
1.00
1.50
2.00
Scaling Factor Multiplier
Scaling Factor Multiplier
Figure 1: Quantization error (MSE) and test perplexity when one selected layer of an LSTM model using PTB dataset is quantized to be 1-bit. (Left): Embedding layer. (Right): LSTT layer.
4.2
FOR IMPORTANCE METRICS
The underlying principle of our importance estimation methodology that relatively smaller quanti- zation eπrоrs are supposed to be applied to weights of larger magnitude. To fine-tune our proposed 1) an exponent
importance estimation scheme, we propose the following three to control a correlation between importance and magnitude of a weight, 2) parameter to clip importance in order to handle outliers in magnitude distributions, and 3) a pruning parameter to exclude weights of low magnitude during quantization optimization.
E' (Exponent) A basic form to calculate normalized importance of each weight is presented as
E
W/
m
Wmam
where Wmaa is the maximum weight magnitude in a given layer (i.e., m. is computed in a layer-wise manner) and the weight of the largest magnitude considered to be the most important. A value between 0.0 and 1.0 for E is primarily adopted in our experiments to yield sub-linear importance increase as weight magnitude increases. E O results in the conventional quantization method with m: for all t..
C' (Clipping Importance) For given distribution of weight magnitude, a few large outliers may distort the entire distribution of m, obtained by Eq. (9) In other words, because of a few exceptionally large weights, most weights may exhibit small importance values. conventional quantization technique to prevent outliers in distribution is to clip weights/activationss and/or gradients (Choi et al. 2018 Zhao et al. 2019 Goodfellow et al. 2016). For Eq. (9) Wmaz is decided to be the weight magnitude at the (C 100)-th h percentile when 0 <<< C < 1. If m, exceeds 1.0, then m; is forced to be 1.0 (note that w; is not clipped).
P (Pruning for Quantization) Due regularization effects, lot of weights employ small magni- tudes and a weight distribution usually follows Gaussian distribution (Goodfellow et al. 2016). As result, a large number of small weights (with less importance as in Eq. (8) may take large portion of total quantization error unless my values of those weights are extremely small. Note that even though pruming prior quantization is an effective method improve quantization quality (Li & Liu 2016,22 Zhu et al. (017),, pruning would yield one additional bit per weight for masking information or sparse matrix formats with low parallelism for DNN inference. In our work, 1) we exclude weights of magnitude smaller than the (P 100)-th percentile from the quantization optimization, 2) find the scaling factors and the binary codes using weights larger than the (P x 100)-th percentile, and 3) all of the excluded small weights are assigned to binary code with the smallest magnitude available from combining scaling factors (while each sign information is maintained). Accordingly, while we adopt parameter pruning idea for quantization, additional pruning mask data is not necessary. In short, small weights are not considered while obtaining the binary codes, and then, replaced with the smallest weight in the binary codes.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Post-training (3 bits per weight) quantization comparison on MSE (quantization error),
average scaling factor values, training loss, and training model accuracy. For importance metrics, E 1.0 is used while P and C are not considered.
Quant. Err.
Average Scaling Factors
Training Loss
Training
Model
Method
(MSE)
Accuracy (%)
Greedy (No Importance)
1.13e-04
0.021
0.405
83.70
BERT (MRPC)
Greedy (E=1.0)
1.01c-04
0.028
0.370
84.71
Alter. (No Importance)
6.79c-05
0.029
0.693
80.81
Alter. (E=1.0)
1.05e-04
0.035
0.212
93.13
Greedy (No Importance)
1.14E-04
0.021
0.726
76.32
BERT (MNLI)
Greedy (E=1.0)
1.02E-04
0.029
0.418
86.93
Alter. (No Importance)
6,84E-05
0.028
0.349
89.75
Alter. (E=1.0)
1.07E-04
0.035
0.282
91.85
3o-04
Mrk
0.58
1.00-044
300
0..00
Im-0
3c-04
1.06-11
M30
0.52
MBR
3OV
MAB
2c-04
0..90
0.81
8c-06
2n-04
00.76
0..76
1.01-444
0.63
0.74
0.74
0.0081
0,3000
1.0000
0.0)
Figure 2: Test accuracy and quantization error (MSE) of fine-tuned BERT-base model MRPC when weights are quantized into bits by our proposed method while one of E, C, or P varies.
4.3 EMPIRICAL OBSERVATIONS
To verify basic operations of our proposed method, perform post-training weighted quantization using fine-tuned models of BERT-base (Devlin et al., 2018) on MNLI and MRPC dataset within GLUE benchmark (Wang al.) 2018). In the case of fine-tuned BERT models, quantize all weights except those of segment embedding layer and a classification layer which show tiny storage footprint. For conventional iterative refinements of a and B values 20 times over which no further noticeable quantization error improvement is recognized. Given weight matrix or tensor, a and B are computed for each row, independently (hence, we study row-wise quantization in this work). Due to the space limit, see Appendix for additional experimental results with various models not included in this section.
weighted Alternating quantization methods, conduct
We first analyze how simple weighted quantization scheme (with E=1.0, C=1.0, and P=0.0) adds distinguished features to the conventional quantization methods. Table presents comparisons on quantization error (MSE), average scaling factor values, training loss, and training accuracy. Note that for Alternating weighted quantization, despite larger quantization MSE (i.e., >i (w; (w;-);;33) training loss and training model accuracy are improved such that it is confirmed that minimizing - xx is preferred to minimizing MSE for post-training quantization. is interesting to see that for the Greedy method, quantization MSE is reduced by weighted quantization. We conjecture that for weight distributions in DNNs, Eq. 5 is probably a better approximation compared to Eq. ( even to minimize quantization MSE. For both Greedy and Alternating methods, scaling factors increase by weighted quantization due to the context of magnitude-
(
importance design.
Let study the impact of our proposed weighted quantization on model accuracy when E, C, and P can vary for fine-tuning. Figure 2 tuned BERT-base model on MRPC when we sweep only one of E, C. or across all layers. It is clear that all
describes test accuracy and quantization error of a fine-
for importance metrics enable new search space for model accuracy that is somewhat uncorrelated to quantization error. Using various combinations of F, C, and P. test model accuracy of fine-tuned BERT and DistilBERT models using Greedy and
See Appendix for detailed descriptions of models and dataset selected for experiments.

---

Under review as a conference paper at ICLR 2021
Table 2: Test score after post-training quantization with various E, C and choices when the quantization bit is 3.
BERT-base
DistilBERT-bar base
MRPC
MNLI
SQUAD
MRPC
MNLI
SQUAD
Method
E
C
P
(Acc)
(Acc)
(fl)
(Acc)
(Acc)
(fl)
No Importance
78.2
73.4
56.3
70.0
58.9
50.4
Greedy
1
1.0
0.0
77.0
79.4
75.2
76.7
59.5
62.0
1
0.99
0.0
82.1
79.3
78.9
78.7
58.5
64.1
No Importance
76.0
81.2
81.9
75.5
68.2
77.1
1.0
1.0
0.0
85.0
81.6
83.7
76.5
75.1
77.8
1.0
0.99
0.0
83.1
81.3
83.3
73.0
71.7
77.1
1.0
0.9999
0.0
86.0
81.6
84.3
74.5
74.3
76.8
0.5
1.0
0.0
81.1
82.7
84.7
79.4
75.1
79.6
0.2
0.0
76.7
82.8
83.5
76.2
71.6
78.1
Alter.
0,5
0.9999
0.0
79.9
82.4
84.1
78.9
74.4
79.5
0.0
1.0
0.02
75.0
81.4
82.8
76.0
66.9
77.0
0.0
1.0
0.05
76.0
81.6
82.7
75.5
66.3
77.3
1.0
1.0
0.02
85.0
81.5
83.5
77.0
75.3
77.7
1.0
1.0
0.05
85.2
81.5
83.7
77.7
75.6
76.7
1.0
1.0
0.10
84.1
81.0
83.5
78.7
74.7
74.8
Full-Precision (ours)
87.7
84.5
88.6
85.3
82.1
86.9
Alternating quantization methods. Even though numerous hyper-parameter combinations outperform Greedy and Alternating methods without importance, the best of each model, and hence, an automatedd apeer-paramtte search process is desirable. Note that in order to enable such an automated process, we need investigate whether set of siper-parameters for importance searched by using training dataset is also valid for test dataset. We extensively explored hyper-parameter combinations (using E, C. P) with different number of refinement iterations using BERT on MRPC and MNLI and confirmed that training model accuracy achieved by weighted quantization is highly correlated with test model accuracy such that our proposed maintain generalization capability (see Fig. in Appendix).
varies for
Increasing the number scaling factors (by decreasing the number parameters sharing a scaling factor) enhances model accuracy despite increased memory footprint and computation overhead during inference. Our proposed weighted quantization obtains larger accuracy improvements when less number of scaling factors is utilized (see Table Table7in in Appendix that compares model accuracy with different number of scaling factors when Alternating method is applied to BERT on MRPC.)
5 EXPERIMENTAL RESULTS
We observed that optimal set of hyper-parameters needs to be achieved empirically. Unfortunately. optimizing F., C, and for post-training quantization to obtain the best model accuracy is chal- lenging because 1) trained models present a variety of weight distributions and 2) hyper-parameters are correlated. As an effort to automate the hyper-parameter search process, we adopt Bayesian optimization (BO) that implemented by publicly available code introduced in (Nogueira 2014-)
Once we perform rough and fast grid search for paraam-arseeters (as shown in 2). then BO conducts fine-tuning of hyper- parametrrss...s. As result, achieve quick post-training quantization even when optimal srpee--praaeters vary for each layer. For BO experiments, training dataset D; is used during hyper-parameter search, and then test dataset D, validates the optimization procedure (refer Table 4h. In other words, given a syper-paramettr vector denoted by {E, C, PJ e g² BO tries to find the optimal x" to be arg maxx fx; Dr), where ff measures accuracy of the model. Then, test model accuracy is measured f(x" D.).
To the best our knowledge, our work is the first post-training binary-coding- baadd quantization considering weight importance. As result, compare results with the conventional Greedy algorithm and Alternating algorithm. We consider three different search methods for our proposed scheme: 1) (manual search) investigate prearranged 16 sets of parammterssssrss for importance
7

---

Under review as a conference paper at ICLR 2021
E
C
P
1.2
1.2
1.0
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.0
0.0
0.0
20
60
20
40
MRPC
MNLI
SQUAD
Figure 3: E, C, and P values searched by layer-wise BO for BERT-base on MRPC, MNLI, and SQUAD, X-axis shows layer index and y-axis shows hyper-parameters optimized differently for each layer. BO is necessary to efficiently and quickly find such diversified E, C, and P values.
Table 3:
Quantization results on various language models (see Appendix for details on model descriptions). Alternating quantization scheme significantly improves test scores when combined with our proposed importance metrics (described 'Ours') that are searched by layer-wise BO.
Quant. Bits
Test Score
Model
Dataset
Metric
FP
Alter.
Ours
AWD-LSTM
2
PPL.
56.4
79.3
62.6
(-16.7)
PTB
(fine-tuned)
3
PPL
56.4
61.2
57.1
(-4.1)
MRPC
3
ACC
87.7
76.0
84.3
(+8.3)
BERT
MNLI
3
ACC
84.5
81.2
82.9
(+1.7)
base
SQUAD 1.1
3
F1
88.6
81.9
85.2
(+3.3)
MRPC
3
ACC
85.3
75.5
83.3
(+7.8)
DistiBBERT base
MNLI
3
ACC
82.1
68.2
79.9
(+11.7)
SQUAD I.I
3
FI
86.9
77.1
81.2
(+4.1)
Longformer
SQUAD 1.1
89.2
85.8
86.6
(+0.8)
Transformer
newstest2017(en2de)
3
BLEU
26.95
25.09
25.23
(+0.14)
metric (described in Table 5). 2) (model-wise BO) for all layers, the same values of E, C, and parameters are locally searched for each layer, and the fixed before proceeding to the next layer (hence, BO for quantization is performed in layer-by-layer manner). For all model-wise or layer-wise BO for various models, same 16 sets hyper-paramest (given in Table 5) are explored first as initial samples. BO outperforms manual search while Mnyer-wiss, BO improves test score further presented in Table (BERT-base), (DistlBERT--buss
P are explored and applied for quantization, and 3) (layer-wise BO) hyper-parameters
(Longformer), [3 (AWD-LSTM), and [4 Trrnsformmr NMT). Among three methods considered, layer-wise BO the best because the optimal set of hyper-parameters turns out to be vastly different for each layer as shown Figure 3
and
Table presents the overall comparison test scores of various language models that quantized by conventional Alternating quantization and our proposed weighted quantization. Compared to conventional Alternating quantization (that our baseline for post-training binary-coding-baat baaedd quantization) of equal importance for each parameter, ours improves test scores for all language models Table We note that weighted quantization yields relatively different amounts of improvements on test scores depending on given model. Even though thorough analysis of such different accuracy enhancement would entail in-depth sensitivity analysis of parameters toward test scores, pruning weights in the context of magnitude (without retraining) provides approximated correlation between importance and magnitude for given weight distribution (see Figure 7 in Appendix). Indeed, the pruning rate of the Longformer is lot lower compared to the other models (as shown in Figure 7 that can partly explain challenges to enhance Longformer quantization. We also note that our proposed method highly depends on the target objective function. As such, Transforners show different guantization results depending on the target selection perplexity (PPL) or BLEU score due to somewhat low correlation between PPL and BLEU score (Appendix C.3).
8

---

Under review as a conference paper at ICLR 2021
We applied our weighted quantization scheme to ResNet models CIFAR-10 and ImageNet (refer to Appendix D) for which model accuracy is also significantly enhanced similar to language models.
6 CONCLUSIONS AND FUTURE WORK
In this paper, we propose weighted quantization framework employing importance metrics that are useful when each parameter shows different sensitivity toward loss function change. For the binary-coding-basee quantization that is our choice for language models because of existing efficient kernel designs (e.g.. BiQGEMM) and high compression ratio, we extract modified Greedy method and Alternating method assuming that each importance value is represented as real number between O and 1. Using various DNN models, we demonstrate that a magnitude-based importance metric effective for post-training quantization in the form of the binary codes. To fine-tune model accuracy, we also propose three set of sypee-parameter:. varies depending each model design. As such, suggest Bayesian optimization as an effective technique to automate hyper-parameter search process. Our proposed be independently optimized to each layer further improve compression ratio and/or model accuracy. would be interesting to study additional hyper-parameters effective for post-training quantization. Since our weighted quantization framework general (rather than depending on a particular approximation such as the Hessian), if proper importance metrics are found, our proposed quantization techniques can be extended a quantization aware training method.
that need to be empirically investigated since an optimal
prppe-pparmmteri
REFERENCES
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.051500 2020.
Eric Brochu, Vlad M Cora, and Nando De Freitas. tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv: /012.2599, 2010.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. MXXi::0005.14663 2020.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. /rXiv:/805.0600 2018.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. In International Conference on Learning Representations (ICLR), 2017.
Insoo Chung, Bycongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baescong Park, Sangha Kim, and Dongsoo Lee. Extremely low bit Transformer quantization for on-device neural machine translation. arXiv preprint arXiv:2009.07453 2020.
Matthicu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, pp. 3123-3131, 2015.
Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas, et al. Predicting parameters in deep learning. Advances in neural information processing systems, pp. 2148-24 2256, 2013.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1870.04805. 2018.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ: Hessian aware quantization of neural networks with mixed-precision.
2019.

---

Under review as a conference paper at ICLR 2021
Mark Ebden. Gaussian processes: A quick introduction, 2015.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//ww.deepleammmgbooooook
Yiwen Guo, Anbang Yao, Hao Zhao, and Yurong Chen. Network sketching: exploiting binary structure in deep CNNs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1440404440000 2017.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. Advances in Neural Information Processing Systems, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations (ICLR), 2016.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015.
Lu Hou and James T. Kwok. Loss- weight quantization of deep networks. In International Conference on Learning Representations (ICLR), 2018.
Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization of deep networks. In International Conference on Learning Representations (ICLR), 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient baleger-aribhmetic-onyy inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2713, 2018.
Yongkweon Jeon, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Jeongin Yun, and Dongsoo Lee. Biggemm: Matrix multiplication with lookup table for binary-coding- bbsedd quantized dnns. arXiv preprint arXiv:2005.000 09904, 2020.
Michael Carbin Jonathan Frankle. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
Soroosh Khoram and Jing Li. Adaptive quantization of neural networks. In International Conference on Learning Representations (ICLR), 2018
Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander Rush. OpenNMT: Neural machine translation toolkit. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pp. 177 184, Boston, MA, March 2018. Association for Machine Translation in the Americas. URL
https: www ac web. .rrb/org//oo antho ogy/ W/ / 817
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint 1808. 2018.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598-605, 1990.
Dongsoo Lee, Parichay Kapoor, and Byeongwook Kim. Deeptwist: Learning model compression via occasional weight distortion. arXiv: 1810. 1810.1223.. 2018.
10

---

Under review as a conference paper at ICLR 2021
Fengfu Li and Bin Liu. Temary weight networks. arXtv:1603.04711, 2016.
Daniel James Lizotte. Practical bayesian optimization. University of Alberta, 2008.
Mitchell Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of English: The Penn Treebank. Comput. Linguist., 19(2):11-330, 1993.
Mark D. McDonnell. Training wide residual networks for deployment using a single bit for each weight. International Conference on Learning Representations (ICLR), 2018.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. arXiv preprint
2017.
Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, pp. 6655 6659, 2013.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. frXiv:9906.04721, 2019.
Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014.
com/ fmfh/Bayesiinnnpf http:://t.ttttoooooom/////a//aap imi zat zationi
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. In International Conference on Learning Representations (ICLR), 2018.
Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv: (eXiv:/804.08711, 2018.
Rohit Prabhavalkar, Ouais Alsharif, Antoine Bruguier, and Ian McGraw. On the compression of recurrent neural networks with application to LVCSR acoustic modeling for embedded speech recognition. In ICASSP, pp. 5970-5974, 2016.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252 2015.
doi: 100/0///s 263-011-00816--
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter. arXiv: arXiv:1910.01 1910.01 108, 2019.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Ma- honey, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of BERT. arXiv:1909.05840 2019.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. In International Conference on Learning Representations, 2020. URL
net. http://pwerreeeee..eeeeeo..ooaaa wttpee/oooom.ddr....eeeee.e.eeee
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. November 2018.
11

---

Under review as a conference paper at ICLR 2021
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. In International Conference on Learning Representations (ICLR), 2018.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv:1409.2329, 2014.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. arXiv:1901. 00004, 2019.
Shuchang Zhou, Yuzhi Wang, He Wen, Qinyao He, and Yuheng Zou. Balanced quantization: An effective and efficient approach quantized neural networks. Journal of Computer Science and Technology, 12:667-682, 2017.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. In International Conference on Learning Representations (ICLR), 2017. 2017.
12

---

Under review as a conference paper at ICLR 2021
A MODELS AND DATASETS
A.1 LSTM MODELS
1-Layer LSTM model (in Fig. (Zaremba et al. 2014): One layer LSTM model with 300 hidden states ji used to predict PTB dataset. We compress the a LSTM layer and embedding layer of the pre-trained language model to draw the Figure [ A scaling factor extracted for each raw, e.g.. for the (10000, 300) embedding layer, there exist 10,000 scaling factors result of quantization.
AWD-LSTM (Merity et al., 2017): We use a 3-layer AWD-LSTM model. The embedding size is 400 and the hidden vector size within LSTM layers is 1550. We train the original model during
500 epochs and then fine-tune during additional 300 epochs F) We compress both models including embedding and softmax layers by our post-training quantization method.
Dataset:
To evaluate our quantization method, we use the test dataset of Penn Treebank (PTB)
dataset (Marcus al., 1993) for the LSTM language models. To find the quantization parameters with BO, we use val lilidd dataset.
A.2 HUGGINGFACE LANGUAGE MODELS
To evaluate recently developed language models, utilize the t rans Transforre formers library (PyTorch version) developed by buggingface (Wolf et al., 2019). For all bert-based models, the last classification layer and the sequence embedding layer are quantized because the sizes of weights are relatively smaller than other weights.
BERT (Devlin et al., 2018) DistilBERT (Sanh et al. 2019): We fine-tune the pre-trained BERT- base and DistilBERT- base models to evaluate our method. BERT-base model consists of 12 encoder blocks with 768 hidden size and DistiBBERT-bar model consists of 6 encoder blocks with 768 hidden size. We follow tne-tt recipes found in the rans formers repository [ For the MRPC and MNLI tasks, the initial learning rate is 2e-5 and the training epoch 3. For SQUAD task, the initial learning rate is 3e-5 and the training epoch is 2.
Longformer (Beltagy al., 2020): To evaluate the Longformer model, we choose the pre-trained longformer-base model that has 12 blocks of Longformer encoder with 768 hidden size. We fine-tune the pre-trained mode for SQUAD v1.1 dataset with the same recipe.
Dataset: We use three language tasks: MRPC, MNLI and SQUAD(vI.I). To search quantization parameters (E, C' and P), we use a randomly sampled fraction of train dataset when the evaluation time is too time-consuming. For the MRPC task, train dataset is small enough to use. For the MNLI task, we use only 10% of train dataset. For the SQUAD task, we only 6.7% of train dataset for bayesian optimization while dev dataset is used for testing because there is no published test dataset.
the whole train dataset because the
A.3 OPENNMT TRANSFORMER
We the pre-trained transformer model for Neural Machine Translation task (Klein et al. 2018 The transformer model consists of encoder blocks, 6 decoder blocks and embedding layers. Note that the embedding layers are shared, e.g. embedding weights are not tied. The vocabulary size is 50k and the size of the hidden vector is 512.
We evaluate the pre-trained model in translation direction: English German (en2de). We use vali id dataset in newstest2017 for BO processes and test dataset for the test evaluation. All datasets are pre-processed by SentencePiece (Kudo & Richardson, 2018). All the translation scores are BLEU scores by sacrebleu script (Post 2018) as the beam size is 1.
Dataset:
Available attps///hm..cooyyyyc////////e/ee
'The detailed parameters are described in http://iithuh.ccom//stssseeeeeee
Avaliable at http://nnnnncccoooooooooooooonne
Available https//gpemmm..nn//ddeespyy
13

---

Under review as a conference paper at ICLR 2021
A.4 RESNET FOR IMAGE CLASSIFICATION
We conduct experiments using ResNet32 on CIFARIO (Krizhessky 2009) and ResNet18 (He et al. 2016) ImageNet (Russakovsky al. 2015). For convolution tensors, a and B are computed for each channel. We maintain full-precision on the first and last layers of ResNet models because those layers are very small while lot of quantization bits are required (McDonnell 2018
For 18, we use ImageNetIK3 training dataset, which is small subset of ImageNet dataset, for fast haper-parameter search, while test accuracy is still measured by the entire ImageNet test dataset. To obtain the same accuracy for the same set of hyper- parameters, the training dataset is not randomly manipulated (e.g., by cropping and flipping).
B BAYESIAN OPTIMIZATION FOR WEIGHTED QUANTIZATION
BO is one of automated machine learning (AutoMLL techniques search optimal for target networks. Given black box function S. BO aims to find the optimal x* to be arg maxx (x).. Suppose observations are described as y=[f(x]), (x))....., fxx)) and y. an output of any unobserved Xx, then under the assumption that fxx) is drawn from Gaussian Process, the distribution of y- y follows (oowsN((Ks NKKK "r K K,K KK"). where
x(x1.x1)
1(x1,Xn)) xn)
K=
K. K.=Ik(x,,x))) ((x,,xx)) and K..== xxx,,,,)....) (10)
x(xI,xi)
x(x1,xa)
The kernel function k(x, x') i) is of the hyper-paramett for BO and measures similarity between x and x' (i.e., the output is high when they are close). There are various kernel functions (Rasmussen & Williams 2006), and
squared-exponentiall kernel that is one of the popular choices for regression (Ebden 2013). To identify which of the unobserved data to be taken as X,+1- the acquisition function needs to be specified. In general, the expected improvement acquisition function aFr (Lizotte 2008 (see Eq. (11)) is most commonly used and is selected for our experiments.
Ex.(x.ly) ly) = (2 (()))((x.),
where
(ix.)-f(xt)- (x_)
if (x..) > and f(x*) = max f(x). if ox.)
Z
(12)
The parameter Ç is the trade-off factor between exploitation and exploration. In our experiments, we set & to 0.2, which implies that exploitation has more influence on determining X After computing agr for unobserved random sampled data x.S, X,+1 is computed as arg maxy. agr dxr(x.ly Further details of BO can be found in (Brochu al. 2010 Nogueira 2014 Snock et al. 2012).
Avaailable http:///wwmmoooooooooooooooooona 1k.zip
14

---

Under review as a conference paper at ICLR 2021
0.83
0.85
0.82
Test ACC.
Test ACC:
0.81
0.50
0.75
0.79
0.75
0.80
0.85
0.90
0.95
0.88
0.90
0.92
0.94
Training ACC.
Training ACC.
Figure 4: Relationship of training accuracy achieved by weighted quantization and test accuracy using BERT MRPC (LEFT) and BERT on MNLI (RIGHT).
Table 4:
Dataset usages and maximum iterations for Bayesian optimization. In the case large
training dataset (such as MNLI and SQUAD vI.1), samples.
Dataset
Maximum iterations for BO
BO Process
Test Evaluation
Model-wise
Layer-wise
Penn Treebank
valid
test
1000
250
MRPC
train 10% of train
test
1000
50
MNLI
test
1000
50
SQUAD vI.1
6.7% train valid (detokenized)
dev
1000
50
test
newstest2017
1000
30
(detokenized)
CIFARIO
train train of ImageNetIK
test
2000
100
ImageNet
test
2000
200
C
RESULTS ON LANGUAGE MODELS
Table 5: 16 sets of hyper-paramente selected for our manual search of importance metric.
Set 1
Set
Set 3
Set 4
Set
Set 6
Set 7
Set
P=0.0 (default)
E
1.0
1.0
1.0
1.0
0.5
0.2
0.5
0.5
C'
0.99
0.999
0.9999
1.0
1.0
0.999
0,9999
Set 9
Set
Set
Set 12
Set
Set 14
Set 15
Set
C=1.0 (default)
E
0.0
0.0
0.0
0.0
1.0
1.0
1.0
1.0
P
0.02
0.05
0.1
0.2
0.02
0.05
0.1
0.2
C.I FINE-TUNED BERT, DISTILBERT AND LONGFORMER
15

---

Under review as a conference paper at ICLR 2021
Table 6: Test score after post-training quantization with various E, and P choices when the quantization bit is 4.
BERT-base
DistilBERT-brie
MRPC
MNLI
SQUAD
MRPC
MNLI
SQUAD
Method
E
P
(Acc)
(Acc)
(f1)
(Ace)
(Acc)
(fl)
No Importance
76.2
78.1
70.7
77.7
64.7
66.6
Greedy
1
1.0
0.0
72.3
80.1
79.0
73.0
73.4
80.7
1
0.99
0.0
80.1
81.0
81.8
76.0
75.2
81.2
No Importance
78,4
82.8
86.2
77.0
77.0
83.4
1.0
86.0
83.8
88.0
84.1
81.4
85.9
1.0
0.99
0.0
84.3
83.5
87.9
82.3
80.9
85.2
1.0
0.999
0.0
86.3
83.6
87.8
83.3
81.4
85.7
L,0
0.9999
0.0
86.5
83.7
88.0
84.1
81.6
85.7
0.5
1.0
0.0
85.0
83.9
88.1
83.3
80.9
85.2
0.2
1.0
0.0
82.6
83.3
87.0
81.6
78.9
84.4
0.5
0.999
0.0
85.0
83.6
87.8
82.1
79.7
84.9
Alter.
0.5
0.9999
0.0
85.5
83.5
87.9
82.4
80.2
85.1
0.0
1.0
0.02
78.9
82.8
86.4
78.4
77.1
83.8
0.0
1.0
0.05
79.9
82.9
86.3
79.7
77.4
83.8
0.0
1.0
0.1
80.4
83.0
86.4
79.4
77.2
83.6
0.0
81.6
82.8
86.2
79.9
77.4
83.5
1.0
1.0
0.02
86.5
83.9
88.0
84.6
81.6
85.8
1.0
0.05
86.3
83.7
88.0
84.1
81.7
85.6
1.0
1.0
0.1
85.8
83.7
87.9
84.1
81.4
85.4
1.0
1.0
0.2
84.3
83.7
87.5
83.8
81.3
84.9
Full- Preci-Prei (ours)
87.7
84.5
88.6
85.3
82.1
86.9
Table 7: Model accuracy of BERT MRPC quantized into 3 bits per weight by Alternating method with different number of parameters sharing a scaling factor.
of parameters sharing one a
E
C
P
48
96
192
384
768
No Importance
85.29
85.05
79.41
76.96
75.74
85.05
85.29
85.29
84.31
84.80
1 0.95
85.05
84.31
85.54
83.09
79.90
I 0.99
85.05
84.31
85.05
82.84
81.62
0.999
84.80
84.56
85.78
84.07
83.82
1 0.9999
84.56
84.07
85.05
84.56
82.35
0.5
85.78
86.03
85.29
83.09
79.90
0.2
86.03
85.05
82.35
80.64
75.25
0
0.05
84.31
84.80
75.25
75.98
76.00
1
0.05
84.07
85.05
84.56
83.84
84.12
Table 8: Hyper-paramene parameter search results BERT-base (quantized into bits per weight) using manual search, model-wise BO, and layer-wise BO.
Hyper-parameter search method train ACC.
test ACC.
Cr,,meterssF.C....
Manual (by Table 5)
86.0
(1.0, 0.9999, 0.0)
MRPC
Model-wise BO
93.8
86.5
(0.693, 0.999, 0.045)
Layer-wise BO
94.5
84.3
Fig. (0.2, 1.0. 0.0) (0.534, 0.9999, 0.002)
Manual (by Table
82.8
MNLI
Model-wise BO
93.7
82.4
Layer-wise BO
94.5
82.9
Fig. 3
Manual (by Table
84.7
(0.5, 1.0, 0.0)
SQUAD
Model-wise BO
90.0
84.8
(0.560, 1.0. 0.0)
Layer-wise BO
90.7
85.2
Fig.
16

---

Under review as a conference paper at ICLR 2021
Table 9: Hyper-parameter search results of DistilBERT-base (quantized into 3 bits per weight) using manual search, model-wise BO, and layer-wise BO.
Plyper-parameter search method
train ACC.
test ACC.
Gtrameters(E,C.2
Manual (by Table
78.9
(0.5, 0.9999, 1.0)
MRPC
Model-wise BO
85.2
79.7
(0.679, 1.0. 0.0)
Layer-wise BO
90.3
83.3
Fig. 5
Manual (by Table
75.6
(L.0, 1.0, 0.05)
MNLI
Model-wise BO
80.7
75.4
(0.687, 1.0. 0.048)
Layer-wise BO
86.9
79.9
Fig. (0.5, 1.0, 0.0)
Manual (by Table
79.6
SQUAD
Model-wise BO
86.6
79.6
(0.430, 1.0. 0.0)
Layer-wise BO
88.4
81.2
Fig.
E
1.2
1.2
1.0
1.0
0.8
0.8
0.5
0.5
0.5
0.2
0.0
0.0
0.0
10
20
30
10
20
30
10
20
30
MRPC
SQUAD
Figure 5: E, C, and P values searched by layer-wise BO for DistilBERT-base on MRPC, MNLI, and SQUAD. X-axis shows layer index and y-axis shows hyper-parameters optimized differently for each layer.
Table 10: Fl scores of Longformer on SQUAD v1.1 after post-training quantization (4 bits per weight) with various C and P choices.
Method
E
C
P
4bit
5bit
No Importance
85.8
88.0
1.0
1.0
0.0
83.5
88.2
1.0
0.99
0.0
83.8
88.3
0.5
1.0
0.0
85.2
88.3
0.2
1.0
0.0
85.9
87.8
0.5
0.999
0.0
85.4
88.3
Alter.
0.5
0.9999
0.0
85.4
88.4
0.0
1.0
0.02
86.0
88.1
0.0
1.0
0.05
86.0
87.7
0.0
1.0
0.1
85.4
87.7
1.0
1.0
0.02
83.6
88.0
1.0
1.0
0.05
83.0
88.1
1.0
1.0
0.1
81.4
88.1
Soll--Precisoo
89.2
Table 11: Hyper-paramette search results of Longformer using manual search, model-wise BO, and layer-wise BO.
Hyper-parameter search method
val ACC.
test ACC.
Parameters( E,C,P)
Manual (by Table
86.0
(0.0, 1.0. 0.05)
4-bit
Model-wise BO
88.9
86.5
(0.11. 1.0, 0.0)
Layer-wit wise BO
89.6
86.6
Fig.
17

---

Under review as a conference paper at ICLR 2021
E
P
1.0
20000
0.2
0.0
Longtormer SQUAD
Figure 6: E. C. and values searched by layer-wise BO for Longformer on SQUAD v1.1. X-axis shows layer index and y-axis shows hyper-parameters optimized differently for each layer.
1.00
0.90
Relative Degradation
0.80
distiBBERT-MPPC
distiBERTTMNNI
0.70
distiBBERT-QUUAD
BERT-MRPC
0.60
BERT-MNLI
BERT-SQUAD
Longformer-SQUAA -QQADD
0.50
Transformer
Binetued-AADDLTTEE
0.40
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Pruning Rate
Figure 7: Test score degradation by (post-training) pruning weights (based on the magnitude) using various pre-trained language models. Weights of layer are pruned by the same target pruning rate. For the same pruning rate, the Longformer presents sharper score degradation that partly explains the difficulty improving test scores by our proposed weighted quantization method compared to the conventional Alternating quantization.
18

---

Under review as a conference paper at ICLR 2021
C.2 AWD-LSTM ON PTB DATASET
Table 12: Perplexity AWD-LSTM model on PTB test dataset after post-training quantization with
various E, C' and P choices.
Trained
Fine-tuned
Method
E
P
2bit
3bit
4bit
2bit
3bit
4bit
No Importance
93.25
66.57
61.84
79.32
61.21
58.11
1.0
1.0
0.0
71.01
62.16
59.97
68.38
59.19
57.14
1.0
0.99
0.0
68.47
62.29
60.46
65.66
59.14
57.23
1.0
0.999
0.0
69.13
62.01
60.04
66.91
59.08
57.19
1.0
0.9999
0.0
69.95
62.10
59.99
67.73
59.16
57.14
0.5
1.0
0.0
72.38
63.04
60.50
65.04
59.05
57.65
0.2
1.0
0.0
81.96
64.76
61.14
71.28
60.07
57.82
0.5
0.999
0.0
72.52
63.07
60.58
65.05
59.05
57.68
Alter.
0.5
0.9999
0.0
72.32
63.05
60.5
64.99
59.03
57.59
0.0
1.0
0.02
91.84
66.42
61.86
78.32
61.13
58.14
0.0
1.0
0.05
89.70
66.22
61.85
76.84
60.95
58.16
0.0
86.41
65.78
61.76
74.70
60.66
58.20
0,0
1.0
0.2
81.01
64.67
61.45
71.80
59.91
57.96
1.0
1.0
0.02
114.22
62.21
59.96
68.75
59.27
57.14
1.0
1.0
0.05
114.41
62.24
59.96
69.57
59.31
57.14
1.0
1.0
0.1
114.91
62.32
59.98
111.77
59.46
57.14
1.0
1.0
0.2
116.81
62.46
60.1
114.6
60.02
57.30
Full-Precision
59.13
56.43
Table Hyper-parament search results fine-tuned AWD-LSTM (quantized into 2/3/4 bits per weight) using manual search, model-wise BO, and layer-wise BO.
Elyper-paramen search method
val ACC.
test ACC.
Parameters (EC,P)
Manual (by Table
64.99
(0.5, 0.9999, 0.0)
2-bit
Model-wise BO
67.76
65.2
(0.9934, 0.8454, 0.0)
Layer-wise BO
64.80
62.63
Fig. 8 (0.5, 0.9999, 0.0)
Manual (by Table
59.03
3-bit
Model-wise BO
61.34
58.97
(0.9643, 0.9989, 0.0)
Layer-wise BO
60.52
58.21
Fig. 8
Manual (by Table
57.14
(1.0, 1.0, 0.02)
4-bit
Model-wise BO
59.55
57.19
(1.0148, 0.9810, 0.029)
Layer-wise BO
59.40
57.11
Fig.
C.3 TRANSFORMER
Table 14: Quantizing Transformer by using bits per weight with different quantization schemes and the metric to be optimized by BO.
Method
Target
valid PPI.
valid BLUE
test PPL.
test BLEU
Pre-trained
1.00373
25.85
1.00398
26.95
Alternating (No Importance)
MSE
1.00646
24.17
1.00678
25.09
Model-wise BO
PPL
1.00339
19.20
1.00368
20.38
Layer-wise BO
PPL
1.00283
23.40
1.00291
24.67
Model-wise BO
BLEU 1.00651
24.24
1.00676
25.05
Laser-wise BO
BLEU
1.00494
24.30
1.00498
25.23
19

---

Under review as a conference paper at ICLR 2021
E
C
1.2
1.0
1,0
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
6
2bit
3bit
1bit
Figure 8: E, C, and P values searched by layer-wise BO for fine-tuned AWD-LSTM model. X-axis
shows layer index and y-axis shows hyper-parameters optimized differently for each layer.
P
1.2
1.2
0.8
0.8
0.5
0.2
0.2
0.0
0.0
100
ppl optimized
bleu optimized
Figure 9: E, C, and P values searched by layer-wise BO for Transformer. BO is performed to optimize PPL. (Left) or BLEU (Right). X-axis shows layer index and y-axis shows hyper-parameter optimized differently for each layer.
20

---

Under review as a conference paper at ICLR 2021
D RESULTS ON RESNET MODELS
Table 15: Post-training quantization comparison on quantization MSE, average sealing factor values, training loss, and training model accuracy. For importance metrics, 1=1.0 is used while P and C are not considered.
Quant, Err.
Average Scaling Factors
Training Loss
Training
Model
Method
(MSE)
Accuracy (%)
Greedy (No Importance)
4.35E-04
0.035
0.783
12.54
ResNet-32
Greedy (E=1.0)
3.02e-04
0.051
0.693
84.12
(CIFAR-10)
Alter. (No Importance)
2.27e-04
0.051
0.324
90.72
Alter. (E=1.0)
3.79c-04
0.062
0.309
91.68
Creedy (REF)
4.98E-05
0.010
7.075
0.20
Greedy (E=1.0)
3.47e-05
0.015
4.853
12.87
Aller. (REF)
2.39e-05
0.016
2.419
47.62
ResNet-18
Alter. (E=1.0)
4.43e-05
0.023
3.033
37.01
(ImageNet)
Alter. (E=2.0)
2.89c-05
0.012
1.674
59.94
Alter. (F=3.0)
2.62e-05
0.019
1.992
54.19
Alter. (F=5.0)
2.48e-05
0.018
2.024
54.16
Alter. (E=7.0)
2.42c-05
0.017
2.282
49.53
Table 16: Model accuracy(%) on test dataset after post-training quantization with various E and C choices. q the number of guantization bits.
ResNet-32 (CIFAR-10))
ResNet- (ImageNet)
Method
E
P
q=3
Q=4
q=3
q=4
No Importance
12.7
24.4
0.2
0.4
Greedy
1.0
0.0
79.9
84.2
12.0
44.2
0.99
0.0
71.5
81.8
14.0
42.6
No Importance
84.9
91.2
43.3
60.1
1.0
0.0
85.5
91.7
32.8
61.1
0.95
0.0
85.9
91.7
44.5
61.5
0.99
0.0
87.2
91.5
35,8
60.9
0.999
0.0
85.5
91.6
36.2
64.5
0,9999
0.0
85.5
91.7
32.0
60.6
0.5
1.0
0.0
87.6
91.9
53.4
61.8
0.33
1.0
0.0
87.6
91.4
48.4
63.0
Alter.
0.2
1.0
0.0
86.5
91.6
48.6
63.5
0.5
0.99
0.0
86.7
91.6
50.6
63.4
0.5
0.999
0.0
87.3
91.9
53.4
61.5
0.0
1.0
0.02
84.6
90.6
44.3
60.6
0.0
1.0
0.05
84.2
90.8
43.5
59.3
0.0
1.O
0.10
85.7
90.9
33.8
57.9
0.0
I.O
0.20
86.8
90.7
41.4
62.0
1.0
1.0
0.02
85.6
91.8
32.5
59.8
1.0
1.0
0.05
85.9
91.7
31.5
59.6
1.0
1.0
0.10
88.3
91.8
31.2
59.4
1.0
1.0
0.20
86.0
91.7
29.3
61.6
Full-Precision
92.8
69.2
21

---

Under review as a conference paper at ICLR 2021
Table 17: The optimal hyper-panee parameters searched by Bayesian optimization when Altemaatig quantization method is utilized and is the number of quantization bits.
Alternating
Proposed weighted quantization
Test
E
P
Train
Test
Acc.(%
[0.2, 1.21
[0.7, 1.0]
10.0, 0.3]
Acc.(%)
Acc. (%)
0.4490
1,0000
0.0121
62.3
54.0
43.3
Layer-wiss (Fig.
T0p
66.4
56.5
ResNet- 18 (ImageNet)
0.5253
0,9501
0.1780
73.0
64.2
60.1
Layer-wise (Fig.
10
74.8
65.3
0.5311
0.9783
0.0085
76.3
68.1
66.8
Layer-wise (Fig. 1.0000
TOp
77.4
68.5
0.4482
0.1846
96.96
89.36
84.9
ResNet-32 (CIFAR-10)
Layer-wise
98.33
90.60
1.0500
1.0000
0.2290
99.80
92.01
4
91.2
Layer-wise
99.85
92.32
E
1.2
1.2
1.0
0.8
0.8
0.6
0.5
0.5
0.2
0.2
0.2
10
15
10
15
5
10
15
Slit
4bit
Sbil.
Figure 10: E, C', and P values searched by layer-wise BO for ResNet18 on ImageNet. X-axis shows layer index and y-axis shows hyper-parameters optimized differently for each layer.
22


Reviews:
[{"review_id": "u4L_1ZNfm_G", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a weighted quantization framework that could be applied to general neural networks for language models. One core idea is that different parameters might show different sensitivity towards a loss function change. Thus it would make sense to take use of importance metrics to do weighted quantization across parameters.\n\nClarity:\n\nThe paper is clearly written. With a good introduction of related work and a pretty self-inclusive references to experiment setup. I also like how the paper is organized to motivate the inclusion of weight importance into quantization. \n\nOriginality:\n\nOriginality in this paper is mostly from introducing the concept of parameter (weight) importance and how it is defined and applied to the quantization problem. \nThis paper is based on an earlier paper (also new), e.g., BiQGEMM (Jeon et al., 2020), which paved the foundation  to support binary-coding-based quantization techniques to accelerate quantized neural networks. A magnitude-based importance metric is proposed and approved effective in the form of the binary codes. To fine-tune model accuracy, three hyper-parameters are explored and empirically investigated to discover best performance (regarding model quality).\n\nSignificance:\n\nBased on the experiment result, I agree that this paper's contribution is significant. If we combine the effort of quantization with distillation, as shown with the metrics in DistillBert, we could achieve great ability of inference speed, a very smaller model size but still maintain reasonable model quality.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pLG4_cx9Ib7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- This paper proposes to consider the importance of each parameter in the post-training quantization of weights. The authors propose to use weight magnitude as the importance indicator and to minimize the weighted distance between the full-precision weights and quantized weights. Experiments are performed on various NLP models and tasks.\n\nStrengths: \n- The paper is structured clearly and the proposed method is simple.\n\nWeaknesses: \n1. It is not clear to me why the change in the loss in (7) and (8) is necessarily related to the magnitude of the weights. From (8), the loss change \\delta L  is only related to the hessian, and each weight's perturbation, but not the weight's magnitude.\n\n2.  The second concern comes from the novelty of the proposed method. Indeed, loss-aware (or importance-aware as in this paper) quantization which considers the quantization effect of each parameter to the loss has already been proposed several years ago in [1,2]. In [1,2], they also approximate the loss using the second-order Taylor expansion like in equation (7) in this submission. Moreover, the proposed importance-aware quantization solution in equation (5)  is exactly the same as equation (8) in [1], except that their importance is derived from the diagonal hessian instead of weight magnitude (which is not quite reasonable, refer to the first point).\n\n3. One other post-training quantization method GOBO in [3], which also uses codes and codebooks to quantize language models, is also not compared. From their reported results, under the same number of bits, GOBO has higher accuracy than the proposed method. E.g., GOBO has 83.76% accuracy for the 3-bit quantization on MNLI  while the proposed method only has 82.9%.\n\n4. From Table 2, there is no winning configuration of (E,C,P) that works well on all studied models and tasks. How to determine these hyperparameters for  new tasks empirically? Tuning these parameters separately for each task can be inefficient.\n\nReference:\n\n[1] Hou et al. \"Loss-aware binarization of deep networks.\" International Conference on Learning Representations. 2017.\n\n[2] Hou et al. \"Loss-aware weight quantization of deep networks.\" International Conference on Learning Representations. 2017.\n\n[3] Zadeh, Ali Hadi, and Andreas Moshovos. \"GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference.\" arXiv:2005.03842, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lplaSf7nsXM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Based on a previous classic binary coding scheme, this paper proposed to introduce a modification $m_i$ on the binarization scaling factor $\\alpha$, by considering the weight magnitude. It further use 3 hyperparamters to refine $m_i$ by constraining its upper/lower bound and exponent. Besides,  this work spent lengthy content to describe how to determine the hyperparamters.\n\nThis paper contains the following drawbacks:\n1. The proposed post-training quantization method has no connection to language model. All we can say is that: Post-training weighted quantization of neural network *in* (or applied in) language models. As author also mentioned that this method also works well in image tasks.\n2. There is a confusion on experimental setting: The quantization should be applied to the *fine-tuned* model. But it seems that author didn't pay attention to the disambiguation of pre-trained and fine-tuned, as I am quite lost in Sec.4.3 where author mention that \"perform post-training ... using *pre-trained* models of BERT-base ... on MNLI and MRPC.\" And didn't emphasize that the quantization is conducted on the fine-tuned models by MNLI/MRPC.\n3. Author spent much effort on how to determine the hyperparameters, which is also one defect of the method: it requires exquisite tuning on hyperparameters, which are sensitive as shown in Table 2.\n\nQuestion:\n1. How does the work deal with parameters in batch normalization layers?", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNM1QCfsq-z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper employs the binary-coding-based post-training quantization (without retraining) for language modeling. The key contribution is that weight importance is considered while determining binary code (a, B). Two methods, Greedy and Alternating, are also modified to use the importance. The algorithm uses a novel normalized importance, which directly uses weight magnitude and some hyper-parameters. Because the performance is sensitive to hyperparameters, Bayesian optimization is used to find task- and model-specific settings.\n\nAdopting weight importance to previous algorithms is quite novel and makes sense. Binary-coding-based quantization is recently introduced but seems to be quite a promising direction, so this paper has some significance. The idea to replace pruned parameters with the smallest ones is also smart.\n\nThe choice of E, C, P as controllable parameters seems reasonable. However, it looks like these parameters are sensitive and differs much task-by-task and model-by-model, that the robustness to the hyper-parameter is not ensured.\n\nMajor questions:\n\n1)\tI am not sure that in Section 4.1, Equation (7) and (8) properly support the state “weight magnitude can be a dominating factor for the loss function perturbation”. ‘Delta-L’ is a function of both ‘h’ and ‘delta-w’, not ‘w’ itself directly. (I might be misunderstood…)\n\n2)\tCan you provide some explanation (or guess)? (a) why sometimes greedy method is better than alter method (b) when does the pruning helps.\n\nMinor comments:\n\n1)\tIt would be great if a detailed scaling factor sharing scheme inside the parameter is given. For example, row-wise/column-wise/block tiling for a weight matrix?\n\n2)\tWhen the sample dataset (all or some) is not given (that’s why post-training quantization is valuable), how can we select proper hyper-parameters? BO maybe not applicable.\n\n3)\tThe term “Original” in tables may be misleading that the values are from the floating-point baseline. For example, for Table 1, I recommend clarifying three cases: float baseline, greedy without importance (previously “Original”), greedy with importance (E=1.0).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper builds upon a recent paper BiQGEMM, providing a binary coding based post training quantization technique. The authors show how to combine magnitude-based importance metrics to these techniques and achieve superior performance. The use of importance metrics for quantization and pruning is not new, and magnitude-based metrics are among the more common metrics. With that in mind, the novelty of the paper is in the integration of importance metrics to the techniques of BiQGEMM. The provided methods lead to several hyper-parameters and the task of tuning these can be non-trivial and time consuming. Due to this the authors devote a detailed section showing how to properly tune these hyper-parameters. This is appreciated and indeed alleviates the problem coming with new hyper-parameters.

The paper received mixed opinions by the reviewers related to its overall novelty, but the resulting conclusion is that although the combination of binary coding based quantization with importance scores is not trivial, the challenges faced relate more to correct implementation as opposed to scientific novelty. Combined with other issues raised by the reviewers such as a need for further comparison with existing work, this lead me to recommend rejection for this paper.

Decision: Reject