Towards Efficient Post-training Quantization of Pre-trained Language Models
Authors: Haoli Bai
Lu Hou
Lifeng Shang
Xin Jiang
Irwin King
Michael Lyu
Venue: NeurIPS 2022 Accept (2022)
Keywords: post-training quantization
BERT
natural langauge processing
training efficiency

Abstract: Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow quantization-aware training~(QAT) that requires end-to-end training with full access to the entire dataset. Therefore, they suffer from slow training, large memory overhead, and data accessibility issues. In this paper, we study post-training quantization~(PTQ) of PLMs, and propose module-wise quantization error minimization~(MREM), an efficient solution to mitigate these issues. By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module. In addition, we design a new model parallel training strategy such that each module can be trained locally on separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., $4\times$ on $4$ GPUs). Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.

Towards Efficient Post-training Quantization of Pre-trained Language Models
Haoli Bai Lu Hou', Lifeng Shang', Xin Jiang', Irwin King², Michael R. Lyu²
HHaawii Noah's Ark Lab, "The Chinese University of Hong Kong (baihaoli, houlu3, Shoulu3,Shhanl (king, 1yu cuhk, edu.hk
Shang. SchangLifeen Lifeng, Jiang Lang.Xinneae kin]@hhaweel com,
Abstract
Network quantization has gained increasing attention with the rapid growth of large pre-trained language models (PLMs). However, most existing quantization methods for PLMs follow quantization-awee end training with full access to the entire dataset. Therefore, they suffer from slow training. large memory overhead, and data uccessibility issues. In this paper, we study post-training quantization (PTQ) of PLMs, and propose module wise quanti- zation error minimization (MREM), an efficient solution to mitigate these issues. By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module. addition, we design new model parallel training strategy such that each module can be trained locally separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., 4x on 4 GPUs). Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
training (QAT) that requires end-to-
Introduction
Large pre-trained language models (PLMs) have achieved remarkable success various natural
language processing tasks 12 [5). However, the increasing size and computation overhead also make it prohibitive to deploy these PLMs on resource-constrainee devices. To obtain compact PLMs, various model compression methods have been proposed, such pruning (3) 44 50) knowledge
distillation 40 42
weight-sharing [T0
25
4o 22
dynamic computation with adaptive depth or
width 5 and quantization [55
Among these methods, network quantization enjoys the reduction of both model size and computa- tion overhead without modifying the network architecture, and is thus extensively studied in PLM quantization (55] 41 144, [56]
R. 38 43). However, most previous methods for PLM quantization methods follow quantization-awann training (QAT), which suffers from several issues. Specifically, QAT usually conducts end-to-end back-propagation training over the whole training set, which can be slow in training time, memory demanding and data consuming. These issues can be somethimes prohibited for industrial PLMs. Post-training quantization (PTQ), on the other hand, serves as an appealing alternative that is efficient in training time, memory overhead and data consumption. Instead of the full training set, PTQ can leverage only small portion of training data to mini- This can be
mize the layer-wise reconstruction error incurred by quantization [4
35
33
23
done by either calibrating the batch normalization statistics (34] or step sizes B51 in quantization
functions. The layer-wis, wise objective also breaks down the end-to-end training, making the prob-
TTiis work is partially done during internship at Huawei Noah's Ark Lab.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

lem more sample-efficient [58] and memory-saving. Nonetheless, we find it non-trivia to directly apply previous PTQ methods for PLMs such as BERT 12 as the performance drops sharply.
In this paper, we aim at improving the performance of post-training quantization for PLM, while simulta- neously maintaining its efficiency w.r.t training time, memory overhead and data accessibility. First, we partition the PLM into multiple modules, and pro- pose module-wise reconstruction error minimiza- tion (MREM). Each module consists of multiple Trans- former layers, which permits the sufficient optimization than the previous layer-wise objective. Meanwhile, the module size can be flexibly adjusted for the proper trade-off between layer-wise correlation and memory overhead of computing devices. While a similar block- wise objective is previously considered in [29], they require second-order Hessian matrices for optimiza- tion, which can be computationally prohibitive for large PLMs. Second, we design a new model parallel strat- egy further accelerate the training process of MREM. By allocating each partitioned module to the individual computing device, design allows all modules to be trained jointly without synchronizing with adjacent partitions. This brings nearly the theoretical speed-up (e.g., 4x on 4 GPUs), superior to previous data paral- lel M 27 or model parallel 21 36 techniques. Third, we develop annealed teacher forcing for our parallel strategy. We find that the naive parallel training suffers from the propagation of reconstruction error, since each quantized module passes the error to its successors before it fully optimized. Inspired by [5]]. we use the full-precision module to provide clean signals to the next quantized module. This breaks the reconstruction error propagation and further improves the performance of parallel strategy.
Accuracy (%)
$5.5
29.8
8.6
21
3,180
Memory (GB) Per GPU
Training Time (min)
QAT
383
#Data (K)
MREM
An illustrative comparison be- tween our parallel post-training quantiza- tion method (MREM) and QAT on four di- mensions of the quantization pipeline: accu- racy, training time, memory overhead, and data consumption. The results are based on a quantized BERT-large model with 4- bit weights and 8-bit activations over the MNLI dataset. Best viewed in color.
Figure 1:
We summarize the contributions of this paper as follows (i) We study the post-training quantization of PLMs, and propose module-wise reconstruction error minimization (MREM), a fast, memory- saving, and data-efficient approach improve the quantized PLMs. (ii) We design a new model parallel strategy based on MREM to accelerate post-training quantization with theoretical speed-up for distributed training. (iii) The parallel MREM can be combined with annealed teacher foreing to alleviate the propagation of reconstruction error and boost the performance. iv) Empirical results on the GLUE and SQuAD benchmarks demonstrate superiority of our PTQ solution over QAT. For instance, Figure 1 shows that the BERT-large model trained by parallel MREM achieves 85.5% accuracy with only 4K training samples. Moreover, consumes merely one-third of memory per GPU and is more than 150x faster than previous QAT training.
2 Motivation
2.1
Quantization Background
Network quantization replaces the original full- full-precision weight or activation x E T* with its lower-bit counterpart Denoting E IR+ as the step size, the b-bit symmetric uniform quantization function (.).) can be written as
& (x(x)
(1)
where ((b) (-2h-1 1,....00 0, 26-1 1} is the set of b-bit integers, and 11(-) is the projection function that maps x/s /s to its closest integer.
To quantize Transformer-bases PLMs, we follow the setting in previous works 55 56
we quantize
both the network weights and activations in each matrix multiplication. We use symmetric uniform quantization for weights, embeddings. and activations, except activations after the self-attention and GeLU function. For these two activations, we adopt asymmetric quantization since their elements are mostly positive. We skip the quantization for all normmaizoniiiooooonn layers, skip connections, biases

---

Training Time (minn)
3,180
20.5
393
QAT
PTQ
GPU Hemorr (GB)
MHU #Data (K)
Accuracy (%)
21.5
75
11.0
GTX 1000
65
609
5.5
84
PTQ
PTO
W4A8
W2AB W244
(a) Training Time.
(b) Memory Overhead.
(c) Data Accessibility.
(d) Performance.
Figure 2: Comparison between QAT and REM-based PTQ over four dimensions. We use BERT- large model over MNLI dataset for illustration. The full-precision (FP) fine-tuning is also included as baseline. We follow the procedure in [5 for QAT, and REM in Equation 2 for PTQ. The training time and memory in (a) and (b) are measured by 4-bit weights and 8-bit activations (i.e., W4A8) NVIDIA V100 GPU.
and the last classification head due to little computation overhead or large performance drop. We leave more details the choice of quantization functions and their algorithmic details in Appendix A
constructs an tiny unlabeled dataset (a.k.a. calibration set) DD from the original training data D. These approaches usually aim reconstruction enror minimization (REM), i.e., minimizing the distance between the output of multiplication between the quantized and the full-precision counterpart as follows:
Post-training Quantization. Recently popular PTQ methods
35
33
23
30
min | â w waal?? s.t. w Qow (w), â = == (a),
(2)
where w and a are full-precision weights and activations, W and are their quantized representations with bw and b, bit-widths, and denotes all step-sizes involved for quantization. REM is usually conducted in a greedy manner. It proceeds to the matrix multiplication only after the training of previous ones. The simple objective in Equation ( is lightweight to solve over D, and is also theoretically verified to be more sample-efficient than end-to-end training (58). While there also other data-free PTQ variants (57
16 (4], REM- -based solutions usually demonstrate better Thus mainly focus the REM-based PTQ for PLMs in this paper.
empirical results
33 23
2.2 Quantizing Pre-trained Language Models: QAT or PTQ?
To conceptually study the difference between QAT and REM-based PTQ over popular PLMs, we consider the following four dimensions, and summarize the results in Figure
Training Time. As QAT conducts training over the full training set D, it takes much more time than
PTQ over the calibtration set D. Recent QAT methods [50 3 further combine two-stage knowledge distillation 24 which can take nearly four times longer than FP, as shown in Figure 2(a).
Memory Overhead. The increasing size of large PLMs makes it prohibited to conduct QAT on nemory--lmiter devices. From Figure 2 QAT (56) even consumes 8.3 GB more memory than FP when armed with knowledge distillation. On the contrary, PTQ only caches intermediate results in which can be fed into single GTX 1080 Ti.
Equation 2
Data Accessibility. Data accessibility can be often prohibited due to data security or privacy issues in the industry. Unlike QAT that needs the full training set PTQ constructs the calibration data D C D by sampling only 1K 4K instances from D, as shown in Figure 2(c)
Performance. QAT usually demonstrates superior quantization results than PTQ, and this holds true for PLMs. From Figure 2(d) QAT remains steady across different bit-widths, while REM-based PTQ drops sharply. Thus the poor performance PTQ the main challenge in quantizing PLMs.
In summary, REM-bassd PTQ is superior to QAT with regard to training efficiency, memory overhead, still often less preferred than QAT due to its severe
and data accessibility. Nevertheless, performance drop especially for low quantization bit-width [55 41 56).
3

---

Full-prec. Model
Quant. Model
Forward Pass
Backward Pass
Teacher Forcing
Device:
Device:
Device:
22
D
Input Queue
Calibration Set
I1
12
Figure 3: The overview of the proposed module-wise reconstruction error minimization (MREM). We partition both the full-pree cull-preciioo model and quantized model into multiple modules and put these modules on different computing devices. By sampling tensors from the input queue, each module can be trained locally without waiting for its predecessors. Teacher forcing is applied to mitigate the issue of reconstruction
propagation the quantized module.
3 Methodology
In this section, we aim at improving the performance of post- cost-training quantization for PLMs, while preserving its merits of fast training, light memory overhead, and data consumption. We extend the existing reconstruction error minimization from the layer-wise to the module-wise granularity to fit Transformer models. Based the module partition, then design that further speeds up the PTQ pipeline. An overview our solution can be found in Figure
parallel training strategy
3.1 Modulc-wise Reconstruction Error Minimization
We propose new PTQ solution called module-wise reconstruction error minimization (MREM) PLMs. Existing REM 3 solves Equation (2) for each matrix multiplication. However, standard transformer layer in PLMs consists of a Multi-Head Attention (MHA) and Feed-Forward Network (FFN), both of which contain a number of matrix multiplications that are coupled together. Greedily tackling each matrix multiplication in REM thus lead to suboptimal quantized networks. Moreover, the insufficiently minimized reconstruction eтror shall propagate and enlarge along with transformer layers, and finally deteriorate the network output 6
Towards that end, the proposed module- wise reconstruction error minimization admits larger granu- larity by jointly optimizing all the coupled linear layers inside each module. Specifically, given PLM with transformer layers, embedding layers and the classification head, we partition them into N modules, where the n-th module include (nn,4n+) 4n++) transformer layers with I, being the first layer of this module MREM aims at minimizing the joint reconstruction errors between all quantized FFN output f. in the module from their full-precision counterpart f1 as follows:
min Wr.
(())
Σ
1 1f1-
(3)
LE(Lnda=s)
where w,, and s, are all learnable parameters and quantization step sizes in the n-th module. Similar to REM, MREM can be optimized sequentially: given previously trained modules, only parameters and quantization step sizes in the current module are updated. Besides the grouped Transformer layers, also minimize the MSE loss in the Transformer embedding and output logits respectively.
Note that the number of modules can be adjusted depending on the memory constraint of computing resources. When N = 1, this reduces to intermediate- -lyer knowledge distillation [24] which can be memory-demanding when quantizing large PI.Ms on a single GPU.
3.2 Accelerated Parallel Training
Based on the proposed MREM, we propose a new model parallel strategy to further accelerate the put different modules on individual computing devices. A set of
training. As shown in Figure 3
"The embedding layers and the classification head are incorporated in the first and last module respectively.

---

input queues I = [II.... is deployed between each pair of adjacent modules. For the n-th module, the queue collects its output of the most recent to steps, i.e,, = (fí., f "',",,.,,,, ..... fi. tot1). Meanwhile, the (n + 1)-th module can always sample with replacement fi, I from the queue without waiting for the n-th module. Similar rules hold for the quantized module and their input queues İ. The design of the input queue resembles stale synchronous parallel [17] that caches the stale output locally so as to reduce the waiting time among workers, where to is the stale threshold.
The training workflow is as follows. Initially, every module is computed one after another in the first to step to fill in the input queue, after which parallel training takes place. Then the module samples input from the queue and calculates the loss ((") correspondingly for n ......VV .V. Mcanwhile, the input queue is updated with the rule of first-in-firstoou throughout the training. In the backward pass, constrain the gradients to propagate locally within each module, without affecting its predecessors. Such a design can avoid the load imbalance issue from straggler modules, bringing nearly the theoretical N speed-up.
The proposed parallel training strategy is superior to the conventional data parallel training in quantizing PTMs. Meanwhile, it is also different from pipeline parallelism 21] We leave the detailed comparisons with these parallelism techniques Appendix C
3.3 Annealed Teaching Forcing
Since all modules get optimized simultaneously instead of the sequential manner, the next module takes the output from the queue before its predecessor is fully optimized. Therefore, the predecessor's reconstruction error is propagated to the following modules before it is sufficiently minimized.
Inspired by teacher forcing 51 in training recurrent networks, the output ft, from the n-th full- precision module naturally serves the clean input to the (n + 1)-th quantized module to substitute
JT.- Thus f1, stops the propagation of the accumulated error the quantized module. Nevertheless, such an approach breaks the connection to previous quantized modules and may suffer from forward inconsistency between training and inference [2] on the quantized model. For the trade-off, we take the convex combination between the full- full-precision fi, and quantized ft, follows:
fr, Af,, - (1- XE |
(4)
where the hyperparameter À controls the strength of teacher forcing. À 1 gives the full correction of reconstruction eгror but with forward inconsistency, while À = 0 reduces to the conventional setting that suffers from the propagated reconstruction error. We adopt linear decay strategy for A: X = max(1 1 +. 0). where To is the preset maximum steps of the decay. Intuitively, a large À is T..
desired at the beginning when each module is rarely optimized. Later, a small À is preferred to transit normal training such that the forward inconsistency can be bridged. The remaining T To steps stick to normal training so that each quantized module adapts to its own predecessors.
Finally, overview of the proposed parallel module-wise reconstruction error minimization with annealed teacher forcing is shown in Algorithm and Algorithm 2 The Update(-) in Algorithm can be any gradient update function such as Adam W G1 with learning rate 17"..
Algorithm 1 Efficient PTQ for PLMs.
Algorithm 2 MREM algorithm.
I: procedure Main O:
procedure MREM (f:,,
2: Partition the PLM into N modules.
2: if
To then
3: Fill in the input queues I. Z.
3: X max(1 (1
Tg 0).
4: for 4- 1, 1,.., N. do
4: Compute Ji.., by Equation
run in parallel.
5: Compute the full- precisrcoi module output ft.
6: while < T do
6: Compute the quantized module output J
Si,m1 I'n-1 Plo- I
7:
7: Compute the loss g(x) by Equation 3
8:
Pi +- MREM (fin-n 1s 1n
8: w 1+=1
+- Update(w:, 7ww. ,7
ae(")
9: Update I', I", with Si, Pt
9: s+1 Update(s def, .3f )
ac(r)
10: Return the Quantized PLM.
10: Return fi,, Fi,
5

---

4 Experiments
In this section, we empirically verify the proposed MREM for post-training quantization of PLMs. We first introduce the experimental setup in Section I.1 Then we present main results in Section 4.2
including comparisons with QAT and REM, and other existing quantization baselines. In Section 4.4
we provide discussions variety of factors in our approach, such as the effect of teacher forcing, the number of model partitions, and calibration data size. Our implementation is based on MindSpore ||
4.1 Experimental Setup
Datasets and Metries. We evaluate post-training quantization on both the GLUE 45 and SQuAD benchmarks [39] The size of calibration data is by default [D] 4. 096, by randomly sampling instances from the full training set. As both RTE and MRPC tasks in the GLUE benchmark contain fewer than 4,096 samples, we use their full training set on these two tasks. We leave the study of data size in Section 4.4, Each experiment repeated ten times with different calibration sets, and both the mean and standard deviations are reported.
We use the same evaluation metrics in 12 benchmarks. For results in Section mis-matched sections of MNLI, and EM (exact match) and F1 score for SQuAD. Additionally, we also report the training time (min), memory overhead (GB) as well as the size of the training set (K). We also provide comparisons with other existing methods in Section 4.3 where we adopt Matthews correlation for CoLA, Spearman correlation for STS-B, and accuracy for the rest ones (i.e., RTE, MRPC, SST-2, QQP, MNLI). We also report the averaged performance on GLUE an overview.
56 for the development set of GLUE and SQuAD
we report accuracies on both the matched section and
Implementation. We use the standardly fine-tuned BERT-base and BERT-large model on down- stream tasks for both quantization-awwat training and post-training quantization. We implement MREM in both the sequential training (abbv. MREM-S) in Section 3.1 and parallel training with teaching forcing (abbv. MREM-P) in Section 3.3 For each module, we train for 2, 000 steps with an initial learning rate of 1e-4 on GLUE tasks, and 4, 000 steps with an initial learning rate of 5e-5 on SQuAD datasets. The learning rate decays linearly as done in [24 56). By default, we partition the model into 4 modules 4 VVIIIA-VION GPUs. For baselines,
mainly compare with QAT and REM, where the former measures how far is PTQ from QAT, and the latter studies the effect of module-wise granularity in PTQ training. For fair comparison of each method, we use the same quantization scheme for all methods, i.e., TWN 26 or LAQ [99 for 2-bit and 4-bit weight quantization, and LSQ LK for all activation quantization. Unlike QAT that picks the best model based on the development set results, MREM is only tested once after training, which does not require further access the development set. We leave more details of baseline implementation in Appendix B.I. The comparison with more published quantization baselines are left in Section 4.3
4.2 Main Results: Comparison with QAT and REM
We first compare MREM-S and MREM-P with QAT and REM over MNLI and SQuAD benchmarks. We take BERT-base and BERT- large backbone PLMs for quantization. The results MNLI and respectively, and results on SQuAD v2.0 are left We again summarize according to the four dimensions in Section 2
SQuADv1.1 are summarized in Table and Table 2
to Table in Appendix B.2
Performance. It can be found that our proposed MREM-S improves the performance of REM significantly given the same training time, and is much closer to QAT. For instance, according to in Table [ MREM-S with 4-bit weight quantization on BERT-base and BERT-large achieves accuracies of 83.5%=oo1 and 86. 1%.1% on the matched section of MNLI, which is on average 10.2% t and 16.1% better than REM, and only 1.1% and 0.8% inferior to QAT, respectively. Moreover, with all modules trained in parallel, MREM-P is still close to or only slightly inferior to MREM-S. From Table MREM-P can even outperform MREM-S with the "W2-E2-A4" quantized BERT-large model on SQUAD 1.1, i.e., the EM score and F1 score are on average 0.4% 1 and 0.2% respectively.
Training Time. Our proposed MREM also enjoys significantly less training time than QAT. From Table MREM only takes 84 minutes to quantize BERT-large with 4-bit weights, which is about 38x
We follow the default fine-tuning settings in Huggingface: Huggingface: http://gggaabbooo/nnnnggggittttn

---

Table I: Results of our proposed MREM-S and MREM-P against QAT and REM the development set of MNLI. "#Bits (W-E-A)" represents the bit-width for weights of Transformer layers, word embedding. and activations. Ace-m and Acc-mm denote accuracies the matched and mismatched sections of MNLI respectively.
BERT-base
BERT-large
#Bits (W-E-)
Quant Method
Time (min))
Mem
DDaaa
Acc
Acc
Time
Mem
# DDaaa
Acc
Ace
(GBN)
(Ki).
m(()))
mmn(%8)
(min)]
(GB))
(K),
m(%o
mm((S)
Jill-prec
N/
220
8.6
393
81.5
81Y
609
21.5
393
86.7
85.9
QAT
320
11.3
393
84.6
84.9
3. 180
29.8
393
86.9
86.7
REM
28
2.5
73.340.
74.9.02
84
5.5
70.020.4
71.840.
4-4-8
MREM-S
36
4.6
83.520.1
83 83.9
84
10.8
86. 12.1
85, 14..1
MREM-P
3.7
83.4
83.7.0.
21
3.6
85.5
85.4g0.
MNLI
QAT
582
11.9
393
84.4
84.6
2, 340
29.8
393
80.5
86,1
24
71.620.4
23..
5.5
66.9
68.6
2-2-8
MREM-S
24
4.6
82.7+0.2
82.740.2
64
10.8
85. 85.420.2
85.3+n.2
MREM-P
3.7
82.3
82.6.o.2
16
8.6.a
84.6202
84.6
QAT
875
11.9
393
83.5
812
280
29.8
393
85.8
85,9
REM
24
2.5
58.3
60.6
64
5.5
48. 28.8
51.4gns
2-2-4
MREM-S
24
4.6
81.1
81.5402
64
10.8
83.610.2
83.74a.2
MREM-P
6
3.7x4
80.8
81.2 81.2g02
16
8.6x4
4
83.0
83.2
Table 2: Results of our proposed MREM-S and MREM-P against QAT and REM on the development set of SQuAD v1.1. denotes results with two gradient accumulation steps under the same total batch size due to memory constraint.
BERT-base
BERT-large
Whits (W-E-A)
Quant Method
Time [min)]
Mem (GB)
#Data (K)
EM (#R)
F1 ($6)
Time (min)]
Mem
#Data
EM (9))
FI (%6)
(GB)
(K)
jull-prec.
11.7
85.7
30.4
93.1
QAT
428
18.4
88
802
87.9
1,920
27.0
88
86.7
B3.D
4-4-8
REM
65
3.1
26.1
60.0.0.5
175
7.3
4
68.320.1
29.3
MREM-S
76
6.4
79.420.1
S7.2.0.1
200
14.5
4
86.220.1
92.5.0.1
vL.
MREM-P
19
5.5
79.6.0.1
87.3-0.1
50
12.3
86.0ga.1
g2.4.0.1
QAT
335
18.4
79.3
87.2
I. 200
27.0
86.1
92.5
2-2-8
REM
60
3.1
4
40.1
65.020.4
160
7.3
4
60.420.5
77.7=0.3
MREM-S
60
6.4
4
77.8=0.2
86.020.1
156
14.5
4
85.4...1
91.920.1
MREM-P
15
5.5xs
4
7T.7
85.9.0.2
39
12.3.x
4
85.3
91.8.0.1
QAT
331
18.4
77.1
85.9
1. 188
27.0
88
S4.7
93.1
2.2.4
REM
60
3.1
10.4g0.
21.2002
160
7.3
4
28.3
45.0g0a
MREM-S
69
6.4
4
72.7g0
82.5goa
156
14.5
4
81.4g03
89 89.4goa
MREM
15
5.5%
73.0ga.
82.740.
39
12.3.
81.8e0
89.6gng
faster than QAT and 7x faster than full-precision fine-tuning. Note that the full-precision fine-tuning time is listed to conceptually compare how much time is further required for quantization. Comparing with REM, MREM does not cache the output after every matrix multiplication, which admits more iterations given the same amount of time. We discuss this further in Appendix [.3]. Moreover, MREM-P is further 4x faster than MREM-S, which achieves the theoretical linear speedup 4 GPUs. These together bring more than 150x reduction of training time when compared with QAT.
Memory Overhead. Our MREM-S and MREM-P take only around a third of the GPU memory by QAT, and half of that by the full-precision fine-tuning, e.g., 29.8 GB for QAT and 10.8 GB for MREM-S on BERT-large. Moreover, we also encounter memory overflow (more than 32GB memory of an NVIDIA V100 GPU) for QAT training on SQuAD due to longer sequence length, and adopt gradient accumulation that inevitably doubles the training time (i.e., underlined figures (* ") in Table 2 On the other hand, such issues can be easily mitigated in both REM and our proposed MREM, both of which can be even fed into a single NVIDIA GTX 1080 Ti GPU. We may further decrease the memory overhead of MREM to REM by increasing the number of modules, but this could harm the performance as discussed in Section
Data Accessibility. Both REM and our proposed MREM follow the common practice of PTQ, relying on only 4, 096 randomly sampled instances on both MNLI and SQuAD, which is tiny fraction of the original dataset. We shall provide more discussion on the calibration size in Section
4.3 Main Results: Comparison with Existing Methods
Next, we compare our MREM with existing otate-of-the-art BERT quantization methods. They include various QAT approaches such as Q-BERT 41). Quant-Noise [15], TemaryBERT [56], and
and
7

---

Table 3: Results the GLUE development set. "Size" refers model storage in "MB". "PTQ" indicates whether the method belongs to post-training quantization. "Avg." denotes the average results of all tasks.
Quant Method
#Bits (W-F-A)
Size (MB)
PTQ
MNLI-m
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg.
pill-pee.
418
84.9
91.4
92.1
B3.2
59.7
50.1
56.3
72.2
83.9
Q-BERT
2.8-8
43
X
76.6
84.6
Q-BERT
24-8-8
53
X
92.6
Quant-Noise
PQ
38
X
83.6
TomaryBERS BERT
2-2-8
26
K
83.3
90.1
91.1
92.8
55.7
$7.9
87.5
72.9
82.7
GOBO
3-4-32
43
V
83.7
68.3
GOBO
2-2:32
62.7
BRECQ
8-4-
61
31.9
62.3
50.7
50.9
0.9
6-4
31.7
52.3
35.9
QDeop
8-4-4
61
V
71.4
79.0
76.8
88.1
40.9
$1.9
79.2
60.7
72.3
REM
4-4-8
28
70.0
44,9
26-1
88 88-62a2
36-3-10
82.1 #0.0
67.220.0
76.3202
REM
2-2-8
28
72.820.5
8
81.9
88. 1s00
322 22.02117
50 00.0020
81,1
63.3
78. 73.6208
REM
2-2-4
28
58 38.3
70.7
76. 3e.3
82 82.9ga
16.4222
44.02114
74.0ana
63.229.
61 21.2.002
MREM-S
4.4.8
50
K3.
10.2
91.2
91 91.410a
66.1
19.1
82.4163
28
82.7202
89.620.1
90.3gaa
91 01.2g004
52.9.10
88.7.01
56.0100
71.1
81.3202
2-2-4
28
81 20.1
88.7+0.1
89-4
90.800
46-4211
87.7201
8388408
70.829.9
80.0g04
MREM-P
4-1-8
50
83. 83.4.0.1
90.2+8.1
91. 91.0ga2
91 91.3e0
51.7499
69.1401
86.3400
71.1409
82 82.2
28
82.3
89.4
90.3
31.3
62.9-12
88.3
85.8
72.920.0
2-2-4
28
80.8 00.
88.
88.9 0.8
90.7 0.6
19.3 nm
85.3
70.4
80.3
Table 4: Ablation studies of teacher forcing at Table 5: Comparison of BERT-base with and with- different training steps over MNLI-m.
out per-channel quantization (PCQ) on MNLI.
#Bits (W-E-A)
BERT-base
BERT-large
Steps
250
79.5
w/o TF
w/ TF
=0.2
82.
w/o TF
22.1
13.1
w/ TF
#Bits (W-E-A)
Methods
w/o PCQ
Ace
w/ PCQ
2-2-8
2,00 000
500
0000
82.240.2
81.0a0.s
82.3.0.5
81.6.0.2
82.7=202
82.5..02
83.42.0.5
84.322.5
84.5.002
84.6-0.2
84.1
84.7-m.2
4-4-8
MREM
REM
83.
13.320.
mi %)
$3.9202
743300
mmn(%)
83.6a0.1
15.920.3
mi
84.020.1
Tr.420
mmm(%)
2-2-4
2,000
250
500
77.920.2
80. 80.44.0.
73.9
71.3_0.4
80.8
79.0-0.2
76.5
82.5...4
80.0=006
79.3_a.4
83.0_o.
81.4-0.2
2-2-4
2-2-8
MREM
REM
REM
58.320.5
82.7
7L.820.
73.4204
82.7.03
82.840.1
50.320.4
74.120.5
62.020.
82.9.0.1
75.620.
4,000
80.7.0.2
81.0-0a
83.1
83.3-0.5
MREM
81. 81.1
81.5g02
81.1a0.2
81 81.5g0.3
the PTQ baselines including GOBO 54
BRECQ 29 and QDrop 48)
The results are from the
original papers. Additionally, we also compare our MREM with REM.
From Table 3 both our proposed MREM-S and MREM-P outperform existing PTQ approaches in most cases. For "W2-E2-A8" quantized models, MREM-S and MREM-P surpass GOBO by 11.7% f and 11.3% f on MNLI-m respectively. Both BRECQ and QDrop lag behind our approach even with the configuration "W4-E8-A4" For instance, the average scores on GLUE have the gaps of 9.3% and 8.0% between QDrop and our "W2-E2-A4" quantized MREM-S and MREM-P, respectively. Our results are even close to QAT approaches in multiple entries. For example, the "W4-F4-A8" quantized MREM-S and MREM-P have the mean accuracies of 83.5% and 83.4% on MNLI respectively, both of which are on par with the "W2/4-E8-A8" quantized Q-BERT.
4.4 Discussions
In this section, we provide further discussions better understand the proposed approach. Unless specified otherwise, all experiments are built upon the BERT-base model over the MNLI dataset
Teacher Forcing. We now study how teacher forcing benefits MREM-P with different numbers of training steps, and results are listed in Table 4 We find that teacher forcing brings consistent improvement for both BERT-base and BERT-large, and the gain is more significant with fewer training steps or lower quantization bit-w.dth. For example, it brings 3.4% 1 and 2.8% 1 on the "W2-E2-A4" quantized BERT-base and BERT-large respectively under 250 steps. This matches our intuition that fewer training steps or higher compression ratio give larger reconstruction error, when the clean input from the full-precision module can benefit more the quantized module. As more training steps bring only marginal improvement, we by default set the training steps to 2, 000.
Additionally, we also plot training loss curves of the 2-nd and 4-th modules under 250 and 2,000 training steps in Figure g and more visualizations can be found in Figure of Appendix B.4 We find that: 1) the loss curves with teacher forcing are apparently lower, especially when trained with fewer steps, which is consistent with Table g 2) the loss curves rise in the halfway during teacher forcing, since the modules need to adapt to quantized input with accumulated errors; 3) the loss curves of the 4-th modules are lower than the 2-nd ones, which matches the intuition that the late modules have
8

---

1:25
Training Lods
TF
Training Loos
WI
Training Loss
TF
1.00
2.0
Treining
0.75
0.50
0.5
0.25
50
100
150
200
25g
90
300
130
200
250
400
550
1200
1800
2000
400
500
1200
1600
2003
(a) 2 (250 Steps). (b) Module-4 (250 Steps). (c) 2 (2000 (t) Module-4 (2000 Steps).
Figure 4: The training loss curves with and without teacher forcing (TF) in MREM-P. The shaded area denotes teacher forcing in the first 40% training steps. We show the 2-nd and 4-th modules trained with 250 steps and 2,000 steps (a). (b) and (c), (d) respectively.
0.50
RBM
0.30
ROM MAEN-S
MAEK-P MREM
MRCH-S
81.5
Accurscy (N)
70-
0.25
0.20
0.20
Accuracy
80
Bec
0.15
0.10
0.10
80.5
40
MAEN-D
0.05
0.05
80.0
5
0.00
0.00
8,00
3:708
(a) Number of Modules and (b) Size of Calibration Memory Overhead.
Error
Propaga- (d)
Error
Propagation
Data.
tion (A8).
(A4).
Figure 5: Discussions on the proposed MREM approach. In (a) and (b), the solid line and shaded area denote the averaged results and standard deviation of "W2-E2-A4" quantized BERT-base model from 10 different seeds. (c) and (d) visualize the propagation of reconstruction error on "W2-E2-A8" and "W2-E2-A4" quantized BERT-base model, respectively.
more errors accumulated and thus benefit more from teacher forcing. We also try tuning the teacher forcing steps (i.e., shaded area) within f20%, 80%] of total iterations, but observe no large difference in performance. We thus choose 40% training steps for teacher forcing by default.
Number of Modules and Memory Overhead. We verify the effect of model partition on the final quantized performance, well as their corresponding memory consumption. According to Figure ((a) by varying the number of modules within 21, 3, 6). it can be found that more model partitions give slightly lower performance, as layer-wise dependencies are less considered during reconstruction error minimization. However, more partitions lead to less running memory, i.., {11.9, 6.7, 3.7, 2.7) GB for these partitions correspondingly. As the decrease of memory diminishes with more partitions, we partition the model into 4 modules by default.
Size of Calibration Data. The size of calibration data directly relates to the data accessibility issue in post-training quantization. To learn its effects, we vary the calibration data size |DI within {32, 64, 128, 512, 1024, 2048, 4096, 8192). and list the results of REM, MREM-S and MREM-P. From Figure 5 it can be found that while REM is ahead of MREM-S/P with fewer than 128 training samples, the accuracy of REM rises slowly and saturates at around 60% afterwards. We hypothesize that the simple training objective in REM can hardly hold more training instances for optimization. MREM-S/P, on the other hand, can better exploit larger calibration data size, since the module-wise granularity admits higher flexibility for the optimization. As find the diminishing gain to increase the training size after 4, 096 samples, we by default take 4, 096 samples.
Reconstruction Error Propagation. We visualize the propagation of reconstruction error for both "W2-E2-A8" and "W2-E2-A4" quantized BERT-base models in Figure [(c] and Figure ((d) respectively. It can be observed that our MREM achieves both lower values and slower rising rates of the reconstruction error than REM across all layers, which verifies the advantage of module-wise granularity to minimize the reconstruction emror. Interestingly, while the reconstruction error generally gets enlarged wisery the first ten layers, begins to decrease afterwards. We speculate this due to the classification head that encourages concentrated hidden representations for the task.

---

Per-channel Quantization. Per-channel Quantization (PCQ) is prevalent in the post-training 33 23 To learn its effect in PLMs, PCQ assigns different quantization step-sizes at each output dimension of the linear layer, which is also known as row-wise quantization in [56]. The PCQ results of REM and MREM are shown in Table 5 It can be found that while PCQ improves REM by around 1.0% to 2.5%, the gain is very incremental on MREM. Our results are also similar to the findings in [56). where the row-wise quantization brings little improvement. As PCQ requires to store more full-precision step sizes with minor improvement, we do not employ PCQ by default.
quantization of convolution neural networks 65
Related Work
Network quantization is a common approach to compress and accelerate deep neural networks g
26
20
19 3
For network quantization of pre-trained language models, previous efforts mostly
adopt quantization-aware training [55
41 56 3X For instance, Q8BERT
(55) converts both
parameters and activations with 8-bit representations with negligible task degradation. Q-BERT 41
exploits the hessian matrix of loss curvature to determine the layer-wise wise quantization bit-width,
achieving a higher compression rate. TernayyBERT [56] proposes to ternarize BERT parameters with 2-bit representations, while BinaryBERT K further uses 1-bit to represent model parameters.
BiBERT 38 explores the fully-binarized BERT with 1-bit weights and activations. Despite the success by these efforts, the heavy fine-tuning of QAT makes it prohibitive given constraints on training time, memory size, and data accessibility.
is thus considered mitigate these issues from QAT. One line of PTQ research quantizes the network purely without using any training data, but removes outliers in the full-precision parameters [57
Post-training quantization 37
34 1o 54 Methodologically. PTQ can be achieved by splitting an outlier neuron with large magmitude into two parts [57]. where the magnitude can be halved. Alternatively, one can scale down outlier magnitude and multiply it back in subsequent layers, a.k.a
weight equalization in 64 Another solution to treat the outliers and normal values in the distribution separately, by keeping two sets of quantization parameters [16 (44 Another line in PTQ
research [35 33 23 minimizes the reconstruction error based on the calibration data. Compared
with training- free PTQ approaches, such an approach significantly improves the performance of the quantized network, and we thus follow this line this paper.
Recently, there are also concurrent efforts with us to apply PTQ for PTMs 3V similar method is developed minimize the quantization error via the calibration data. Moreover,
In
PTQ is also explored for large generative PTMs (e.g. GPT-3 (5) from 125M to 175B
ty 53
6 Conclusion
In this paper, we study post-training quantization for pre-trained language models. We show that existing quantization-aware training solutions suffer from slow training, huge memory overhead, and data privacy issues when accessing the full training set. To mitigate these issues, propose module-wise reconstruction error minimization, an efficient solution to quantize PLMs. MREM can be conducted either sequentially or in parallel, where the parallel training can achieve the speedup close to the theoretical limit without apparent performance degradation. Experimental results show that the proposed solution greatly improves the performance. Meanwhile, it significantly reduces the training time and memory overhead with only thousands of training instances. We leave the broader impact and limitations of this work in Appendix D
Acknowledgement and Disclosure of Funding
We gratefully acknowledge the support of MindSpore for this research, as well as the insightful suggestions from the anonymous reviewers. The work described in this paper was partially supported by the National Key Research and Development Program China (No. 2018AAA0100204, the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 2410021 of the Research Impact Fund, No. R5034-18, and No. CUHK 14210920 of the General Research Fund).
10

---

References
[1] Mindspore. httpp:://w...i.idpoorr.. www.mmiinndd....mmm en
[2] Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 3203-3210, 2020.
[3] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. In Anmual Meeting of the Association for Computational Linguistics, 2021.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. Preprint
2013.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[6] Shangyu Chen, Wenya Wang, and Sinno Jialin Pan. optimization using limited training data. In Proceedings of the AAAI Conference Artificial Intelligence, pages 3329 3336, 2019.
Deep neural network quantization via layer- wise
[7] Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King. Learning binarized graph representations with multi-faccted quantization reinforcement for top-k recommendation. In Proceedings of the 28th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2022.
[8] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 2015.
[9] Jeffrey Dean and Sanjay Ghemawat. Mapreduce: simplified data processing on large clusters. In Communications of the ACM, volume 51, pages 107-113, 2008
[10] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. International Conference on Learning Representations, 2019.
[11] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers scale. Preprint, 2022.
[12] Jacob Devlin, Ming- Weig-Weiioe Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
[13] Steven K Esser, Jeffrey I. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra $ Modha. Learned step size quantization. In International Conference on Learning Representationss 2019.
[14] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2019.
[15] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme fixed-point compression. Preprint Kii:22004007320 2020.
[16] Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz. David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piecewise linear quantization for deep neural networks. In European Conference on Computer Vision, pages 86, 2020.
[17] Qirong Ho, James Cipar, Henggang Cui, Jin Kyu Kim, Seunghak Lee, Phillip B Gibbons, Garth A Gibson, Gregory R Ganger, and Erie P Xing. More effective distributed ml via stale synchronous parallel parameter server. In Advances in Neural Information Processing Systems, page 1223, 2013.
[18] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems, 2020.
[19] Lu Hou and James T Kwok. Loss-aware weight quantization of deep networks. In International Conference Learning Representations, 2018
[20] Lu Hou, Quanming Yao, and James T Kwok. Loss-aware binarization of deep networks. In International Conference on Learning Representations, 2017.

---

[21] Yanping Huang. Youlong Cheng, Ankur Bapna. Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee. Jiquan Ngiam, Quoe V Le. Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, 2018.
[22] Zhiqi Huang, Lu Hou, Lifeng Shang, Xin Jiang, Xiao, Chen, and Qun Liu. Ghostbert: Generate more features with cheap operations for bert. In Annual Meeting of the Association for Computational Linguistics, 2021.
[23] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. In Proceedings of the International Conference on Machine Learning, 2021.
[24] Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling hert for natural language understanding. In Findings of Empirical Methods in Natural Language Processing, 2020.
[25] Zhenzhong Lan, Mingda Chen, Schastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Reprexentatinn, 2020.
[26] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. Preprint arXiv: wtw.ii:6660.04440 007711, 2016.
[27] Mu Li, David G Andersen, Alexander Smola, and Kai Yu. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems, volume 27, pages 19-27, 2014.
[28] Yuhang Li, Xin Dong, Sai Qian Zhang, Haoli Bai, Yuanpeng Chen, and Wei Wang. Rin: Reparameterized ternary network. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4780 07800-477,200 2020.
[29] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu. Qi Zhang, Fengwei Yu, Wei Wang. and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference Learning Representations, 2021.
[30] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34, 2021.
[31] I Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2018.
[32] Paul Michel, Omer Levy, and Graham Neubig, Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, 2019.
[33] Markus Nagel, Rana Ali Amjad. Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post- trstnira quantization. Proceedings of the International Conference on Machine Learning, pages 7197-7206, 2020.
[34] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight cqualization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325 1334, 2019.
[35] Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. Preprint arXiv:1911.07 2019.
[36] Deepak Narayanan, Aaron Harlap. Amar Phanishayee, Vivek Seshadri, Nikhil Devanur. Gregory Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training, In Proceedings of the 27th ACM Symposium Operating Systems Principles, pages 1-15, 2019.
[37] Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. Preprint, 2022.
[38] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xi- anglong Liu, Bibert: Accurate fully binarized bert. International Conference on Learning Representations, 2022.
[39] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. Preprint arXiv:1606.05250. 2016.
12

---

[40] Victor Sanh, Lysandre Debut, Julien Chaumond. and Thomas Wolf. Distilbert, distilled version of bert: smaller. faster, cheaper and lighter. Preprint arXiv:1910.01 108, 2019.
[41] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer, Q-bert: Hessian based ultra low precision quantization of hert. In Proceedings of the AAAI Conference on Artificial Intelligence. 2020.
[42] Siqi Sun. Yu Cheng, Zhe Gan, and Jingjing Liu. Patient kuowledge distillation for bert model compression. In Conference on Empirical Methods in Natural Language Processing, 2019.
[43] Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang. Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. Annual Meeting the Association for Computational Linguistics, 2022.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
2017.
[45] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint aiv: 804.07461, 2018.
[46] Jiaxing Wang, Haoli Bai, Jiaxiang Wu, Xupeng Shi, Junzhou Huang. Irwin King, Michael Lyu, and Jian Cheng. Revisiting parameter sharing for automatic neural channel number search. In Advances in Neural Information Processing Systems, volume 33, 2020.
[47] Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In Proceedings of the International Conference on Machine Learning, pages 9856,9856,220 2020.
[48] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In International Conference on Learning Representations, 2021.
[49] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv:2209. 13325, 2022.
[50] Liangjiang Wen, Xuanyang Zhang, Haoli Bai, and Zenglin Xu. Structured pruning of recurrent neural networks through neuron selection. Newral Networks, pages 134 141, 2020.
[51] Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 12))2700220, 1989.
[52] Ji Xin, Raphael Tang, Jacjun Lec, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. In Anmual Meeting of the Association for Computational Linguistics, 2020.
[53] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zero- quant: Efficient and affordable post-training quantization for large-scale transformers. arXiv:2206.018m 2022.
[54] Ali Hadi Zadeh, Isak Edo. Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention- based nip models for low latency and energy efficient inference. Preprint (eXiv:2005.03842 2020.
[55] Ofr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8hert: Quantized 8bit bert. Preprint 1rXiv:1910.66199 188, 2019.
[56] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-awe -ware ultra-low bit bert. In Conference on Empirical Methods in Natural Language Processing, 2020.
[57] Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. Proceedings of the International Conference on Machine Learning, 2019.
[58] Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Song, Quoe Le, Qiang Liu, and Dale Schuurmans. Go wide, then narrow: Efficient training of deep thin networks. In Proceedings of the International Conference on Machine Learning, pages 11546 11555, 2020.
[59] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, 2020.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In the code document.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] Their license permits public use for research purpose.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] GLUE and SQuAD are widely used benchmark dataset NLPP research.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "eGCOJSJnmP5", "reviewer": "Reviewer_zLnQ", "summary": "The paper presented another way to perform post train quantization (PTQ) on BERT models with a parallel optimization method to speed up the fine-tuning process.", "questions": "As listed under \"Weaknesses\".", "limitations": "This paper provides a decent discussion of the limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The paper offers a detailed ablation study on each of the auxiliary training methodologies they are using on top of the main PTQ algorithm. \n2. The paper provides extensive experimental results covering previous SOTA methods in the field of PTQ and QAT. \n3. the paper provides a detailed explanation of the methodologies and background, helping the understanding of the paper.\n\nWeaknesses: \n1. This work provides very few intuitive/theoretical explanations of why their method could sometimes work as well as QAT methods. In particular, since the paper adapts the well-known quantization function Q(w) = s*round(clip(w/s, MAX, MIN)) (I assume this paper uses the classical form of this quantization function as the author failed to provide explicit details), which is originally a QAT method, it would be ideal that (a) some comparison of loss, accuracy, reconstruction errors, etc. of QAT and/or (b) the STE optimization results over the calibration set is presented. So far, it is hard to see which part of the method is contributing to the most accuracy.\n2. This work seems to be heavily based on _Learned Step Size Quantization_, a well-known work in the field of quantization. It seems to me that the only work this paper made is directly adapting this method to the field of PTQ and showing that it is working well with both limited data (PTQ calibration) and module-wise REM. Due to the lack of ablation study in point 1, the novelty seems to be limited as this paper failed to present how significant their optimization methods could bring benefits to LSQ.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NBk6qL-gZ9", "reviewer": "Reviewer_JA5e", "summary": "The paper proposes an approach for post-training quantization of pre-trained language models. The core of the idea is to run the unquantized and quantized networks in parallel for a small calibration dataset, and update the quantized network parameters such that the quantized network activations match the unquantized reference activations at certain module boundaries in the network. Similar to teacher forcing, the accumulated error in the activations as a result of quantization may be periodically reset/reduced, allowing better training in a regime where total accumulated error would otherwise be too large. Additionally, the model may be spread across multiple GPUs during quantization, and queues may be introduced to allow the GPUs to run asynchronously without waiting for stragglers.\n\nThe method results in quantized model accuracy that is noticeably higher than a baseline for post-training quantization. Compared to quantization-aware training, the proposed method is faster, avoids memory pressure by using less memory, and has better parallelization (though accuracy for the proposed method doesn't quite match QAT).", "questions": "Could the authors please explain when quantization with respect to being before/after/joint-with task-specific fine-tuning?\n\nMy initial assumption would be that PQT occurs after task-specific fine-tuning (though I don't think this is ever stated explicitly in the paper, so I could be wrong). However, Tables 1&2 include a \"full-precision\" row alongside the quantization approaches, and MREM uses less time/memory/data than full-precision fine-tuning. These numbers wouldn't be measuring the same quantity if MREM is run only after full-precision fine-tuning has already occurred. I am further confused by what \"time\" means in the QAT context. With QAT it's technically possible to run the entire task-specific fine-tuning with QAT enabled, or to first fine-tune a bit in the standard full-precision way and then add QAT and optimize the task loss some more. It's again not clear to me what is being done, and what columns like \"time\" actually represent.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A strength of the method is its performance across the four key axes identified in the paper: training time, memory overhead, data requirements, and accuracy. The paper provides a clear description of the base method and how to parallelize it.\n\nIn terms of the weaknesses of the paper, I would appreciate the authors' clarification regarding the Questions section below; as it stands I have some confusion on how to interpret the results.\n\nThe paper also presents REM as the most suitable baseline for PTQ, but the accuracy results (Table 3) instead feature GOBO and no REM. It would be better to have more thorough experiments or exposition instead of this jump. It would also be helpful to have a PQT baseline in Table 3 that runs on more than just two GLUE tasks, because this may be too few to draw good comparisons of MREM accuracy vs. other PQT approaches.\n\nMinor:\n- superscript formatting for 1e-4/5e-5 on lines 206-207 is awkward to read (e^x usually means exp(x))\n- 103: misspelled calibration", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I8XcRed3I3", "reviewer": "Reviewer_LBus", "summary": "This paper presents a method of quantizing the parameters of pre-trained language models by dividing the model parameters into sets (or modules) and quantizing each of them separately via techniques similar to distillation referred to as \"quantization error minimization\". To improve the speed of this process, the authors propose learning parameters of each module in parallel (even though in general the output of one module is the input to the next) by (1) maintaining a cache of old activation values and asynchronously updating their values and using them as desired by each module, (2) using \"teaching-forcing\" by providing the original activation from the unquantized model in the beginning to reduce error propagation.\n\nWith experiments on quantizing BERT-base and BERT-large to different bit-sizes with downstream evaluation on GLUE and SQuAD, the authors show improvements over baselines in terms of training speed and memory required while using less data for quantization and mostly maintaining task performances. \n", "questions": "See weaknesses.", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed solution is simple, where each component--module-based minimization, teaching forcing, and parallel computation--has sound motivation and leads to empirical improvements.\n2. Clear improvements in training time and memory overhead compared to baselines. \n\nWeaknesses:\n1. Choice of #Bits for baselines and proposed methods (In table 3) are not the same, doesn't that make the comparison unfair?\n2. The proposed method still lags behind QAT. Even though they require more data and more training time, this is a one-time investment and if a quantized model however obtained performs better downstream, why would it not be preferable? While data accessibility is a valid concern, other concerns listed in 2.2 are not convincing in favor of using PTQ. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3-8Cny8pgO0", "reviewer": "Reviewer_Z1iD", "summary": "The paper proposes a technique named MREM to quantize a PLM without QAT, which requires huge resources (time, GPU memory, number of samples). Specifically, MREM adopts reconstruction error minimization in a module-wise manner. Each module consists of several Transformer layers, and each module is trained (almost) independent of the others by either taking full-precision output or the actual output from the previous module. To fully utilize such module-wise behavior, the paper also suggests dividing modules into each device. In addition, annealed teacher forcing is also used to simulate the error propagation through layers. The experiments demonstrate the savings (=advantages) of MREM for various resource types. Results of GLUE and SQuAD datasets show that MREM outperforms vanilla REM and reduces the gap between QAT and PTQ.", "questions": "The authors position MREM as a type of PTQ. However, I’m curious if it is okay to view the method that includes parameter update as a variant of PTQ. If this is a widely accepted classification, please ignore this question.\n\nCan you compare the QAT case that only use the same number of (4K) samples? In that case, the resource usage gap will be much decreased, but how about the accuracy gap? Furthermore, if we allow using 4 GPUs to reduce the memory usage, can QAT also use multiple GPUs (reduce the batch size or employ model-parallel) for a fair comparison?\n", "limitations": "The authors addressed the limitations of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow. Especially, the comparison between QAT and PTQ in Section 2.2 provides good motivation for the paper. The experiments are very well organized and support the advantages of the proposed method. Previous works are also sufficiently addressed.\n\nTeacher forcing seems to be a good approach to dividing modules and performing separate optimization for each. The linear annealing schedule is reasonable, and the authors sufficiently support the necessity of the teacher forcing by experiments.\n\nHowever, as the authors pointed out, the key idea that divides the network into several modules and applies reconstruction error minimization is already proposed in BRECQ [30]. There is some novelty in this work, but the improvements are somewhat incremental yet practical. Furthermore, there is no comparison between MREM and BRECQ in the Tables.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept