ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
Authors: Zhewei Yao
Reza Yazdani Aminabadi
Minjia Zhang
Xiaoxia Wu
Conglong Li
Yuxiong He
Venue: NeurIPS 2022 Accept (2022)
Keywords: Post-Training Quantization
Layer-by-Layer Knowledge Distillation
BERT
GPT-3
GPT-Neox-20B

Abstract: How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements.
In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-based models, termed as \OURS. 
\OURS is an end-to-end quantization and inference pipeline with three main components: 
(1) a fine-grained hardware-friendly quantization scheme for both weight and activations; 
(2) a novel affordable layer-by-layer knowledge distillation algorithm (\lwd) even without the original training data access;
(3) a highly-optimized quantization system backend support to remove the quantization/dequantization overhead.
As such, we are able to show that:
(1) \OURS can reduce the precision for weight and activations to INT8 in a cost-free way for both \bert and \gpt-style 
models with minimal accuracy impact, which leads to up to 5.19x/4.16x speedup on \bert/\gpt-style models compared to FP16 inference, separately;
(2) \OURS plus \lwd can affordably quantize the weights in the fully-connected module to INT4 along with INT8 weights in the attention module and INT8 activations, resulting in 3x memory footprint reduction compared to the FP16 model;
(3) \OURS can be directly applied to two of the largest open-sourced language models, including \gptneox, for which our INT8 model achieves similar accuracy as the FP16 model but achieves 5.2x better efficiency.
Our code is open-sourced at~\cite{code_compression}.

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
Zhewei Yao', Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He
Microsoft
(zheweiyao, yazdani rrzaa, minjiaz, xiaoxiamu, conglong.11, yuxhe/@microsoft.ccmm
Abstract
How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due their prohibitive memory/computation requirements. In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-basee models, termed as ZeroQuant. ZeroQuant is an end-to-end quantization and inference pipeline with three main components: (1) a fine-grained hardware-friende quantization scheme for both weight and activations; (2) novel affordable layer-by-layee by-lerrr knowledge distillation algorithm (LKD) even without the access to the original training data; (3) highly-optimized quantization system overhead. As such, reduce the precision for weights and
backend support to remove the
able to show that: (1) ZeroQuant activations to INT8 in a cost-free way for both BERT and GPT-3-style models with minimal accuracy impact, which leads up 5.19 16x speedup those models compared to FP16 inference; (2) ZeroQuant plus LKD affordably quantize the weights the fully fully-connected module INT4 along with INT8 weights in the attention module and INT8 activations, resulting in 3x memory footprint reduction compared to the FP16 model; (3) ZeroQuant can be directly applied two of the largest open-sourced language models, including GPT-JoB and GPT-NeoX for which our INT8 model achieves similar accuracy the FP16 model but achieves up to 5.2x better efficiency.
1 Introduction
Large-scale natural language models have been widely adopted in different applications, e.g., natural language understanding using BERT [64] and generation tasks using GPT-style models [49]. Although those models have achieved cutting-edge accuracy results, as the model size keeps increasing dramatically, the requirements of memory footprint and the computational cost deploy them become a major bottleneck, even on cloud servers with powerful GPU devices.
One promising way to alleviate this challenge is quantization, which can reduce the bit precision for both weight and activations for lower memory footprint and faster compute (e.g.. INT8 Tensor cores on 74/A100), However, quantization usually requires retraining (also known quantization aware training, or QAT short) to recover the accuracy degradation from representation loss of weight and activations. To enable QAT, the full training pipeline is usually required, including the training data and compute resources, to finetune the model. Access to those components is now oftentimes not available, and QAT is also time-consuming process, particularly for those large-scale models.
Recently, zero-shot quantization [10, 47] and post-training quantization (PTQ) [46, 39] are proposed to address the training-dale -dat access and compute requirement challenges since PTQ generally requires
"Code released as a part of http:////ggtt.bb..ccccorrrrrrtre http.////mmoooooooo/////eeeeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

no (or minimal) retraining. But most of those works primarily focus on computer vision problems on relatively small scales. More recently, [7] shows promising PTQ results on BERT. However, (1) its main focus is on high-precision quantization (INT8/FP16) on BERT (2) it does not consider other billion-scale generative models (GPT-3-style models [9]). More importantly, most of these works do not report real latency improvement, putting the usefulness of these methods in improving inference latency into question. For example, existing work often do not discuss the quantiztion/dequanizatooo cost associated with different quantization schemes, which in fact has a big impact to the performance benefit of using low precision.
Besides, for extreme quantization (e.g., INT4), knowledge distillation is usually used to boost perfor- mance, which adds another source of expensive computation cost as compared to QAT. Furthermore, in order to achieve better accuracy performance, hidden-states knowledge distillation, e.g., [3, 81], is usually applied for the quantized model. This would put significant pressure on the GPU memory and the compute resource requirement since both the teacher and student models needed to be loaded into the GPU memory for training.
In this paper, we present ZeroQuant, an end-to-end post-training quantization and inference pipeline, to address those challenges, targeting both INT8 and INT4/INT8 mixed-precision quantization. Specifically, our contributions are:
We apply fine-grained hardware-frieet friendll quantization schemes on both weight and activations, i.e., group-wise quantization for weight and token-wise quantization for activations. Both quantization schemes can significantly reduce the quantization error and retain hardware acceleration properties. We propose a novel layer-by-layer knowledge distillation method (LKD) for INT4/INTS mixed- precision quantization, where the neural network is quantized layer-by-layer through distillation with minimal iterations and even without the access to the original training data. As such, at any given moment, the device memory is primarily populated only with single extra layer's footprint, making billion-scale model distillation feasible with limited training budget and GPU devices. We develop a highly optimized inference backend, which eliminates the expensive computation cost of quantizaiion/dequniizaiii operators, enabling latency speedups on INT8 Tensor cores on modern GPU hardware.
Our empirical results show that:
Zero Zuant enables quantizing BERT and GPT-3-style models into INT8 weight and activations to retain accuracy without incurring any retraining cost. Compared to FP16 inference, our INT8 model achieves up to 5. 19x/4. 16x speedup on BERT /GPT-33300 on A100 GPUs.
ZeroQuant plus LKD do INT4/INT8 mixed- preecisonn quantization for BERT and GPT- 3 3-style models. This results in a 3x memory footprint reduction with marginal accuracy loss compared to the FP16 model. Also, thanks to the lightweight of LKD, can finish the quantization process in 33s (10 minutes) for BERT base (BERTiagge We also demonstrate that
LKD can use other datasets to achieve similar performance the original training data.
We demonstrate the scalability of Zeroouunn on two of the largest open-sourced language models, i.e, GPT-JaB GB and GPT-NeoX 20B. with INT8 quantization. ZeroQuant can achieve 3.67x speedup over the FP16 model for GPT-J&B and (2) reduce the GPU requirement for inference from 2 to I and latency from 65ms to 25ms for GPT-NeoX20B (i.e., 5.2x better system efficiency in total).
2 Related Work
Model compression has been explored from different aspects [26, 38, 40, 35, 44, 21, 25, 19, 76, 41, 27, 56, 60, 29, 61, 69, 34, 15, 39, 32). Among those, quantization is one of the most promising directions as it directly reduces the memory footprint and compute intensity. Here, we focus on quantization for NLP models and briefly discuss the related work.
The majority of quantization works can be categorized into quantization-awan training (QAT). [57, 78] are the first few works to quantize BERT models using integer numbers for both weight and activations. Particularly, [57] utilizes Hessian information to push the weight bit-precision to even INT2/INT4, and it also proposes group- group-wise wise quantization to quantize the weight matrix in a more fine-grained granularity compared to single matrix quantization. [22] introduces quantization noise to alleviate the variations of QAT. [81, 3] leverage very expensive knowledge distillation [27] and data augmentation [29] to ternarize/binarize weights. [30] combines knowledge distillation [29] and learned step size quantization [20] to quantize the weight to 2-8 bits. Recently, [62] also uses

---

knowledge distillation to compress GPT-2 models on task- problems to INT2. All those works quantize models using the original training datasets. More importantly they need retraining or finetuning the full model to recover the accuracy, and such compute cost on extra-large models, like [58, 12], can be hardly affordable for most research labs or practitioners.
One solution to overcome the compute cost challenge is post-training quantization (PTQ). However, PTQ often induces significant drop accuracy because the network can be sensitive to quantization errors. Along this line, one of the first works applied to Transformer-bass The authors introduce centroid- based quantization method, where outlier numbers use FP32 format and the rest numbers are quantized using non-ume unn-niform quantization. As such, it is hard to get the real inference latency benefit on general compute accelerators, e.g., CPU and GPU, because the parallel processing units in these hardware do not support efficient computation of mixed data types. More recently; [7] introduces
[65] models is [77].
activation quantization (FP16) for part of the model to overcome the high dynamic activation ranges. However, to the best of our knowledge, (1) How to apply PTQ on GPT-3-style models while achieving high accuracy has not been studied in any of previous work yet: (2) How to apply PTQ billion (or even dozen of billions) scale model is still under-explored: (3) Efficient inference system backend still missing, especially for fine-grained quantization schemes, making it hard achieve low latency commodity hardware. ZuroQuant resolves all those limitations by considering the system backend into the algorithm design and we verify its capability both BERT and large-scale GPT-3-style (up to 20 billion, i.c., GPT-NeoX2on models for various tasks.
Background and Challenge
3.1 Transformer Architecture
The transformer architecture usually has three components: an embedding layer, a stack of en- coder/decoder layers, and final classifier. In this paper, we focus quantizing the encoder/decoder layers, i.e., the transformer block, because is often the most memory and compute intensive compo- nents in the entire architecture. With transformer block, there are two sub-layers, the multi-head self-attention (MHSA) and the feed-forward connection (FFC). We give a short review later and please refer to [65] for more details. At high level, transformer models can be broadly categorized to three branches: encoder-only models (BERT) [64], decoder-only models (GPT-3-style) [49], and encoder-decoder models (T5) [50]. In this paper, focus encoder-only and decoder-only models but our approach can be applied to encoder-decode models as well.
5oft
Wo
LN
GeLU
W4hhh
Figure 1: The illustration of a Transformer-blos
Transformer Block Assume the input of an encoder layer is X, the query, key, value, attention output, FFC dense, and FFC output matrices are WG. WR. W.. W.. Wn-4h, and W4h-h. respec- tively. Then the forward propagation of transformer-llock is illustrated in Figure 1, where LN is the layer normalization, Softmax is the softmax operator, and GeLU the activation function.
3.2 Quantization Background
Quantization maps high-precision numbers, e.g., FP16/FP32, to its low-precision counterpart, e.g., INT4/INT8, to reduce the model footprint and improve the compute performance. In this work, use uniform symmetric scalar quantizers. That is to say, if we have a vector/matrix. x, the quantization applied as
Xquantize rownd (clampl x - -hit-1
1 ghit - -1)).
(1)

---

Table I: Post training quantization results of GPT-3 on 20 zero-shot evaluation datesets. Here WxAy means X X-/y-bit bi for weight/activation. Particularly. for W4/8, we quantize the MHSA's 's weight to INT8 and FFC's weight INT4. Please see Table 1.1 for the results of all 20 tasks.
Precision
Lambeda
PIQA
OpenBookQA (*)
RIE (t)
ReCoRd (f)
Ave. 19 Tasks
Wikitext-2
WIGAI6
49.3
66.3
29.4
53.8
75.1
38.9
21.5
W8A16
49.3
29.6
38.5
22.1
W1GA8
44.7
64.8
28.2
52.7
69.2
37.8
24.6
W8A8
42.6
28.0
37.8
26.2
W4/8A16
0.00
51.4
30.2
52.7
16.1
28.9
1.76c5
where bil is the number of bit we use to represent the quantized value, and S is the scaling factor. For weight matrix quantization,
generally computed as S max (abs(x)): since the weight matrix static during inference. On the other hand, activations' range dynamic during inference that an accurate S requires dynamic calculation during inference. However, to achieve best latency reduction, coarse-grained static quantization is usually applied in practice, where S is calibrated using training data (e.g., momentum based averaging) and fixed during inference [24]. Although static quantization achieves better latency reduction, it also limits the quantization representation for activations, which discussed below.
3.3 Post Training Quantization
Post-training quantization (PTQ) exhibits great compression efficiency compared quantization- aware training (QAT) since PTQ is usually applied to quantize the model without retraining. A common strategy of PTQ is to feed the training data to the network and calibrate the scaling factor, S, using the running mean. Please see Appendix A.1 for more details.
Some work has been done for BERT base models [7] with INT8 weight and mixed INT8/FP16 activation quantization. However, there is no investigation for (1) even lower bit- bit-precision PTQ on BERT models and (2) large-scale GPT-3-style models. Here, we briefly diseuss the challenge of the application of PTQ on both BERT (in Appendix C) and GPT---sttle models.
Activation Range of Each Thken for Different Layers
of Each Different Matrices
50-
pax alo value Esch Tokee
40-
Mat Ros Vabur bach
30
0.4-
20
30-
1011122134410
10 Layer Number
16 16
17
20
23
Layer Number
Figure 2: The activation range (left) and row-wise weight range of the attention output matrix (right) of different layers on the pretrained GPT-3350M- See Figure C. for the results BERT base-
The results of GPT-315IM with PTQ are shown in Table 1. As can be seen, the INT8 activation quantization (i.e., the row of W16A8 causes the primary accuracy loss. Further pushing the weight to INT8 (i.c., the row of W8AS) does not change the accuracy of zero-shot evaluation tasks but leads the causal language modeling task (Wikitext-2) to worse perplexity score, which demonstrates the sensitivity of generation tasks as compared to other zero-shot evaluation problems. For W4/8A16 16, on some accuracy-baut based tasks, GPT-33suM still achieves reasonable performance like OpenBookQA but it loses accuracy on the majority of the rest tasks. Particularly, for Wikitext-2, GPT-3350M with W4/8A16 cannot generate any meaningful text anymore. Please also see Appendix C for the analysis for BERT.
Dynamic Activation Range To investigate why INT8 activation leads to significant accuracy drop for both BERT and GPT-3-style models, we plot the token-wise (i.e., the hidden state of each token) range of each activation for different transformer layers of GPT-3350M in Figure 2 (left). As can

---

be seen, different tokens have dramatically different activation ranges, For example, the maximum range of the last layer is around 35 but the minimum range is close to 8. This larger variance in the activation range makes it difficult to use a fixed quantization range (usually the maximum value) for all tokens to retain the prediction accuracy, because the limited representation power for small range tokens is going to hurt the accuracy performance.
Different Ranges of Neurons in Weight Matrices Similarly, we plot the row-wise (i.e., the output dimension) weight range of the attention output matrix (W,) of GPT-3350M in Figure 2 (right). There 10x difference between the largest magnitudes of different rows and this leads to the worse generation performance of the INT8 weight PTQ. This also makes it very challenging when INT4 quantization applied as the INT4 only has 16 numbers and a 10x smaller range leads (or 3) numbers for the representations of those smaller-range rows.
This analysis results also indicate why more expensive hidden-states knowledge distillation [3, 37] is used for ultra-low precision quantization to close the aceuracy gap. However, as the training cost of knowledge distillation for large-scale models too high, a lightweight and efficient method is desirable for PTQ.
Methodology
4.1 Fine-grained Hardware-friendly Quantization Scheme
As shown in Section 3, even applying INT8 PTQ to BERT/GPT-3-style models leads to significant accuracy degradation. The key challenge is the representation of INT8 cannot fully capture the different numerical ranges of different rows in weight matrices and different activation tokens. One way to address this is to use woiue-wise (token-wise) quantization for the weight matrix (activations).
Group-wist Quantization for Weights Group-wise weight matrix quantization has first been proposed in [57], where a weight matrix W E Raxm is partitioned in to g groups, and each group is quantized separately. However, in [57], the authors only apply this for quantization aware training. More importantly, they do not consider the hardware efficiency constraint and they do not have a system backend support. As such, they lack the real latency reduction benefit.
In our design, we consider the hardware constraint from Ampere Architecture of GPUs (c.g. A100), where the compute unit is based on Warp Matrix Multiply and Accumulate (WMMA) tiling size [54] to achieve the best speedup. Later, we will show that our group-wing wise quantization leads to much better accuracy compared to single-matrix quantization due to its finer- mmer-granularity quantization while still achieving great latency reduction.
Token-wise Quantization for Activations As mentioned in Section 3 and Appendix 3.2, a common practice for existing PTQ work is use static quantization for activation. where the min/max range is calculated at an offline calibration phase. Such a method might be sufficient for small scale models where the variance in the activation range small. However, as analyzed in Section 3, there is a huge variance in the activation range for large-scale transformer models such as GPT-3150M and BERT buse. As such, a static quantization scheme (often applied to all tokens/samples) would lead to significant accuracy drop. One natural idea to overcome this issue is to adopt fnne-grained token-wise quantization and dynamically calculate the min/max range for each token reduce the quantization error from activations. Our evaluation in Section 5 also shows that token-wise quantization for activation significantly improves the accuracy of GPT-3-style and BERT models.
However, directly applying token-wise quantization using existing DL frameworks, such as the Py PyTorch quantization suite, would lead to sigmificant quantization and dequantization cost because token-wise quantization introduces additional operations that lead to expensive data movement overhead between the GPU compute units and the main memory. To address this issue, we build a highly optimized inference backend for token-wise wise quantization of transformer models. For example, the inference backend of ZeroQuant employs so called kernel fusion technique to fuse quantization operator with its previous operator, like layer normalization, to alleviate the data movement cost from token-wise quantization. Similarly, the dequantization cost of the different GeMMs' output is alleviated by scaling the INT32 accumulation using both the weight and activation quantization scales, before writing the final FP16 result back to the main memory for the next. FP16 operator (like GeLU). Those optimization will be discussed in more details in Section 4.3.
5

---

LN/GeLU
Quantize
GeMM
Dequantize
LN/GeLU
GeMM Dequante
Quantize
Figure 3: The illustration of normal (left) and fused (right) INT8 GeMM.
Token-wise quantization can significantly reduce the representation error for quantized activations. Also, as it does not need to calibrate the activation range, later we will show that there is no quantization-ellted cost (e.g.. activation range calibration) for a moderate quantization scheme (INT8 weight with INT8 activation) for ZeroQuant.
4.2 Layer-by-layer Knowledge Distillation with Affordable Cost
Knowledge distillation (KD) is one of the most powerful methods to alleviate the accuracy degradation after model compression. However, there several limitations of KD, especially for hidden-states KD on large-scale language models: (1) KD needs to hold a teacher and a student model together dur- ing the training, which dramatically increases the memory and compute cost; (2) KD usually requires full training of the student model. Therefore, several copies (gradient, first/second order momentum) of the weight parameters need be stored in memory update the model: (3) KD generally requires original training data, which sometimes are not accessible due to privacy/confidential issues.
To address those limitations, present the target model for quantization has N transformer blocks, L1, LN the accessible dataset has input (X, Y), which can be the original training data datasets from other resources. Our LKD quantizes the network layer-by-layer and uses its original (i.e., unquantized) version as the teacher model. More specifically, assume layer Ly is going to be quantized, and its quantized version is LR- Then we use the output of the L (i.e., by running inference on X over the first i layers) as the input of Lk and Lk. measure the difference, and do the model update to Lks ie..
layer-by-layer distillation (LKD) algorithm. Assume
LIKDD MSE LE-1 L (x Le LE-1 L ... Lx(x).
where MSE is the mean square loss, and can be also replaced by other losses (e.g., g., KL divergence) the same I to L for both teacher/student model. As such, the only extra model cost we have is T&; (2) the memory overhead of optimizer states significantly reduced the only optimizing layer is T&G (3) as we never optimize the end-to-end model, the training does not depend on the label anymore. Later, we will show that LKD does not rely on the original training data Section 5.6.
well. As be seen, (1) LKD does not need to hold separate teacher
4.3 Quantization-Optiniii Transformer Kernels
Both optimizing the inference latency and model size is crucial for serving large-scale transformer models in practice. During inference, the batch size is often relatively small, so the inference latency of the model primarily depends the time of loading inference needed data from the main memory. By quantizing the weights and activations to lower precision, we reduce the data volume needed to load those data, which allows more effective of memory bandwidth and higher loading throughput. However, simply converting weights/acttvattions to INT8 does not guarantee improved latency because there additional data movement overhead associated with quantizationddequantzztiooo operations as shown in Figure 3 (red box). Such an overhead becomes expensive and in some cases surpasses the performance benefits of using low precision. To reap the accuracy improvement from token-wise quantization while obtaining improved latency, we now present our optimizations that maximize the memory bandwidth utilization speed up inference latency for ZeroQuant.
CUTLASS INT8 GeMM To support INT8 computation, we use CUTLASS [6] INT8 GeMM implementation tuned for different batch sizes. Unlike standard GPU backend library, such as cuDNN, using CUTLASS allows us to more flexibly fuse quantization operation before and after GeMM to reduce kernel launching and data-movement overhead.
Fusing Token-wise Activation Quantization Token-wise quaniizationddequatizzaooo introduce many additional operations that lead to extra data movement cost. To eliminate these cost, use

---

Table 2: Result of BERT hase on the development set of GLUE benchmark (except WNLD). [57]- uses 128 groups for weight matrix which is hard to get GPU acceleration. [7]" uses mixed INT8 and FP16 activation, and it directly reports the average metric of MNLIMMPC/OOP/STSS B, which is basically the average of the two metrics we used for our runs.
Precusion (Michhod)
COLA
M
mM
MRPC
QNLI
QQP
RTE
5ST-2
STS-B
Ave.
Ave. Time (s]
W16A16 (Baselinc)
59.72
84.94
85.06
86 27/90.57
92.15
91,51/88.50 56
72.20
93.23
90.06489.5 59
83.05
N/A
WHAS 157] (QAI)"
83.91
83.83
52.83
WXAR [78] (QAT)
58.48
189.56
90.62
187.96
68.78
92.33 02.24
89 04/-
WBAS (QAT)
61.21
64.80
$1.64
83.82/88.85 85
91.29
91,29/88,20 28
71.12
92.89
86,391SS 18
83.37
2900
WXAS (PTQ)
56.36
79.99
49.49779.67
87.35
89.912966888
40.38
86.58/86.44
77.41
[7] (PTQ))
$8.63
82:67
82.67
88.74
68.95
88,00
82:46
Unknown
(ZeroQuant)
59.59
84.83
$5.13
36.03/90.39
91.95
91.45/58.46
71.12
93.12
90.09/99.62
83.75
W4RRA (PTQ)
0.00
16.74
16.95
63/0000
50.74
181 00
47.29
16,48/15.91 91
33.11
6
W. (ZeroQuant)
$7.29
82.69
$3.27
$4.56/88.40
90.04
86.52/79.49
70.76
92.78
88,06758.60
81.65
0
W4 (ZanQQumt-LLD))
58.50
83.16
83.69
84.80999.31 51
90.83
88.94/94413 12
70.04
92.78
88.49088.67
82.35
31
W/URAB (ZenQQuant)
$6.89
82.46
$3.06
$1.07/88.03
90.13
87,04/80.50
70.76
92.78
68.07758. 414
81.55
0
W4/KAB (ZenoQuamtLLD))
58.80
83.09
83.65
85.78/99.90
90.76
89.16/84.8
71.84
93.00
88.16/88.55
82.71
31
kernel fusion [68] to fuse quantization operation for activation with its previous element-wise and/or reduction operations such as bias-add, GeLU, and LayerNorm into single operator, as illustrated by the green box in Figure
For the dequantization operation (e.g., dequantizing the integer output from the GeMM operator), we similarly fuse it with our custom GeMM schedule to avoid additional read/write accesses to the main memory as illustrated by the blue box in Figure 3.
By doing the above optimizations, we are able to show significant latency reduction for BERT and GPT-3-style models in Section 5. Please see Appendix D for more details about our system optimization.
5 Results
Experimental Details To evaluate the proposed ZeroQuant, we test it on both BERT and GPT-3 models. For BERT, we tested both BERT and BERT large on GLUE benchmark; and for GPT-3-style models, we tested the GPT-33s0M (i.e., GPT-3-style model with 350M parameters) and GPT-31.B (i.e., GPT-3-style model with 1.3B parameters) on 20 zero-shot evaluation tasks, including 19 accuracy- based tasks and language modeling generation task. To illustrate the scalability of the proposed ZeroQuant, we also directly apply it to two of the largest open-sourced GPT-3-style models, i.c., GPT-J6B [67] and GPT-NeoX20B [5]. We use fixed set of hyperparameters for all the LKD-related experiments even though tuning them may benefit our results. Please see Appendix A.2 for more training details and see Appendix A.3 for the reported metrics for BERT. To provide a comprehensive study, we also include a tuning result in Appendix on BERT and an ablation study for different proposed components in Section 5.5.
Notation Explanation We use WxAy to represent using x-bit for weight quantization and y-bit for activation quantization. Unless specific explanation, for W4/8, we quantize the MHSA's 's weight to INT8 and FFC's weight to INT4; for A8/16, we use FP16 activation for self-attention calculation (i.e., the GeMM related to W wa/k/u) and use INT8 for the rest calculation. We use ZeroQuant to represent the method with only fine-grained quantization schemes and use ZeroQuant-LKD to represent the method with both fine-grained quantization schemes and LKD.
summary of results is shown in Appendix B. We also include (1) the results required by reviewers in Appendix G; (2) the speedup explanation.
5.1 Main Results of BERT
BERT base We report the results of BERT base in Table 2. For W8A8, the average accuracy of PTQ degrades more than 10 points. However, ZeroQuant can achieve 83.75 scores, which is only 0.2 lower than baseline. Particularly, as ZeroQuant has no activation range calibration phase, the cost of ZeroQuant is 0 which is even cheaper than standard PTQ. As compared to [7], our method achieves a better average score (1.29 higher). Meanwhile, compared to INT8 activation used in ZeroQuant, [7] uses mixed INTS and FP16 activation.
We also compare our method with our internal trained QAT and other QAT works [57, 78). As can be seen, with comparable accuracy results as those QAT methods, ZeroQuant can save the retraining cost from 2900s to Os for INT8 quantization.
7

---

Table 3: Result of BERT large on the development set of GLUE benchmark (except WNLD. + +We extensively tuned the learning rate for QAT (see Appendix F for more details).
Precisioon (Method)
OoLA
MNL.-m
MNL.J-mm
MRPC
QNLI
QQP
RTE
55T-2
STS-B
Ave.
Ave. Time (6)
WIGA16 (Baseline)
63.35
86.45
35.91
87.99/91,02
92,24
91.08/55.08
74.01
93.46
90.3450.11
85,00
N/A
WBAS [78] (QAT)
-90.9
91.74
90.12/-
WHAS (QAA))
59.85
86.65
36.25
85. 43.29/29.43
92.55
91.6VSS..C 6D)
61.37
93.23
87 87.55/57.06 05
82.75
7181
WXAS (PTQ)
60.57
75.69
76.94
51.1388.95
88.40
84,04/74.5 35
46.93
91.74
62.75555.77 77
73.54
31
W3AS (ZeroQuant)
63.38
86.52
85.64
87.75/91.50 .5
91,05/58.05
72.56
90,4550.11
84.81
W4IAA
0:00
16.85
33.24
11.12/00.89
51.25
63.1820
52.71
-5.7418851
35.75
W6 (ZenoQuami
17.5091.16
90,03/8641
18.01
16.1
W4RAAT6 (ZenQQum-LLD)
63.72
84,90
84.81
87.99/91.39
91,45
90.34/86
51.62
92.44
39.46/89.22
81.85
550
WARRAS (ZeroQuant) 62.34
84.62
$4.25
87. 75/79/1.5 38
91,87
89 19.86/86.00
47.65
91.97
39/99/99.1
17
81.06
0
WA/NAS (ZroQQant-LLL)
63.51
84,70
34.71
88.7399999
91.73
90.25/66.74
49.82
92.09
89 19,34/59908
81.02
350
Table 4: Post training quantization result of 3PT-3350M on 20 zero-shot evaluation datasets. Please see Table I.1 for the results of all 20 tasks.
Precision (Method)
Lambada (f)
PIQA (1)
OpenBookQA (t) RTE(F)
ReCokd (?)
Ave. 19 Tasks (?)
Wikinest-2 (.)
Time Cost
W 16A16
49.3
66.3
29.4
53.8
75.1
38.9
21.5
NiA
(PTQ)
64.1
28.0
53.1
37.8
2622
WSA8 (ZeroQuant)
51.0
66.5
29.2
53.4
74,9
38.7
21.7
0
14
0.00
51.4
50.2
$2.7
16.1
28.9
1.76c5
7 mins
W4/8A 16 (ZanQuant)
10.1
58.5
27.2
52.0
56.5
33.5
88.6
0
W4/SA (2040////00
39.8
63.8
29,4
53.1
70.1
37.0
30.6
hours
W4/SA8 (ZenQuant)
57.7
28.0
52.7
33.4
W478A8 (ZeoQuess-LLD)
37.4
61.8
28.2
53.1
68.5
36.6
31.1
hours
For the more aggressive weight quantization with minimal (or no) training quantization, i.c., 14/8A16, PTQ fully loses all accuracy (pure random prediction). However, ZeroQuant can still achieve an 81.65 average score. On top of ZeroQuant, if we add our LKD, the accuracy can be further boosted to 82.35 with a cost of 31s per task using only single GPU, which is 93.5x cheaper than INT8 QAT quantization. We also test ZeroQuant and ZeroQuant- under the W4/8A8 quantization scheme and both of them achieve similar accuracy performance as W4/8A16. If hyper-parameter tuning is applied to LKD, ZeroQuant-LKD can achieve an 83.22 average score under W4/8A8, which is similar to QAT's W8A8 result. Please see Appendix for more details.
BERT large We test our methods on BERT large as well and the results are shown in Table 3. Similar to BERT base. ZeroQuant achieves much better accuracy than PTQ methods. As compared to QAT methods, ZeroQuant has comparable results on larger datasets (like MNLI/QQP) and has better performance on small tasks (e.e., COLA/MRP//RTE). We actually tune QAT for multiple learning rates but cannot get even better performance for those small tasks (see Appendix F for more details).
For more aggressive quantization schemes, like W4/8A16 and W4/8A8, ZeroQuant and ZeroQuant- LKD still achieve good accuracy except for RTE but the model size is about 3x smaller than FP16 counterpart. This is aligned with the INT8 QAT results, which lose significantly more accuracy on RTF. Thanks to the lightweight cost of LKD, it only takes about 550s to finish each task even on BERT Iarge> which is 13x cheaper than QAT.
5.2 Main Results of GPT-3-style Models
GPT-3350M We first test ZuroQuant and ZeroQuant-LLD on GPT-3350M and report the result in Table 4. The first interesting finding of zero-shot evaluation on GPT-3-stype models is that the accuracy performance of accuracy-base
tasks is more tolerant to quantization than generation tasks. For instance, W8A8 PTQ has 1.1% average accuracy drop on 19 accuracy- tasks as compared to 4.7 points loss on Wikitext-2. Comparing ZeroQuant with PTQ using W8A8, we can reduce the accuracy gap from 1. to 0.2% and the perplexity (PPL.) gap from 4.7 to 0.2 with no activation range calibration cost.
For W4/8A16 quantization scheme, PTQ can hardly predict reasonable answers for the majority of tasks and its generation performance on Wikitext-2 fully crashed. As comparison, ZeroQuant still achieves non- -rivial performance on some tasks but its generation performance significantly degrades on Wikitext-2. LKD brings a significant performance boost for this W4/8A16 16 setting. Note that ZeroQuant- increases the accuracy from 33.5 to 37.0 and decreases the PPL from 88.6 to 30.6 compared to ZeroQuant, and the entire cost of this is just 3.1 hours on a single A100 GPU. Note that this is about 0.027% GPU hours of the full pretraining cost (128 100 GPUs for 32 hours). Similar to W4/8A16, 16, ZeroQuant-LKKD achieves much better performance than ZeroQuant on W4/8A8 by using the lightweight LKD.
8

---

Table 5: Post training quantization result of GPT-3133 on 20 zero-shot evaluation datasets. Please see Table 1.2 for the results of all 20 tasks.
Procision (Method)
Lambuda (T)
PIQA (1)
Open (f) RTE(T)
ReCoRd (1)
Ave. 19 Tinks (T)
Wikitext-2 ())
Time
W16A16
61.3
71.4
33.6
53.1
K2.6
42.4
15.3
NIA
WSAS (PTQ)
5488
67.7
16.6
54.5
75.7
40.5
18.9
13 mins
W3A8 (ZeroQunnt)
62.6
20.7
33,4
52.7
80.9
42.3
15.7
0
(PTQ)
27.0
50.9
W4/8A16 (ZeroQuant)
43.9
66.5
30.0
52.7
77.3
39.38
21.9
16 Lun-QualLL
59.4
69.5
31,6
52.7
79.7
41.5
17.6
bours
W4ISAS (ZenoQuant)
46.8
66.4
28.8
52.7
76.2
39.24
24.1
0
W4/SAS (ZeroQuass
68.1
29.0
52.0
77.4
39,90
18.2
bours
Table 6: The speedup of our W8A8 as compared to W16A16. We measure the end-to-end average latency for the entire BERT model, and the time reported is in milliseconds.
Seg Len
128
256
BS
Precisioo
128
2,45
3.22
5.51
67.08
5.70
19.27
71.25
140.0
BERT
WBAS
1.08
1.16
1.42
1.76
2.58
3.90
6.74
12.92
1.22
1.44
2.08
2.88
4.10
7.80
14.66
28.13
Speedup
2.27
2.78
2.71
3.13
3.86
4.60
5.08
5.19
2.57
2.81
2.74
3.66
4.70
4.70
4.89
4.98
W16A16
5.45
6.38
8.73
13.88
26.34
48.59
92.49
183.4
6.39
8.94
14.66
27.99
51.94
98.78
195.9
384.5
RERT
WBAS
2,08
3.79
2.55
4.16
41.24
79.90
Speedup
2.62
2.47
3,66
4.24
5.01
3.52
4.75
1.81
L33 The results of GPT-3 are shown in Table Similar GPT-3 for W8A8, Zero- Quant has much better performance than PTQ with less no activation calibration cost, particularly for the generation task Wikitext-2 (3.2 points lower). Also, for W4/8 quantization, LKD can bring non-trivial performance gain for ZeroQuant. The cost of LKD is about 0.02% of the full pre-training cost (128 A100 GPUs for 120 hours)
5.3 Latency Reduction of BERT and GPT-3-style Models
We compare the inference speed of BERT between FP16 and our INT8 versions in Table 6 on a single 40G-A100 GPU. Using efficient quantization kernel implementation and operator fusion, the INT8 model can achieve 2.27-5. 1.xx speedup on BERT hase and 2.47-5.01x on BERT large.
We also include the latency comparison of GPT-3-style models between FP16 and our INT8 version. Particularly, we use the model to generate the first 50 tokens based on a given text and measure the average latency. Our INT8 model leads to 4.16x/4.06x speedup for GPT-33xm//PPT-3, as compared to the FP16 counterpart.
5.4 A Showcase of GPT-J&B and GPT-NeoX208
To demonstrate the scalability of ZeroQuant, we applied it to two of the largest open-sourced models, i.e. GPT-J&B and GPT-NeoX20B. which have 6B and 20B parameters separately.
We report the results of GPT-J&u in Table on three generation datasets, i.e., PTB [42], Wikitext-2, and Wikitext-103 103 [43]. As can be seen, as compared to FP16 precision, ZeroQuant achieves similar PPL on all three different tasks. To compare the latency, we again use the average latency number to generate the first 50 tokens. Our W8A8 can get up 3.67x speedup compared the FP16 version.
To quantize GeT--NeXX20 to W8A8 for all GeMMs, the accuracy significantly decreases. We retrieve the quantization of each weight matrix and of each activation, and finally find out that the activation quantization for the attention calculation (i.c., the input of self-attention) causes the accuracy loss. We conjecture that this is because of the sensitivity of the self-attention module for extra-large models (20B) but cannot verify this for other models due to the lack of sourced extra-large models and the full evaluation pipeline. As such, quantize the rest to INT8. The results are shown in Table 8. Our W8A8/16 achieves similar accuracy performance but can reduce both the GPU resource requirement (from 2 A100 GPUs to 1) and the latency from 65ms to 25ms, which together lead to 5.2x better throughput/efficiennyy.
leave the input activation for self-attention in FP16 6 and
5.5 Ablation Study of Different Components
To investigate the performance gain of each component we introduced in Section 4, i.e., group- weight quantization, token- wise activation quantization, and lightweight layer-by-layer knowledge distillation, we here do an ablation study on BERT largo with W4/8A8.

---

Table 7: Post training quantization result of GPT-
Table 8: Post training quantization result of GPT-
Jon on three zero-shot generation tasks
Ne0X on 19 zero-shot evaluation datasets.
Precision PTB Wikitext-2 Wikitext- 103
Latency
Please see Table 1.4 for the results of all 19 tasks.
W16A16 20.47
10.35
10.35
29.13ms (1x)
Precision Lambada PIQA Ave. 19 Tasks
Latency
W8A8 20.97
10.51
10.52
7.94ms (3.67x)
W16A16 W8A8/16
71.7
77.7
50.5
2x65ms (Ix)
71.9
78.3
50.4
1x25ms (5.2x)
Table 9: Ablation study of different components for BERT large on the development set of GLUE.
The quantization scheme used here is W4/8A8. Here, GQ is the abbreviation of group-wise weight quantization, TQ is the abbreviation of token-wise activation quantization.
GQ
TQ
LKD
CoLA
MNLI-m
MNLI-mm
MRPC
QNLI
QQP
KTE
SST-2
STS-B
Ave.
-0.79
33,07
32.94
68.38/80.54
49.42
63.18/0.00
52.71
52.29
-4,27/-1.90
35.85
x
59.81
66.63
68.79
68.63/71.17
83.87
78.24/61.2 10
46.93
89.45
54.58/32.52
66.52
7
62.34
84.62
84.25
87.75/91.38
91.87
89 89.86/86.09
47.65
91.97
89. $9.39/89.17
81.06
63.51
84.70
84.71
88.73/91.99
91.73
90.25/86.74
49.82
92.09
89.34789.08
81.62
Table 10: Post training quantization result of GPT-33s0M 20 zero-shot evaluation datesets The quantization scheme here is W4/8A8. Please see Table 1.3 for the results of all 20 tasks.
Method
Data Resource
Lambada (?)
PIQA (1)
OpenBookQQ (?)
RTE (1)
ReCoRd (1)
Ave. Tasks (1)
Wikitext-2 (J)
ZeroQuant
10.5
$7.7
28.0
52.7
55.3
33.4
92.1
ZeroQuant- LKD
Random data
26.1
59.3
29.2
50.5
64,9
34.5
40.6
ZeroQuant-IKD
Wikipedia
33.9
62.4
28.0
52.7
69.5
36.2
30.4
ZorQQuantLLD
Original data
37.4
61.8
28.2
53.1
68.5
36.6
31,1
We present the results in Table 9. As can be seen, group-wise weight quantization boosts the accuracy (random-guess prediction) from PTQ non-trivial result (66.52). Further adding token-wise quantization improves 14.54 points accuracy performance. On top of those (i.e., ZeroQuant), LKD further brings 0.56 point gain.
5.6 No Access to The Original Training Data
As mentioned in previous sections, the original training data are oftentimes hard to access due to the privacy and/or confidential issues. Therefore, we here study the performance of our L.D when there is no direct access to the original training data. As the distillation objective of our LKD does not depend on the label, the training data used for LKD can be very flexible.
We compare the performance of GPT-3350M on W4/8A8 quantization scheme using three different training data resources, i.e., random data (using random integer number to generate token ids), Wikipedia (using Huggingface to get the data²), and original PILE dataset.
The results shown in Table 10. Compared to ZeroQuant, LKD using random data boost the accuracy by 1.1% and reduce the PPL from 92.1 to 40.6. The reason why random data can still significantly improve the performance is that L.DD does not optimize the end-to-end pipeline and it only layer-by-layer learns the internal dependency from the teacher model. Therefore, random data can also provide meaningful information. Using Wikipedia data from Huggingface can further improve the accuracy to 36.2 and reduce the PPL to 30.4, which is comparable to the results using the original data. This indicates that clean text dataset can be used for LKD when we do not have access to the original full dataset.
6 Conclusions
With the rapid growth of large model sizes, we have reach a point to consider how to serve those models practice. Although several works demonstrate that post-training quantization can be applied to BERT models, to the best of our knowledge, there have been no existing works on (1) billion-scale GPT-3-style models, (2) ultra-low precision post-training quantization, and (3) end-to-end solution of how to efficiently serve the quantized model online. In this work, we offer fine-grained compression schemes for both weight and activations to enable INT8 quantization for up to 20B-scale models (GPT-NeoX2og). We also offer novel affordable by-areb knowledge distillation for ultra-low precision quantization, which leads to 3x model size reduction compared to FP16 model while achieving minimal accuracy degradation. Furthermore, provide system backend support and show up to 5.19x speedup on BERT models and 5.2x better efficiency on GPT-NeeX2o0-
10

---

References
[1] Ardavan Afshar, loakeim Perros, Evangelos E Papalexakis. Elizabeth Searles, Joyce Ho, and Jimeng Sun. Copa: Constrained parafac2 for sparse & large datasets. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 793-802, 2018.
[2] Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, et al. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale. arXiv preprint arXiv:2207.0003: 2022.
[3] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020.
[4] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
[5] Sid Black, Stella Biderman, Alex Andonian, Quentin Anthony, Preetham Gali, Leo Gao, Eric Hallahan, Josh Jevy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Jason Phang, Michael Pieler, Shivanshu Purohit, Tri Songz, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large scale autoregressive language modeling in pytorch, 2021.
[6] NVIDIA blog. CUTLASS: Fast Linear Algebra CUDA C++. ia. com/blog/ cwtcccsssssaeeeec letes-linear-aaaa alggerr---ggebb---uuuaau December 2017.
.nvid nvid
[7] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109. 12948, 2021.
[8] Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj. Rajarshi Das, An- drew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the are dataset. arXiv preprint arXiv: 1 00358, 2018
[9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[10] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178, 2020.
[11] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpic and Lucia Specia. Semeval-2017 task I: Semantic textual momilaartt-multliinn and cross-lingual focused evaluation. arXiv preprint arXiv: 00055, 2017.
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[13] Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: /905. 10044, 2019.
[14] Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1-220, 2013.
[15] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Fukasz Kaiser. Uni- versal transformers. arXiv preprint arXiv:1807.038812 2018.

---

[16] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- -re-training of deep bidirectional transformers for language understanding. arXiv preprint 1xXiv:8810.04055. 2018.
[17] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint wrXiv:2002.06305. 2020.
[18] William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential para- phrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
[19] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. HAWQ: Hessian aware quantization of neural networks with mixed preccisoon In Proceedings of the IEEE International Conference on Computer Vision, pages 293-302, 2019.
[20] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. arXiv preprint arXiv: arXiv: 1902.08/53, 2019.
[21] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv: 1556, 2019.
[22] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme fixed-point compression. arXiv preprint arXiv:2004.07326, 2020.
[23] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
[24] Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 13630, 2021.
[25] Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307, 2020.
[26] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems. pages 1135- 2015.
[27] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Workshop paper in NIPS, 2014.
[28] Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Question (201), URL https://aaaa. quora. com/First- Saaiiiss--ooor--ttttt----eeeeeee Peueseeeeeeeeeeee-euuuuennniiir 2017.
[29] Xiaoqi Jiao Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10351, 2019.
[30] Jing Jin, Cai Liang, Tiancheng Wu, Liqin Zou, and Zhiliang Gan. Kelsq-bert: A quantized bert combining knowledge distillation with learned step size quantization. arXiv preprint arXiv:2101.05938 2021.
[31] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint drXiv:1705.03551, 2017.
[32] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael w Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference on machine learning, pages 5506- 5518. PMLR, 2021.
[33] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv: 2017.
12

---

[34] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for sulf-supervised learning of language representations. arXiv preprini arXiv: (909: 1942, 2019.
[35] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[36] Hector Levesque, Emnsst Davis, and Leora Morgenstem. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012.
[37] Fengfu Li, Bo Zhang, and Bin Liu, Ternary weight networks. arXiv preprint 1eXXvv1605.0477 2016.
[38] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 100.::00000000 2016.
[39] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34, 2021.
[40] Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William Dally. Exploring the regularity of sparse structure in convolutional neural networks. Workshop paper in CVPR, 2017.
[41] Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, and Jing Bai. Ladabert: Lightweight adaptation bert through hybrid model compression. arXiv preprint arXiv:2004.04124, 2020.
[42] Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. Using Large Corpora, page 273, 1994.
[43] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[44] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? arXiv preprint arXiv: 1905. 10650, 2019.
[45] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:1809.02777 2018.
[46] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[47] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[48] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint
2016.
[49] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[50] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
[51] Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. Fixed encoder self-attention patterns in transformer-basee machine translation. arXiv preprint 1026: 2020.
13

---

[52] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. 2016.
[53] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505 3506, 2020.
[54]
Greg Ruetsch. Using tensor cores in cuda fortran. Nvidia Blog, 2021.
[55] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732-8788 8740, 2020.
[56] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.000 108, 2019.
[57] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of bert. In AAAI, pages 8815-88211 2020.
[58] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari. Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron- -tring nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 2022.
[59] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1631-1642, 2013.
[60] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: www.:/9800.99 09.55500055,, 2019.
[61] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for lssturee-lmited devices. arXiv preprint arXiv:2004.0298/ 2020.
[62] Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong, Compression of generative pre-trained language models via quantization. arXiv preprint arXiv:2203. 10705, 2022.
[63] Sandeep Tata and Jignesh M Patel. Piqa: An algebra for querying protein data sets. In 15th International Conference on Scientific and Statistical Database Management, 2003., pages 141-150. IEFE, 2003.
[64] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nip pipeline. 1rXiv:1905.05950, 2019.
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1998-6008 2017.
[66] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rX 2018.
[67] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language com
Model. https:
transforment janeeer-jaaa,aaaa May 2021.
[68] Guibin Wang, Yisong Lin, and Wei Yi. Kernel fusion: An effective method for better power efficiency on multithreaded GPU. In Peidong Zhu, Lizhe Wang, Feng Xia, Huajun Chen, Ian MoLoughlin, Shiao-Li Tsao, Mitsuhisa Sato, Sun-Ki Chai, and Irwin King, editors, 2010 IEEF/ACM Ini'l Conference on Green Computing and Communications, GreenCom 2010, & Int'l Conference on Cyber, Physical and Social Computing, CPSCom 2010, Hangzhou, China. December 18-20, 2010, pages 344-350. IEEE Computer Society, 2010.
14

---

[69] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprini arXiv:2002. 10957, 2020.
[70] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv::805, 12471, 2018.
[71] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lunguage Technologies, Volume (Long Pupers), pages 1112-1122, 2018.
[72] Adina Williams, Tristan Thrush, and Douwe Kiela. Anlizing the adversarial natural language inference dataset. arXiv preprint arXiv:2010. 12729, 2020.
[73] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. HuggingFace'ss Trans- formers:
State-of-the-art natural language processing. ArXiv, pages arXiv-1910, 2019.
[74] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint wrXiv:2004.09602 2020.
[75] Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsuper- vised selection of justification sentences for multi-hop question answering. arXiv preprint arXfv:1911.07176, 2019.
[76] Zhewei Yao, Linjian Ma. Sheng Shen, Kurt Keutzer, and Michael W Mahoney. MIpruning: A multilevel structured pruning framework for transformer-basee models. arXiv preprint wrXiv:2105.14633 2021.
[77] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention-based nlp models for low lateney and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 81 1-824. IEEE, 2020.
[78] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: Quantized 8bit bert. arXiv preprint arXiv:19/0.06///. 2019.
[79] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 1rXiv:/905.078.0 2019.
[80] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: /8/0. /2885, 2018
[81] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang. Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009. /28/2, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Appendix H.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] ZeroQuant a common machine learning technique.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you including theoretical results...
15

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL.?? [Yes] We have very comprehensive experimental details in Appendix A.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes] All details are Appendix
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A] Our results are based on single set of hyperparameters. So it does not rely on the randomness. However, we do provide a tuning results in Appendix E.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Those are in Appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? Yes]] We eited all dataset and github repository used in the paper.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yl8WYHV_WoW", "reviewer": "Reviewer_R9ah", "summary": "The work uses zero-shot row and group-wise quantization that is optimized with fused kernels to perform inference on BERT and large transformers with more than 6B parameters. Layer-by-layer knowledge distillation and post-training quantization is applied to reduce the quantization model degradation.\n", "questions": "- how many seeds do you use for your BERT experiments? For reliable results at least 5 seeds are needed from my experience.\n- I benchmarked raw Int8 vs FP16 and I cannot replicate the speedups that you get. What GEMMs are you using? The best comparison is with a recent cuBLAS version (gold standard) vs Int8 CUTLASS (which is very slow for me compared to cuBLAS, but it fast compared to FP16 CUTLASS GEMM)\n", "limitations": "The limitations are not addressed. While current GPUs only support Int8/Int4 Tensor cores, a discussion on FP8 transformers is missing. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- a broad range of evaluations that go beyond BERT\n- fused functions and open-source software. This takes a lot of effort and is a highly valuable contribution\n\nWeaknesses:\n- speedups benchmarks seem unrealistic. Raw GEMM comparisons within CUTLASS are often favorable when comparing Int8 vs FP16 for fused or infused operations, but comparison with cuBLAS usually yields small speedups.\n- comparison to row/token quantization and row/token+column quantization are missing (Wu et al., 2020)[1]. This is the state-of-the-art for zero-shot quantization, it should be included for comparison with current methods.\n- the paper does not explain the quantization with equations, it is difficult to follow which quantization scheme is actually used. It appears to be absolute maximum or symmetric quantization. But it could also be a form of truncated min-max quantization. A discussion on symmetric vs asymmetric quantization is missing. An evaluation with asymmetric quantization methods is missing (it might address the outliers found in the token statistics)\n- It seems evaluation on BERT favored the author's method due to extensive hyperparameter search. The best result from the hyperparameter search was reported and no multiple seeds were run. This information is only found in the appendix but should be in the main paper. In my experience, at least 5 seeds are required for reliable results and it is best to use the same hyperparameters for both baseline methods and newly developed methods. Otherwise, results are often dominated by noise. \n\nI am happy to increase my score by up to 2 if these weaknesses are addressed and other reviewers do not find other major issues with this work.\n\n[1] Integer quantization for deep learning inference: Principles and empirical evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pUdIEPttrS", "reviewer": "Reviewer_sGAq", "summary": "This paper provides a zero-shot and training-free quantization scheme for large-scale language-based Transformer models.\n\nSpecifically, they investigate the challenges of NLP Transformers and propose group-wise or token-wise quantization for weights and activations, respectively.\n\nAlso, the consider an affordable layer-by-layer knowledge distillation algorithm (LKD) without having to access the original training data, which is often not available.\n\nIn addition, they incorporate above contribution to build a highly optimized quantization system, which can perform real-device speedups.", "questions": "- How about vision transformers (ViTs), will these family of models also suffer form the same problem and therefore can use the proposed techniques?\n\n- The distillation may require even larger models? Which one are you considering? What if we use smaller models to distill the larger models when applying quantization?\n\n- Any opensource idea for the quantization itself? A miminalist codebase that provide soly quantitzation function will make it more convincing.\n\n- A literature review of current staus of quantization on large-scale language models are expected, e.g., a roadmap of the past few years and which point does this paper fit in.\n\nI am open to further boost the score given good answers/rebuttals.", "limitations": "No negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- I am glad to see this post training quantization method can achieve real device speedups, i.e., not fake quantization with dequantization involved for mimicing the behaviour on general GPU devices.\n- The authors are working on large models, which often require a lot of GPU resources to train or retrain. So post training quantization (and data free quantization) is expected. Such a motivation makes a lot of sense to me.\n\nWeaknesses\n\n- The author put a lot of benchmark results while I am somehow get lost how many accuracy drops (a range there will be better) are expected when applying your proposed quantization techniques?\n- Will the proposed techniques also be generalized to small Transformer models? How about CNN models, will there are any differences among them when applying the quantization scheme?\n- How do you handle the softmax functions in the Transformer models? I assume that it will need more specific step as its quantization will lead to more accuracy degradation as I herad form recent publications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pv9HjNw4VZd", "reviewer": "Reviewer_zNp9", "summary": "This paper studies an important task to quantize very large transformer models leveraging three tricks, namely hardware friendly WA quantization scheme, layer-wise distillation, and an optimized backend. Results are fairly SOTA covering mainstream large nets on comprehensive datasets, with clear ablations.", "questions": "- For Fig. 2, what does Dequante mean?\n- Eqn. 1 LKD, though mentioned but has other losses been tried?\n- Table 2, the W4/8 A16 LKD improvement margins are observed smaller than other cases, any specific potential reasons?\n\nMinor\n- Though this is an NLP-oriented paper, how will the method scales potentially to ViTs?", "limitations": "These have been addressed well.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n+ This is a very solid paper in experimental fronts with all software-hardware perspectives highly optimized to push the frontier further.\n+ The evaluation is very thorough.\n+ The method scales well across datasets, and scenarios considering dataset access, etc.\n+ The paper is very well written.\n\nWeekness\n- Though it works great, the deployed methods are known methods and tricks, with very limited new inspiring factors to the literature. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ny2fhyMnxlP", "reviewer": "Reviewer_e8j8", "summary": "This manuscript proposes a new post-training quantization framework for large Transformer-based language models (e.g., GPT-3). Specifically, ZeroQuant consists of hardware-constraint group-wise weight quantization and *kernel fusion* based token-wise activation quantization, low-cost layer-by-layer distillation (LGD), and optimized transformer kernels backend support. The proposed framework successfully quantizes the large language models into INT8 or INT4 representation with negligible performance degradation, which results in remarkable compression performance and real-time speedup.", "questions": "1. In Tab. 3, why are the results of Q-BERT and MP-PTQ [3] not compared? I think it's better to conduct a fair comparison between these two methods.\n2. I wonder whether the authors are willing to release the backend scheduling codes since I think it will be of value to the community and industry.\n\n[3] Bondarenko, Yelysei, Markus Nagel, and Tijmen Blankevoort. \"Understanding and Overcoming the Challenges of Efficient Transformer Quantization.\" *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. 2021.", "limitations": "See Weakness.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n 1. This manuscript is well-motivated, targeting a relevant problem of modern large Transformer-based language model post-training quantization and deployment. The idea of combining group-wise weight quantization under hardware constraints and token-wise activation quantization with the optimized kernel is pretty reasonable. It has its practical value from the modern industrial aspect.\n 2. The idea of LKD is simple but effective. The experiments also show its low-cost benefits, and the results of LKD w/o original training data are interesting to me.\n 3. This manuscript looks solid with extensive experiments across many benchmarks and advanced large language models like BERT and GPT-3. The real-time speedups on A100 devices are reported.\n 4. The overall presentation quality is good.\n\n### Weakness\n\n1. Group-wise weight quantization (GQ) was first proposed by Q-BERT [1], and ZeroQunt directly adopts it but with the hardware constraint, which weakens the technical novelty. As shown in Tab. 9, GQ is the core component that boosts performance. I wonder what the result of the TQ-only is, which is now missing in the ablation study? Do the benefits mostly come from GQ? Please complete the ablation study.\n2. The proposed quantization-optimized Transformer kernels are built upon the CUTLASS library [2]. I suggest the authors include more illustrations of the scheduling framework and show some more in-depth study of the benefits brought by the proposed kernel optimization strategy. Besides, since the proposed deployment flow is targeted at GEMM, it will be a limitation for mobile devices which generally have no GPUs but CPUs.\n3. I suggest the authors include some visualization results of the quantized models to support the claim that the proposed method can deal with the dynamic range of full-precision activations and weights.\n\n[1] Shen, Sheng, et al. \"Q-Bert: Hessian based ultra low precision quantization of bert.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. 2020.\n\n[2] NVIDIA blog. CUTLASS: Fast Linear Algebra in CUDA C++. https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/, December 2017.\n\n---\n#### Post Rebuttal:\n\nI appreciate the detailed rebuttal from the authors. I have carefully read the rebuttal and other reviews, and I have no further concerns except for a conservative view about the technical novelty. As such, I decide to raise my rating to Accept and I believe this paper is a solid paper that will be of interest to the community and industry. I hope the authors could incorporate the valuable evaluations and the reviewers' suggestions into the final version. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept