Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
Authors: Xiuying Wei
Yunchen Zhang
Xiangguo Zhang
Ruihao Gong
Shanghang Zhang
Qi Zhang
Fengwei Yu
Xianglong Liu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that $\boldsymbol \gamma$ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden. The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post-training BERT quantization to the full-precision (FP) level. Our code is available at https://github.com/wimh966/outlier_suppression.

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
Xiuying Weil.2, Yunchen Zhang2+ Xiangguo Zhang', Ruihao Gong
Shanghang Zhang' Qi Zhang" Fengwei Yu² Xianglong Liu State Key Lab of Software Development Environment, Beihang University T Research, *Peking University University of Electronic Science and Technology of China (ueixiuying, zhangyunchen, zhangxiangguo, gongruihao> com shanghangēpku, edu. en, xlliu@buaa edu. cn
Abstract
Transformer architecture has become the fundamental element of the widespread natural language processing (NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the criti- cal bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To funda- mentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that y in LayerNorm (LNN acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, outlier suppression framework including two components: Gamma Migration and Token-Wil Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more -uantization-friendly model without any extra burden. The Token- Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining clipping range with minimal final quantiza- tion loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post- training BERT quantization to the full-precision (FP) level. Our code is available at httpps://il higggtt..bb.....g wommwimm66////mmmmm outlier wtwpllerrsupppsssioooo
propose
Introduction
Transformer [1] has been one of the most common architectures in natural language processing along with lots of popular self-suue models, such as BERT [2], RoBERTa [3], XL.Net [4] and BART [5]. While these pre-trained models have demonstrated significant superiority in performance, the memory and computation overheads have been popular concern, particularly in the real development. Therefore, model compression [6. 7, 8, 9] has attracted much attention from both academia and industry. Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

These days, researchers focus more on quantization of Transformer-base models. [21] proposes an 8-bit quantization scheme for BERT-like models. [22] advises a group-wise quantization technique and analyzes mixed-predd mixed-precision using second-order Hessian information. [23, 24] combine distillation with quantization. [25] approximates nonlinear operations to implement integer-only quantization. Nonetheless, few studies investigate the inherent bottleneck of quantizing' Transformer-based models.
Recently, some papers [26, 27] indicate that the Transformer-based models hold significantly large outliers (even close to 100) and these extreme outliers behave in structured patterns (mainly gather at few embedding dimensions and even become larger on unique tokens). These special outliers can bring devastating damage to the quantization performance (e.g., 12% drop even for the 8- bit [26]). To combat this challenge, existing method [26] chooses bypassing solutions such as a finer quantization granularity. However, this scheme causes an increased computation cost and unavoidably hinders the acceleration effect.
In this paper, to suppress the outliers rather than walk around them, we make an in-depth analysis to investigate the inducement of the outliers and the impact of clipping the outliers. For the inducement, we find that the scaling parameter 7 in the LayerNorm structure works as an outlier amplifier, which amplifies the outliers in the output. By extracting the actívation becomes more robust for quantization. Then by further studying the clipping impact, we discover that the influence of final performance when clipping the outliers varies greatly where some more aggressive outliers covering a large area can be clipped safely without accuracy degradation, but the accuracy can drop suddenly when the important outliers are clipped. More interestingly, though those less important outliers might present in a long tail form, they are only provided by few tokens.
Motivated by the analysis, we propose an outlier suppression framework push the limit of low-bit Transformer language models. Such framework contains two key components: Gamma Migration and Token-Wise Clipping, corresponding the above two findings. The Gamma Migration produces a more quantization- frienddl model by migrating the outlier amplifier y into subsequent modules in an equivalent transformation and bringing more robust activation for quantization without extra computation burden. The Token- Wise Clipping further efficiently finds suitable clipping range with minimal final guantization loss in a coarse-to-fine procedure. The coarse coarse-grained stage, which leverages the fact that those less important outliers only belong to a few tokens, can obtain a preliminary clipping range quickly in a token-wise manner. The graiggri stage then optimizes it. Our proposed framework can be applied to different models and tasks, and coupled with existing methods. More essentially, the thought of outlier suppression shall shed new light on the study of NLP quantization.
To summarize, our contributions are as follows:
1. We delve into the inducement and clipping impact of outliers in the NLP models and draw two critical findings that help handle the bottleneck of transformer quantization.
2. Based on the findings, an outlier suppression framework containing Gamma Migration and Token- Wise Clipping is proposed. This framework is efficient, easy to implement, and blug-and-play
3. The Gamma Migration suppresses the outliers from the inducement aspect and produces a more quantization-friende friendly model without any extra inference time. It transfers the outlier amplifier in LayerNorm to the subsequent modules in an equivalent transformation and contributes to activation with less quantization error.
4. The Token-Wiie Wise Clipping scheme suppresses the outliers from the aspect of importance and produces superior clipping range efficiently. It can skip over those unimportant outliers quickly leveraging the large variance of token range and then focus on the influential area.
5. Extensive experiments on various NLP models (BERT, RoBERTa, BART) and tasks (text classi- fication, question answering, and summarization) prove that our outlier suppression framework sets up a new state of the art for transformer quantization, and for the first time, pushes the 6-bit post-training quantization (PTQ) and 4-bit quantization-an aawre training (QAT) accuracy of BERT to the full-precision level.
2 Preliminaries
Basic Notations. We mark matrices as X and vectors as æ. Operator - denotes scalar multiplication, and is adopted for clement-wise multiplication on matrices vectors. Also, we use Wa as
2

---

[CLS]
great
idea
[SEP]
[CLS]
great
[SEP)
310
309
308
308
307
307
306
300
306
(a) X
(b) 7
(c) X'
Figure 1: Presentation of outliers over X, 7 and X' of LayerNorm BERT-SST-2. For example, at dimension 308, and both have sharper values. By excluding can he seen that X' holds milder distribution than X. More evidence is put in Sec. D.1.
matrix-vector multiplication. Specifically, considering the tokens in NLP tasks, Xt,; stands for the element token and embedding j, and æ, represents the embedding of token I.
Quantizer. Quantization usually includes two operations.
σ = clip(1 H. + 2, O, 2 1), 5
(1)
where (step size), (zero point) quantization parameters, is the bit setting. The first operation called "Quant" maps continuous numbers (x:) to discrete points (7) for tnoger--rithmetiic-e matrix computation. The second operation called "DeQuant" recovers it to & after multiplication.
Outlier analysis
For Transformer-basee models, standard 6/8-bit PTQ or 4-bit QAT would cause severe accuracy degradation. Investigating each quantizer, recognize that the output of LayerNorm structures and GELU functions hold some sharp outliers, which should be responsible for the large quantization error. Evidence and experimental results in Sec. B.2.
To deeply investigate the relationship between the harmful outliers and quantization performance. we explore the underlying inducement and impact of clipping the outliers. Before that, some brief descriptions (see Sec. C.I for detailed ones) about the outliers are given first to help understand the following two parts. The outliers show structured characteristics that they mainly gather at some certain embedding dimensions, and upon these dimensions, the outliers provided by unique tokens like the separate toke and comma even hold more aggressive values.
3.1 Inducement of outliers
For the inducement of outliers, we find that the scaling parameter in LayerNorm amplifies the outliers from embedding dimensions. And the phenomenon that some tokens have sharper outliers might be caused by the uneven token frequency in the pre-training phase (see Sec. C.2). In this part, we mainly explain the first inducement to solve these outliers from the origin. For another one, due to the high cost of adjusting the pre-training, we discuss the clipping impact in the next part suppress these outliers from the clipping perspective.
Considering the challenges of quantizing the LayerNorm, the natural action is to dive into its internal structure. For token at 7th embedding dimension, first normalizes the input using mean (u,) and variance (ar?) each forward pass, then scales and shifts the value with parameter 7 and 3,.
X
U; 7 + B
LayerNorm X
Va +
Then, by observing the parameter distribution of LayerNorm, we surprisingly find that the multiplier Y (Fig. 1b) and the output X (Fig. la) hold outliers at the same embedding dimensions. Besides, (0,3))
the adder B3 denotes smaller range (0.3) compared the output range (c.g., (-60, 0)),

---

ignore it for identifying the key point. That is to say, plays crucial part for the outliers in Fig. la, especially can amplify the outliers across tokens by serving as a shared parameter.
This observation enlightens us to remove the amplification effect by extracting 7 from Eq. (2) and the Non-scaling LayerNorm Eq. (3).
Non scaling LayerNorm: X X,, Ut 33 of +
(3)
7
Fig. Ic and Fig. la show that the output of the Non-scaling LayerNorm denotes a milder distribution with weaker outliers than the normal It not only coineides with that y does strengthen the outliers but also reveals that X' behaves more friendly than X for quantization.
To quantitatively validate the more quantization-friendy distribution X' holds, we adopt the cosine similarity metric evaluate the quantization loss. From Table I, the second row with higher similarity, namely less quantization error, explains that the quantization performance can be improved using Non-scaling LayerNorm.
Tensor
0
2
3
6
7
8
9
10
11
X
97.16
97.03
97.61
94.37
93.41
93.53
93.31
93.61
94.56
95.62
96.13
98.57
X
99.23
99.22
99.11
99.02
98.99
99.00
98.99
98.83
98.70
99.05
99.44
99.07
Table 1: Cosine similarity (%) of the quantized value (6-bit) and the real signal for and X' across 12 LayerNorm after Multi-Head Attention on BERT-SST-2. Higher is better. More evidence in Sec. D.1.
3.2 Impact of outlier clipping
In this part, explore the impact of clipping the outliers design method that find an appropriate clipping range for quantization. The experiments are designed for the clipping impact on the accuracy and token of FP models.
Impact on accuracy. When clipping the outliers and evaluating the final performance, we find that the importance of outliers is highly varied. Take the outliers after GELU an example here (others in See. D.2), Fig. shows that clipping the more aggressive outliers sharply (clipping signals in 10-100 to 10) even does not hurt the full-precision performance with accuracy still at 91.02, while the accuracy drops suddenly to 85.93 with too many outliers cut.
Impact on token. Another key point is the unimportant outliers which can be clipped without even any accuracy drop in FP models only correspond the separator token [SEP] attends to larger values. We are also aware of the different ranges provided by different tokens. From the red points Fig. which represents the proportion of clipped tokens, it can be clearly seen that the more aggressive outliers though occupy a large range from 10 to 100 only matches with 3% tokens. Destroying those sharper outliers belonging to a few tokens will not affect the performance.
few tokens. Motivated by [26], they refer that
05.95
33.03
21.
91 .3.
91 D2.
91.02
91.02
91,05 01,02
No Oli
accwaac
10a
The ratio of cut tokens
10a
92.9
to all tnkens
10a
10a
10a
3.14
104
2.94
-2.93
22.8
2.55
1.08
20
40
Clip Value
100
Figure 2: To detect the impact of clipping the outliers, we first draw the distribution using (mean + 3 std) as its left border, then enumerate the value to cut the tensor RoBERTa-QNLL Red points reflect the proportion of clipped tokens. More evidence in Sec. D.2.

---

The former investigation of accuracy impact suggests taking the final performance into account to find a superior clipping range, where some local optimization methods like [28] are not suitable here. The latter finding in token impact encourages us to leverage the token's s indication to quickly skip over the unimportant area, especially when it presents long tail form where some methods like [29] suffer low efficiency. Based on these, we will introduce our method in Sec. 4.2.
4 Method
In this section, we propose our outlier suppression framework based on the above analysis. Firstly, the Gamma Migration technique is adopted to obtain more quantization- frienndl model by migrating the gamma subsequent modules. Secondly, the Token- Kise Clipping further finds suitable clipping range efficiently by leveraging the large variance of the token range.
4.1 Gamma Migration
As pointed out in Sec. 3.1, activation without going through the scaling parameter provides less quantization error. In this way, we split the LayerNorm function, migrate y into follow-up structures and quantize the output of the Non-scaling LayerNorm. The transformation is equivalent for the FP model and brings more robust activation for the low-bit one. The overall flow is illustrated Fig. 3.
Migration equivalence on FP model. Naturally, referred in Eq. (3), we extract the parameter and transform the LayerNorm into Non-scaling one, thus seperate Xi, from X t.j-
7
Since the residual connection is frequently adopted after LayerNorm ([30, 31, 32J), it is necessary to illustrate the way to migrate parameter into two branches. To be specific, considering the LayerNorm after Multi-Head Attention (Fig. 3), Y will excluded from the LayerNorm and moved to the shortcut branch and weight of the next layer. Then the LayerNorm becomes the Non-scaling one, the shortcut branch establishes a new parameter y, and the weight of the next layer can absorb the
Now, show how the weight absorbs y. For linear layers, have the following equation:
71
Y1 Y2
Vn
W(x (
O
Y2
(W
O
71 Y2
Vn
(5)
...
7/1
1/2
7n
where æ serves as a column vector and 7 E IR". The proof is available in Appendix A. Because 7 is shared parameter, each token's 's embedding satisfies Eq. (5), which promises success of transferring the r into the next layer's 's weight.
DeQuant
DeQuant
X
X
X
Layer Norm
Quant
FFN
Non-scaling
Quant
Figure 3: Comparison of the quantization flow before (left) and after (right) Gamma Migration. The original LayerNorm the Non-scaling LayerNorm y. For other detailed applications such LayerNorm in encoder- decoder structure, see Fig. 6, Fig. 7.
Quantization after migration. Deriving from the above equivalent transformation, we outline the quantization pattern after the migration process. From Fig. 3, the "Quant" process is employed at X', then the quantized output engages in the matrix multiplication on one branch, multiplies parameter and experiences the "DeQuant" process another branch. In fact, this means delaying the y calculation from LayerNorm to the shortcut branch. Hence, this new design will not increase the computation overhead.
Effect of migration. We then analyze the effect Gamma Migration on weight and activation, respectively, to reveal that the activation quantization burden has been greatly alleviated with relatively
5

---

slight influence on weight. To begin with. suppose that the absolute max range of output in the original LayerNorm |max(X')) [maz(>)) for the reason that outliers emerge at the same embedding dimensions among 7, activation before X' and after X scaling function. For activation, extracting the y will reduce the activation range by |mar((7) times. And the results in Table 1 have already validated the profit the transformation brings to activation. For weight, the weight matrix does not have the same embedding outlier phenomenon as the activation. Therefore, the weight range will not be amplified |mar(>>) times after the migration. Experimentally, we also calculate the cosine similarity for the changed weight and observe that y has little impact on weight (Table 2).
Tensor
10
original weight 99.95
99.95
99.95
99.95
99.95
99.95
99.95
99.95
99.95
99.95
99.95
99.95
changed weight
99.95
99.95
99.95
99.90
99.90
99.92
99.94
99.95
99.95
99.95
99.91
99.94
Table 2: Cosine similarity (%) between the quantized value (6-bit) and the real signal for original weight and the changed weight across 12 Intermediate layers on BERT-SST-2. It can be seen that there is little disparity between two rows, especially compared with Table
4.2 Token- Wise Clipping
Based on the analysis, we propose the Token-Wii Clipping method which considers the final loss when finding clipping range and takes a coarse-to- to-fie paradigm to minimize it efficiently in a token-wise manner.
Regarding the very different accuracy impact of clipping the outliers, we search the clipping range, cquivalently the step size s, which has the minimal distance between the final quantized output f() and the real one f defined as Eq. (6). To implement the process efficiently, especially when the unimportant outliers cover a wide area, a coarse-to- coure-to--fine -ine paradigm is designed below.
- (Ifs) ffl
Coarse-grained Stage. At this stage, our aim is to quickly skip over the area where clipping causes little accuracy influence. According to Sec, 3.2, the long tail area only matches with a few tokens. Therefore, suggest using the max value of the embedding at token to be its representatives (min value as representatives for negative outliers). A new tensor with I' elements can be constructed by taking out the maximum signal for each token:
{max((x), 1), max((2), ..., max((T))
where 0² is marked as the collection of upper bounds, of as the collection of lower bounds.
Then for a clipping ratio on o", calculate the corresponding clipping value c" and use it cut the tensor.
c" = quantileeo", aa),
where the quantile function computes the a-th quantiles of o".
10c
Fine Adjustment
10a
Coarse-grained Stage
10c
moxxxx
Fine-grained Stage
Speel
10
Fast Convergence
10a
max(xx,)
fmoxtxx
20
40
60
80
100
Clip Value
Figure 4: Flow diagram of the proposed Token Wise Clipping

---

Through grid search of token-wise clipping ratio, step size 8 c'-d' 2--1 the the bit-width) with minimal quantization loss Eq. (6) is obtained. We mark it as for later optimization.
Fine-grained Stage. At this stage, our aim is to make some fine-grained adjustments in the critical area to further provide a guarantee for the final effect. In detail, with the initialization 5o, a learning procedure based on gradient descent is used update parameter towards loss L(s) with learning rate 17, as described in Eq. (9).
(L(s) 8 1) as
Benefits. We mainly explain the benefits of the coarse gaase-grained stage here from efficiency and quantization performance, where the experimental comparisons with other existing approaches are put in Sec. D.3. For efficiency, because the wide range of outliers only corresponds passing through the unimportant area from the token perspective needs much fewer iterations than from the value perspective. Moreover, the representative collection reduces the size of the tensor (o" distilled from X), so the method can run very fast each iteration. For quantization performance, the first coarse step has already produced a suitable clipping range (Sec. 5.2), which offers a good initialization point for upcoming tuning.
few tokens,
5 Experiments
In this section, we conduct two sets of experiments verify the effectiveness of our outlier suppression framework. Sec. 5.2 shows the effect each component. Sec. 5.3 lists the results compared with other existing approaches across text classification, question answering, and summarization tasks. On the whole, we evaluate GLUE benchmark [33], SQuAD [34, 35], and XSum [36] and CNNi [37] across BERT, RoBERTa, and BART models. Here, 4-4-4 presents 4-bit weight, embedding, and activation. And the model size under a certain bit is put in Table 17.
5.1 Experimental setup
Implementation details. To begin with, we identify the quantization nodes and take a reasonable scheme like the one in FasterTransfone Transformer [38] (Details see Sec. B.1). For PTQ, equipping our framework, we use 256 samples to calibrate the model. For QAT, our methods work on the calibration phase and later are combined with LSQ+ [12], strong baseline for the training phase. For training, hyper-parameters like learning rate are searched both for our methods and baseline techniques for fair comparisons. Details see Appendix F.
Baseline. For PTQ, we compare with the prevalent calibration mechanisms including MinMax [39]. OMSE [28]. Percentile [29]. Quast [40] and PEG [26]. For QAT, we present the results of Q-BERT [22], Q8BERT [21] and PEG [26]. Also, because our framework applying in QAT is coupled with LSQ+ [12], we show the results of the pure LSQ+, and another canonical quantization
approach PACT [41]. Last but not least, the results combined with knowledge distillation (KD)
proposed in TernaryBERT [23] are included as well.
5.2 Ablation Study
In this subsection, we ablate the design elements in the proposed frame- work (Table 3). As general plug-in module, Gamma Migration helps both the MinMax and Token- Clipping. And the Token-Wiis Clip- ping also surpasses the baseline by large margin: 17.53% on QNLI, 13.22% on MRPC (comparisons with other calibration algorithms see Sec. D.3). About the phenomenon that the fine- finengrained stage sometimes does not improve much upon the coarse coarse-grained grained one, we think it t's due to the already good enough results produced by the coarse step.
0.72
Oura, spochs
L501, epochs
LS01, 12 epochs
0.43
0.35
0.12
0.09
5000
10200
15000
20000 23000
Steps
Besides, Fig. 5 conveys that with a good initialization point provided by framework, the training of QAT becomes much faster and easier.
Figure 5: QAT fine-tuning pro- cess BERT-SST-2.
7

---

CoLA
MNLI
MRPC
QNLI
QQP
RIE
SST-2
STS-B
Method
(Mat.)
(acc m/mm)
(fllace)
(acc)
(fl/ace)
(acc)
(acc)
(Pear/Spear)
FP32
62,50
87.75/87.23
93. 93.1/90.44
92.68
88.78/9.6
80.51
95.18
91.04/90.72
Baseline (MinMax)
0.0
34.9/35.0
7166/67.4
62.13
51.88/74337
49.82
77.87
44.11/46.74
MinMax Gamma Migration
0.0
53.53/54.64
87.97/82.84
78.56
78.04/85.3
55.6
85.67
61 01.03/3..22 22
Token- Wise Clipping (Coarse)
34,95
80.56/80.84
85.05/79.41
79.46
85,96/89.31 31
66.43
91.63
82.03/82.45
Token-Wiss Clipping
81.13/81.26
85,59/79.9
79.66
85.83/89.26
64.62
91.63
83.10/3..51
Gamma Migration Token-Wise Clipping
46.35
83.38/83.32
87.50/83.33
86.82
86.82/90.01
67.51
92.2
86.8386693
Table 3: Results of our proposed Gamma Migration and Boken-Wiss Clipping for RoBERTa with 6-bit PTQ.
5.3
Main Results
5.3.1
Results on classification tasks
Method
Bits
CoLAA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Avg.
(W-E-A)
(Mail)
(acc m/mm)
(f1/acc)
(acc)
([1/ccc)
(ace)
(ace)
(Pear/Spear.)
BERT
32-32-32
59.60
84,94/84,76
91.35/87.75
91.84
87.82/90.91
72.56
93.35
89.70/9.22
83.83
MinMax
8-8-8
57.08
82,77/83.47
89 89.90785.78
90.76
87 37.8499077 7.
69.68
92.78
86.83/88.56
82,28
OMSE (28)
8-8-8
57.15
84,04/84.29 299
90.10/85.78
91.12
87.64/90.54
72.20
93.23
87. 871.0/88.00 65
82,90
Ours
8-8-8
61.64
84.38/84.53
91.44/87.75
91.49
87.92/90.77
72.20
93.81
89.23/89.0 01
83.96
OMSE
6-6-6
35.44
74.00/33.30
51/77/7
84.66
76.07/82.12
64.26
86.27
85.57/86. 05
73.52
Percentile [29]
6.6.6
37.32
72.40/71.69
85.09/79.90
79.37
52.58/80.19
61.73
87.27
86.38/87.2 29
72.93
EusyQuant [40]
6.6.6
38.16
75.82/75.66
82.51/77.45
84.94
75.31/81.81
65.34
87.27
85.50/86.33
74.49
Ours
6-6-6
54.40
82.02/81.69
87.45/83.33
89.82
84.69/88.94
70.76
91.86
88.65/88.55
81.19
PEG [26]
8-8-8
59.43
81.25
88.53
91.07
89.42
69.31
92.66
87.92
82.45
Ours
8-8-8
59.83
82,93/82.89
91.33/87.99
90.02
87.4 87.45/90.34
70.04
92.66
88.42/88.81
82.81
PEG
6-6-6
9.46
32.44/32.77
83.64/78.43
49.46
29. 99.93/62.97
70.76
90.14
52.79/53.22
54.11
Ours
6-6-6
42.27
78.54/78.32
85.33/81.13
85.36
78.47/84.66
68.59
91.74
87.33/88,19
77.31
RoBERIa
32-32-32
62,50
67.75/87.23
93.1/90.44
92.68
88. 78/91/
80.51
95.18
91.04/90.72
86.40
MinMax
8-8-8
41.62
87.52/86.88 88
91.56/88.48 48
92.11
88.6099 1.44
76.90
94,82
91.00/90.66
82,94
OMSE
8.8.8
38.59
87.32/87.14
92.39/89.44 46
92.51
87. 97.99/90.99
76.53
94.61
90.95/90.65
82.58
Ours
8.8.8
62,50
87.61/87.J8 .31
92.39//9.44 46
92.53
88.64/91.49 1.49
78.34
94,95
91.08/M0.73
85.96
OMSE
6-6-6
1.81
72.89/72.6
85 38/78/888.68 688
76.53
85.24/88.99 94
64.26
91.17
80.81/81.99
69.63
Percentile
6-6-6
20.73
72.23/73.68
84 84.33/78.44
77.16
82.21/87.44
62.82
88.19
79.41/79.66 64
70.98
EusyQuant
6-6-6
9.28
74.96/75.87
84,31/76.4F
74.04
85.52/89.18 12
62.45
89.56
80.89/82.38
70.01
Ours
6-6-6
46.35
83.38/33.32
87.50/8..33
86.82
86.82/90.01
67.51
92.2
86.83/86.93
79.62
BART
32.32.32
56.32
86.45/86.55 55
91.37/87.50
92.31
88,34/9 1.39
79.06
93.35
90. 19.1118999. 94
84.61
MinMax
8-8-8
55.38
85.87/86. 14
89. 49.44785.29
91.20
88.07/9 1.24
77.98
93.69
89.90/89.7 73
83.89
OMSE
8-8-8
54,56
85.6/86.25 25
90.3 177
90.74
88.21/91.3
78.7
93.58
90.07/89.88 88
83.94
Ours
8-8-8
55.53
86.28/86.17
90.40/86.52
91.44
18.25/91.35
80.51
93.92
90.20/89.95
84,50
OMSE
6.6.6
31.06
41.92/42.08
56.37/54.36
$2.72
78. 78.96/86.02
51.99
87.39
84.38/85.69 69
61.01
Percentile
6.6-6
26.21
74.72/75.29
83.52/74.26
53.71
82.64/87.48
67.15
87.96
63..9/65..0
67.31
QasyQuant
6-6-6
25.66
43.48/43.27
59, 99.26//9.56
30.76
81.8988.67
52.71
87.73
85. 39.39966.774
61.31
Ours
6-6-6
44.51
82.46/82.98
86.41/80.88
86.34
83.60/88.45
71.12
90.94
87.56/87.38
79.10
Table 4: PTQ performance on GLUB benchmark. results taking the same quantization nodes with PRG [26]
for fair comparisons. For the percentile, search the typer-parameter in [0.999, 0.9999, 0.99999] and report the best on dev set.
PTQ. Table 4 shows the results of PTQ on GLUE tasks. For 8-bit BERT models, although previous methods generally behave well, our methods can still achieve satisfying outcomes even on small datasets such as CoLA (4.49% upswings) and STS-B (1.33% upswings). To fully exploit the limit, we try more inspiring setting with weight and activation quantized to 6-bit. It can be seen that ours is indeed close to FP value within 2.64% overall. Meanwhile, we also compare with PEG [26] fairly by taking their quantization nodes. To be noted, their per-embedding group (PEG) quantization certainly brings extra computation overhead and might not be available on real deployment while ours brings favorable results and can enjoy lossless acceleration on hardware. Besides, the experimental results on RoBERTa and BART consistently demonstrate our superiority whereas existing methods suffer from a non-negligible accuracy drop. On average, ours achieves up to 8.64% and 11 11.9% better accuracy on RoBERT and BART. To conclude, our proposed methods push the limit of 6-bit quantization to new state of the art.
8

---

QAT. In particular, we prove the compatibility of our methods on QAT. Table 5 lists the results on BERT, other see Sec. D.4. In a much harder setting (4-4-4 bit quantization). our outlier suppression framework wins near- sear--loating-poe quantization. Yielding good initialization, ours obtain an acceptable accuracy drop (0.7% on QQP, 1.7% on MNLI) without any distillation and data augmentation trick, versus 4.19% and 3. 1.6% of LSQ+. Furthermore, ours still enables performance improvements working with knowledge distillation, especially at 2-bit weight and embedding.
performance with a reduction of 2.70% on average on 4-bit
Bits
CoI.A
MNLI (acc nu/m))
MRPC
QNLI
QQP
RTE
SST-2
STS-B (Pear/Spear.)
Method
Avg.
(W-E-A)
(Mail)
([I/ace)
(acc)
(I1/acc)
(ace)
(aoc)
BERT
32-32-32
39.60
84.94/84.66
91.35/87.75
91.84
87.82/90.91
72.56
93.35
89.70/89.28
83.83
QBBERT [21]
8-8-8
58.48
87.96/-
89.04/-
Q-BERT [22]
8-4-8
78.08/78.96
85.55
PACT
4-4-8
55.23
83.98/83.90
91.58/88.24
91.12
88. 19/199111.0
71.84
91.86
99.73/89.2
82.89
LSQ+
57.70
84. 14.17/84.02
09.75/85.76
91.27
88.18/91.16
70.76
91.97
89.74/89.3
82.84
PEG [26]
4-4-8
57.42
84.22/84.52
89.99885778
90.46
18.15/9
92.78
89.36/88.99
82.45
Ours
84.82/84.89
91.26/87.75
88.45/91.40
92.55
89.71/89.24
84.05
35.45/35.22
81.22/68.38
0.0/63.18
PACT
4-4-4
14/74/74.85
84,9 84.97/80.15
87.31
81. 31.68/86.14 14
83.03
81.64/81.4
69.37
LSQ+
4-4-4
81.40/81.97
68.34/83.82 82
13/11//7.
82.34
14.36/83775
Ours
4-4-4
50.56
83.05/83.24
89.08/84.31
$7.00/90.33
87.64/87.36
81.13
4
4-4-8
57.22
83.69
87.77
91.29
89.64
70.04
92.32
89.13
82.64
Ours
4-4-8
59.57
85,00/84.31
91.07/87.75
91.31
88.35/91.32
72.20
92.43
89.57/89.20
83.60
PEG
4-4-4
0.0
25.45/35.22 22
31 31.62/0.0
49.46
0,0/63.18
52.71
49.08
-0.0219/-0.01999
29.25
Ours
4-4-4
51.93
83.03/3.224
89.39/85.05
90.33
87.38/90.62
91.74
88.36/87.91
LSQ+(+KD)
4-4-4
14.98
83, 59/89.6
92.47/89.46 46
91.16
87.96/91.01
67.87
85.55
84, 1wt.17///000 96
75,99
Ours(+KD) 4.4.4
56.67
84.50/84.65
91.61/88.24
91.45
88.59/91.42
74.37
92.55
89, 89.13/888.78
83.56
I.SQ((+DD)
2-2.4
9.44
83.45/83.38 38
88. 03.00882.60
90.66
87. 1/90/90
55.60
83.60
36 36.69/35..8
66.63
Ours(+KD)
2-2-4
47.02
84.56/8431
90.97/87.25
90.83
88.08/91.12
65.70
91.86
86. 86.12/85.78
80.56
Table 5: Comparison among different QAT strategies with low-biz bit activation on GLUE benchmark for BERT. +
results taking the same quantization nodes with PEG [26] for fair comparisons. *: combined score for MNL.I, MRPC, QQP and STS-B.
5.3.2 Results question answering tasks
To demonstrate the wider applicability of our methods, we evaluate them SQuAD datasets. When going down to 6-bit quantization, the performance of other methods drastically drops. Ours still outperforms them by 4.73% and 15.55% BERT and RoBERTa on SQuAD v1.1. Also, the boost can be 12.31 % and 4.96% on RoBERTa and BART on SQuAD v2.0.
5.3.3 Results on summarization tasks
is of high value to validate the effect of our methods on summarization tasks. We choose classical datasets CNNiDailyMail and XSum and report the ROUGE 1/2/L. score of BART. Table 7 illustrates that our approaches also benefit the encoder-decoder models, and can bring a near- bear-floating-pee performance on 8-bit and about 4% enhancement on 6-bit.
Bits
BERT
RoBERTa
BART
Method
(W-E-A)
SQuAD v1.1
SQuAD v2.0
SQuAD v1.1
SQuAD v2.0
SQuAD v1.1
SQuAD v2.0
Full Prec.
32-32-32
88. 88.28/80.89
77.34/73.60
92.25/85.85
83.30/80.26
91.63/84.79
80.82/77.41
OMSE [28]
8-8-8
87.9080.16
76,88/73.08
91 91.48/84.5
82.53/79.4 41
90.49/83.1 11
79.62/76.12
Ours
8-8-8
87.60/79.80
76.93/73.14
91.57/84.86
82.94/79.72
91.08/84.07
80.55/77.04
OMSE
6-6-6
79.77/69.10 10
67.52/63.00 .09
70.64/58. 8D
45. 45.80/39.95
81. 44/707..61
67. 87/89/63.00 29
Percentile [29]
78.55/67.1
67.24/53.22 28
56.38/51.55
82.45/72.87
48/63/3322
EasyQuum [40]
6-6-6
80.47/70.08 08
71,95/68.00 06
67.85/55.9
47 17/92222
82.41/71.72
69, 99.93/64.94
Ours
6-6-6
84,48/55553
74.69/70.55
80.79/70.83
68.47/64.10
83.68/15.34
74.44/70.36
Table 6: Comparison among typical PTQ approaches in terms of fl/em SQuAD.

---

Method
BissW-E--)
CNN DailyMail
XSum
Bis((WE-A)
CNN DailyMail
XSum
Full prec.
32-32-32
03.22/2..85/2..0
42.82/20. 1/34499 11/44,99
32-32-32
15.62/22.85/42.88
42.82/20. 1/34,99
OMSE [28]
8-8-8
44 44.89/22.0 01222, 18
41.588/18777 77/33.7
6-6-6
5..66/15544//4.49
1..11/2.13/1..22
Percentile [29]
8-8-8
44.67/21 (4/7/11////11.8
47/18866//33.3
6-6-6
37.02/15 33/444445
30. 10/1009430 43/22. 70
EasyQuant [40]
44.98/22.07/42.24
41.65/18 81/33.77
38.86/16 9.99
17.61/2.7911..286
Ours
8-8-8
45.96/23.15443.45
42.29/19.63/34.66
6-6-6
41.00/18.41388551
34.61/12.8627.388
Table 7: PTQ results of BART model on summarization tasks in terms of ROUGE 1/2/L.
6 Conclusions and Discussions of Limitations
In this paper, we analyze the outlier phenomenon from the inducement and clipping impact on transformer language models. Based on these, we establish an outlier suppression framework to suppress the outliers. There also remain some open problems worthy of more in-depth investigations. For example, is valuable systematically explore whether the conclusion in this paper benefits other fields such as computer vision. Besides, as we supplement in the Appendix that the outliers occur not only in the fine-tuned (BERT) models but also the pre-trained ones, it's also meaningful to dive into the pre- pre-training process for better understanding.
Acknowledgment
We sincerely thank the anonymous reviewers for their serious reviews and valuable suggestions to make this better. This work was supported in part by National Natural Science Foundation of China under Grant 62022009 and Grant 61872021, and Beijing Nova Program Science, and Technology under Grant Z191 100001 19050 and the Fundamental Research Funds for the Central Universities.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[2] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv: 18.0050 2018.
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint a9Xi: 1692, 2019.
[4] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[5] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising gequence-to-sqqueee pre-training for natural language generation, translation, and comprehension. arXiv preprint urXiv:I910. 13461,, 2019.
[6] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruming, trained quantization and huffman coding. arXiv preprint 4rX1v:1510.00149, 2015.
[7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503.0253), 2015.
[8] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 111.v.161..0157 2016.
[9] Mingzhu Shen, Feng Liang, Ruihao Gong, Yuhang Li, Chuming Li, Chen Lin, Fengwei Yu, Junjie Yan, and Wanli Ouyang. Once quantization-awan training: High performance extremely low-bit architecture search. In Proceedings of the IEEF/VF International Conference on Computer Vision (ICCV), pages 5340-5349, October 2021.
10

---

[10] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li. Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full- pell-pprciiion and low-bit neural networks. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[11] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. arXiv preprint arXivv1902.088:
2019.
[12] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 696-69 697, 2020.
[13] Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang. and Junjie Yan. Towards unified int8 training for convolutional neural network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[14] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning, pages 4466-4475. PMLR, 2021.
[15] Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint 19Xvv:9111.07190 2019.
[16] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178, 2020.
[17] Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for accurate data-free quantization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2021.
[18] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[19] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021.
[20] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In International Confer- ence on Learning Representations, 2022.
[21] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized Sbit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neurflie Edition (EMC2-NIPS), pages 36-39, IEEE, 2019.
[22] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821, 2020.
[23] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-awar ultra-low bit bert. arXiv preprint arXiv:2009. 12812, 2020.
[24] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprini 17X. 2020.
[25) Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference on machine learning, pages 5506- 5518. PMLR, 2021.
[26] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint 1X0i:2109.11944 2021.
[27] Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. Positional artefacts propagate through masked language model embeddings. arXiv preprint arXiv:2011.04399 2020.
11

---

[28] Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3009-3018. IEEE, 2019.
[29] Jeffrey L. McKinstry, Steven K. Esser, Rathinakumar Appuswamy, Deepika Bablani, John V. Arthur, Izzet B. Yildiz, and Dharmendra S. Modha. Discovering low-precision networks close to full-precision networks for efficient embedded inference, 2019.
[30] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 2019.
[31] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Uni- versal transformers. arXiv preprint arXiv: /807 00000039999.. 2018.
[32] Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
[33] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804 2018.
[34] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 160666666600000 2016.
[35] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't tt know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 784-789. Association for Computational Linguistics, 2018.
[36] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't t give the details. just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807. Association for Computational Linguistics, 2018.
[37] Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequenne RNNs and beyond. In Proceedings The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290. Association for Computational Linguistics, 2016.
[38] NVIDIA. Faster transformer. https://githas http:://ggtabb..../////////aaaa http://///////aaaaaaaaaaaaaaaao
2022.
[39]
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference:
A whitepaper. arXiv preprint arXiv: 08342, 2018.
[40] Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang. Easyquant: Post- training quantization via scale optimization. arXiv preprint arXiv:2006. /6669, 2020.
[41] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 2018.
[42] Intel. Nncf. https: / com/openvinotoollottoooooooooo /nnttt//////////// wttp///t///e/eee///eeeeeeeeeeeee htte://eeedeeee/ettteteeeeeeeeee htwp:gggggggaeeeeattffereerreeee 2022.
[43] Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, Uri Weiser, et al. Robust quantization: One model to rule them all. Advances in neural information processing systems, 33:5308-5317, 2020.
[44] Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2 2021.
[45] Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell'Orletta. Outliers dimensions that disrupt transformers are driven by frequency. arXiv preprint arXiv:2205./1388 2022.
[46] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.06602 2020.
[47] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint 27 2020.
12

---

[48] Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong, Compression generative pre-trained language models via quantization. arXiv preprint arXiv:2203. 10705, 2022.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Discussions we leave some topics as future work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Detailed proofs be found in the supplementary materials.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide code of experiment part of our supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We defer detailed training settings in the supplementary materials.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Since we comprehensively evaluate the robust generaliza- tion for various models different datasets, it would be computationally expensive have the error bar.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "tGK7Qdoxc87", "reviewer": "Reviewer_GEPe", "summary": "This manuscript focuses on Transformer quantization for Bert. It first analyzes the performance regression from outliers during model quantization, specifically in LayerNorm. Following this, the authors propose a Gamma Migration approach to change how we consider the parameter Gamma in LayerNorm, during quantization. In addition, they propose a coarse-to-fine algorithm to find the outliers during token-wise clipping.", "questions": "* What is the total number of parameters of the models in the experiments? With a fixed size training set, the degradation caused by quantization should be highly related to the number of parameters. Please consider adding these numbers to the tables. If possible, please also conduct experiments on different model sizes to see if the proposed approach is beneficial to Transformer models in GLUE in general.\n* For PTQ, what would be the performance of 4-bit quantization? I understand that 4-4-4 PTQ with any quantization paradigm might resulting in garbage results. However, it would interesting to see something like 4-8-8 or 4-6-6. The rationale is that, in real-world scenarios like on-device models, memory usage could be the bottleneck, while the inference speed with 8bit or 6bit are fast enough. In this case, 4bit weight quantization would be very helpful in reducing memory usage.\n", "limitations": "The authors mentioned that \"In Discussions we leave some topics as future work.\" However, I would suggest to explicitly summarize them either in a separate discussion section or change conclusions to something like \"conclusions and discussions of limitations\". Also, I don't think the discussions in the results section covered \"the limitations and potential negative societal impact of their work\" well enough. Please revise according to NeurIPS requirements.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is well organized and very easy to follow\n* The references are enough\n* The technical novelty is enough (see summary)\n\nWeakness\n* Although the authors conducted extensive experiments, there are still some key aspects in results missing. See below\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G3b7IZuEjOk", "reviewer": "Reviewer_LNdx", "summary": "The papers pursues a better solution to deal with outliers in special tokens that are considered as the critical bottleneck for the quantization accuracy. They identify that it is the \\gamma variable in LayerNorm which amplifies outliers and propose an outlier suppression framework, consisting of Gamma Migration and Token-wise Clipping to overcome the quantization bottleneck. Extensive results show that the proposed framework outperforms existing quantization algorithms and for the time time fully recovers full-precision BERT results with a  6-bit post training quantization model and 4-bit model produced by quantization-aware training.", "questions": "- Missing citation: [Compression of Generative Pre-trained Language Models via Quantization](https://arxiv.org/pdf/2203.10705.pdf(https://arxiv.org/pdf/2203.10705.pdf), they developed a dynamic scaling approach and it would be nice to understand how are the observations in two papers are related with each other.\n- Figure 2 is cited in texts multiple times but is not present in the paper.\n", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n- The proposed approach is well motivated by the observation of the layernorm outliers and the experiments are extensive and empirically strong on a wide range of tasks including sentence classification, question answering, and summarization.\n- The analysis and visualization are very informative and well presented\n\n### Weaknesses\n- One figure is missing from the description\n- Some results could be better justified. e.g., It seems that Gamma Migration and Token-Wise Clipping complement each other, but why? How to understand the joint effect of the two techniques and how do they compare to approaches like QBERT or Q8BERT?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0bjfZMrd0Vx", "reviewer": "Reviewer_U3Hv", "summary": "The paper tries to make improvements on the problem of quantizing transformers, wherein the key problem is the presence of outliers. The paper presents an analysis of the problem and tries to track the origin of outliers and find that the scaling parameter (gamma) in layer norm acts as an outlier amplifier. The paper then presents a method named Gamma Migration, which moves computations associated with the gamma parameter to subsequent layers. This method improves quantization performance against a simple minmax baseline. The key gains in the paper come from token-wise clipping, in which the authors propose coarse to fine grained pipeline to clip the more aggressive outliers.", "questions": "1. The ablation study points to the limited efficacy of gamma migration. Have you conducted an analysis of why is this the case? Given that gamma amplification is presented as the origin of the outlier problem, a conclusive analysis of its impact will be useful.\n2. Have you tried the proposed methods (Gamma migration and Token-wise clipping) on GPT-2? \n3. Do you have any comments on whether the work is generic to transformers or is highly applied on the text domain? For example, does the analysis hold on vision transformers or multimodal models?\n4. The tasks tackled are only classification tasks, where word-similarity post quantization can give a good estimate of downstream performance. Have you evaluated any generation tasks on BART?", "limitations": "The authors haven't adequately addressed the limitations and the pain points on reproducing this work. The negative societal impact is irrelevant for this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper are:\n\n1. The empirical results are considerably strong. The 6-bit PTQ results are the first such high-quality results on GLUE.\n2. Given the importance of pre-trained transformer LMs such as BERT, Roberta, the proposed methods can help system developers and deployers. \n3. Each of the methods (although highly applied) is well motivated, with good intermediate quantifications to illustrate their utility.\n\nThe paper has the following weaknesses:\n\n1. In general, the paper writing can certainly be improved, e.g. a number of acronyms such as PTQ, QAT are never introduced to the reader. \n2. Limited novelty of the work. The ablation study points out that token clipping is the most important component, and that gamma migration is of limited importance. This does not gel well with the fact that the analysis on the origin of the outliers is presented as a significant work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept