Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Authors: Yelysei Bondarenko
Markus Nagel
Tijmen Blankevoort
Venue: NeurIPS 2023 poster (2023)
Keywords: transformers
LLM
softmax
attention
outliers
quantization
post-training quantization
TL;DR: We show why modern transformers learn outliers in activations & propose methods that solve this problem and significantly improve the quantized performance.

Abstract: Transformer models have been widely adopted in various domains over the last years and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways for reducing the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher-bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op", or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - _clipped softmax_ and _gated attention_. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers.

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort Qualcomm Al Research"
Amsterdam, The Netherlands j..men/@@ti.quuacccom.comm www.ccuu.lcc.oooommoo
{ybond, markusn, tijmenn
Abstract
Transformer models have been widely adopted in various domains the last years, and especially large language models have advanced the field of AI signif- icantly. Due their size, the capability of these networks has increased tremen- dously, but this has come at the cost of significant increase in necessary com- pute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op" or just a partial update of the residual. To achieve the exact zeros needed the attention matrix for no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two sim- ple (independent) modifications to the attention mechanism clipped softmax and gated attention. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models //github. com/ qualcomm ai rwweearcheeaaeeeeeeeeeeeeereeeee researchhouttiere owwoourrteeeeeeo free- cransformers
(BERT, OPT) and vision transformers. Our source code is available at https:
Introduction
Quantization has been one of the most impactful ways to reduce the computational complexity of transformer networks. Previous work has shown that quantizing networks to 4-bit weights is possible without losing too much accuracy [66, 69]. Some research even shows 4-bit weights might be optimal when trading off model size and bit-width [12].
However, quantizing transformers is not always trivial. When quantizing the activations of a trans- former, significant problems arise with outlers specific layers. This has been noted by several researchers that suggest fixes to transformers after training to ameliorate their effect [13, 67]. These methods are frequently tedious and either require retraining the network, require implementing specific hardware for input-channel quantization [13] or require parts of the activations to still be in higher bit-widths reducing the effectiveness of the activation quantization [67].
In this paper, we set out to solve the transformer outlier problem entirely by changing the architecture of the network itself. We hope make transformers easy quantize from the get-go without needing
"Qulcoom Al Research is an initiative of Qualcomm Technologies, Inc.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

any post-pun post-processing. To do so, we thoroughly analyze why these outliers appear. Previous work has found the existence of these outliers [4. 13], but in our work, we come to a fuller understanding of these outlying values. We find that the outliers occur because attention heads are trying not to update the hidden state, and the process, strong outliers appear due to the softmax function. This happens for language and vision transformers and different specific transformer architectures. This understanding is the foundation for two new tweaks we suggest to transformer architectures that can remove the problem of the outliers entirely.
2 Background and related work
In this section, we briefly cover the basics of neural network quantization and discuss why modern transformers are difficult to quantize.
Quantization One of the most powerful ways to decrease the computational time and memory consumption of neural networks is quantization, which uses low-bit representations for the weights and activation tensors. On top of that, using low-bit fixed-point representations, such as INT8, one can further reduce energy consumption since the fixed-point operations are more efficient than their floating-poin counterparts [23, 59].
We simulate the quantization process in floating-poin according to Jacob et al. [26]. We use the following definition of the quantization function:
X == (x; 8, 2, b) = (clip (1=s)
x + 0, 2 11) -2),
where x denotes the quantizer input (i.e., network weights activations), the step-size, z E 2 the zero point, and E IN the bitwidth. [-1 denotes the round-to- nund-to--eareet-i mnereest-integer operator. This quantization scheme is called uniform affine asymmetric quantization [24, 32, 76] and it is one of the most commonly used quantization schemes because it allows for efficient implementation of fixed-poine arithmetic. In the case of symmetric quantization, we restrict the quantization grid to be symmetric around z = O.
IR+ the scale factor
In this work, we focus post-training quantization (PTQ) methods, which take pre-trained FP32 network and convert directly into fixed-point network without the need for the original training pipeline [2, 5, 7, 25, 32, 35, 41, 43, 44, 75]. These methods require either data or only small calibration dataset and are easier to use compared to quantization-avve awaar training (QAT, Bhalgat et al. 3, Esser et al. 16, Gupta et al. 21, Jacob et al. 26, Krishnamoorthi 32) methods that have you train the entire network for more epochs. For more details on neural network quantization, we refer the reader to [19, 46).
Outliers in Transformers Multiple studies have shown that modern transformer-based language models tend to learn outliers in weights and activations [4, 13, 31]. These outliers are present only in a small fixed set of embedding dimensions, but they appear regularly and consistently across multiple layers and data sequences. It was also shown that those outliers play crucial role in the model predictions and clipping them or by setting to zero the corresponding parameters significantly degrades the model task performance [31, 49]. The strongest in magnitude outliers typically appear at the output of the ferd-forward network, FFN, although Dettmers et al. [13] showed that for big enough wansformer-basee language models they start appearing after every linear layer, including query, key, and value projection layers. This phenomenon holds for many tasks, training objectives and models (both encoder and decoder transformers), including BERT [14], RoBERTa [37], DistilBERT [53], MobileBERT [55], ELECTRA [9], BART [33], XLNet [68], GPT-2 (50), and OPT [74].
Because of these strong outliers, applying per-tensor PTQ for the FFN's output and the residual sum will likely cause a notable error because of the following trade-off between the range and the precision. On the one hand, using large quantization range for small-ranged values leads to loss in representation (high rounding error). On the other hand, a small quantization range for large values leads to very high clipping emror. For the of significant transformer outliers, frequently, good trade-off can be found between the rounding and clipping error, resulting in an overall high error.
There have been numerous attempts to fix the issue of transformer quantization [4, 12, 13, 17, 27, 28, 51, 54, 62, 63, 69, 71]. Most of these approaches resort to finer quantization granularity (row-wise,

---

5000
(a) FFN output in layer #10
(b) FFN output in layer #11
Figure 1: Histograms of outlier counts token positions (blue) and hidden dimensions (green), recorded from the MNLI-m validation set on BERT-base. We use zero-based indexing for dimensions.
channel-wise, group-wise weight and activation quantization), use higher bitwidth and/or different numeric format to represent those outliers better require extra fine-tuning (in the form of QAT and/or knowledge distillation). In other words, they adapt quantization to work with outliers, which often comes at the expense of general applicability or extra inference overhead.
In contrast, in this work, we want to address the root cause of the problem and understand why pre-training protocol that significantly reduces
outliers are learned in the first place and suggest the magnitude outliers yielding way more quantization friendll models that can be effortlessly quantized using PTQ without strong degradation of performance.
Outlier analysis
Outliers BERT models Section 2 we discussed that outliers are present only a few designated embedding dimensions but they appear regularly and consistently across multiple layers and data sequences. We also discussed that the strongest magnitude outliers in BERT typically appear at the output of FFN in the last encoder layers.
We start by taking the pre-trained DERT-base-uncased checkpoint from HuggingFace [65] and fine- tune it on MNLI dataset from the well-known GLUE benchmark [61] (see experimental details in C.I). To identify the outlier dimensions, we pass the MNLI-m validation set through the network and record all outliers' at the FFN output in layers #10 and #112. As we can see in Figure 1, there are indeed only a few hidden dimensions where outliers ever occur. We also notice that the majority of outliers (> 97%) correlate with the position of delimiter tokens [SEP], ".", and ",".
To better understand the role of those outliers, we analyze the attention patterns the corresponding attention heads. BERT-base uses multi-head attention with Theads = 12 and each head operating on consecutive subset of dhead 64 features. Therefore, the hidden dimension #180, which happens to have the highest outlier count in both layers #10 and #11, corresponds to attention head #3. In Figure (and more examples in Appendix A) we show examples of the attention matrices, values and their product for that head.
common pattern found is that the attention head assigns almost all of its probability mass to [SEP] tokens, and other less informative tokens like dots/commas, while these tokens also have small values in V associated with those tokens. This results in small magnitude product between the two (see Figure 2a). This effectively corresponds to a (soft) no-update of the hidden representation, where only small noise added after the residual. In other cases (Figure 2b and 2c), we observe that a significant portion of attention probability is still spent on delimiter tokens. However, by allocating some of the probability mass on other tokens (together with the small values for the delimiter tokens), this results in a (soft) selective update of the hidden representation.
These patterns in self-attentio seem be learned "workaround" for the limitations of having the softmax and the residual connections in cases where the attention head does not want to update the representation of some all of the tokens. These observations [8], Kovaleva et al. [30] that also argued that attending exclusively or almost exclusively to delimiter tokens such as [SEP], periods/commas acts "no-op" when the attention head's function is not applicable.
in line with Clark al.
Wee follow Bondarenko et al. [4] and consider outliers values that exceed 6 standard deviations from mean of the corresponding activation tensor.
We use 1-based indexing for encoder layers and attention heads throughout the paper.

---

(a) Attention layer #11, data sequence #1
(b) Attention layer #11, data sequence #5
(c) Attention layer #10, data sequence
Figure 2: Visualization of the patterns in the self-attention, specifically the attention probabilities, values, and their product (left, middle and right columns, respectively). in attention head #3 for BERT-base, computed on several data sequences from MNLI-m validation set.
(a)
(b)
(c)
(d)
Figure 3: A summary of our outlier analysis for ViT demonstrated on a random image from ImageNet validation (a) An input image. (b) Outliers in the output of layer #11. (c) Cumulative attention weight spent on every patch (matrix of attention probabilities summed over rows) in the attention head #1, layer #12. (d) corresponding matrix of attention probabilities. (e) An average magnitude of values for outlier and non-outlier patches.
Outliers VIT We conduct a similar analysis for Vision transformer [15] trained on ImageNet [52]. For this study, we use pre-trained checkpoint following our experimental setup from Section 5.
We highlight our findings in Figure and provide more examples in Appendix A.2. Our analysis shows many similarities to the BERT case. Instead of delimiter tokens, the majority of outliers seem correlate with some random uninformative patches (e.g., in the background). We also see that the corresponding attention head in the next layer allocates the majority of attention probabilities to the same patches. Finally, those outlier patches on average have distinctly smaller magnitude of values compared to non-outlin non-outlier ones, leading to similar no-update behavior. The fact that those values are as close zero as was in the BERT case might be related to the smaller model capacity?, or relatively shorter training procedure.
Hypothesis Based these observations, pose the following hypothesis how this behavior of attention heads is related to outliers:
1. In order for attention block to not update representation of token the residual, some attention heads want to allocate most of their attention probability mass to some fixed and common set of tokens that have low information content (e.g., delimiter tokens background patches) that can be learned to have a small value function output.
We use ViT/S-16 16 configuration that has only 22M parameters.

---

2. From the definition of the softmax function", it is easy to see that this would require an input of the softmax have a relatively big dynamic range (Figure 4, 1. In fact, the limit case where softmax is exactly zero, this would require an infinite dynamic range:
softmax (x); =0
+00
3. Since Layer Normalization ([1), normalizes the outliers, the magnitude of the FFN output in the previous layer (8) has to be very high to still produce a sufficiently big dynamic range after the LayerNorm. Note, that this is also applicable for the transformer models with LayerNorm applied prior to the self-attention or linear transformations instead, a variant adopted by GPT, OPT, and many vision transformers [15, 38, 57, 58).
4. Finally, as softmax will never output exact zeros, it will always back-propan propagate a gradient signal to grow bigger outliers The outliers will thus tend to become stronger in magnitude, the longer the network is trained.
4 Method
In this section, we introduce our proposed modifications for the softmax attention mechanism. Based on our insights from Section 3, the core idea of these modifications is to grant the model the ability to produce very small the magnitude (or even exact zeros) output attention function, without producing outliers.
Layer Norm
next layer
Linear
Attention
Recall that the self-attention [60] is defined as follows:
K
(x))
Attention(x) softmax
V(x)
dhend
Layer Norm
previous layer
biggest
outliers
where Q. K and V are learnable linear projections of the input x. Most modern transformer models employ multi-headed variant of self-attention, where d'odel features are partitioned into Theads groups dhesd features, and the final output is the concatenation the outputs of (3) applied to each group.
L
Lineal
Linear
Figure 4: A schematic illus- tration of the attention layer in BERT. Hidden activation tensor is denoted by x. < is element-wise addition. A problematic output of the FFN that generates largest in magni- tude outliers is highlighted in red. Notice how those outliers in the previous layer influence the behavior in the attention mechanism in the next layer.
4.1 Clipped softmax
First, we propose replace softmax function in (3) with the follow-
ing clipped softmax:
sofpmad_oft (, ==
clip ((S - 7) softmax(x) 7, 0)
(4)
Here x the input and ( 2 1, 7 I O are the stretch factors which hyper-parameters of the method. This formulation was proposed before in [40] in the context of binary stochastic gates. We can view (4) as stretching the output of the softmax from (0, 1) (7, <) and then clipping back to (0, 1) so that we can represent exact zeros if y < 0 and exact ones if > 1. Specifically, the values of the softmax larger than 1-y rounded to whereas values smaller Ç->
than are rounded to zero.
With this drop-in replacement, we can achieve exact zeros (and ones) with a finite range for the softmax input. In addition to that, whenever values clipped they will not give gradient, preventing the outliers to grow further.
"softmaxx (x), exp (x:) Ei_ exp (x3) Lety y = softmax (x). Then dy # Vi,
ox;
5

---

4.2 Gated attention
alternative way of architecting the model to have small attention output without outliers is to equip it with an explicit conditional gating mechanism, as shown in Figure 5. The idea is that the model can use the gating to either keep or nullify the update the representation of certain tokens and not rely on the attention probabilities and values to achieve the same outcome.
Specifically, we propose the following modification to the attention function:
(x(x)((x))
Gated attention(x) := sigmoid (G(x))
softmax
V(x).
(5)
Vdhcad
Here G' is the gating function, @ is an element-wise multiplication across the token axis and everything else remains the same as in (3). The gating function G' is parameterized by small neural network
that learned jointly with the rest of the model. We replace the attention formulation with the proposed variant in every layer the transformer network.
Gating module design Recall that the input to the attention layer x has shape (T, dmudee) that is reshaped into (Thedds T, dhead) for the multi-headed self-attention, where T is the sequence length. We chose to define the gating function {I..... Theads J. we specify G; Rdders + R and the output of the gating module is πT, E R that is computed follows:
Linear
per-head basis. For each head E
Attention,
. = (x.,..) AI E (1....,7)
(6)
G
Q
K
V
i. sigmoid (
(7)
note that gating modules are shared between different token positions but shared across attention heads.
want our gating module to be as lightweight as possible. To start with, we experiment with G,'s 's parameterized by single linear layer. This gives us a gating module that is computationally inexpensive and has memory overhead of just Thhaads (chead + 1) dmedal extra parameters (which equivalent to extra token) per attention layer". We also investigate the effect of using several other gating functions in Appendix B.1.
Figure 5: A schematic il- lustration of our proposed gated attention.
Experiments
In this section, we evaluate the proposed modifications to self-attention several language models (BERT, OPT) and the vision transformers (ViT). We first test the different hyperparameters for the methods and provide insight into how they work. Then out test our method in terms of accuracy, and the difference in quantization improvement after training. All detailed hyperparameters of our experiments in Appendix
We experiment with BERTT-hase-unasee (109M parameters) pre-training using the masked language modeling (MLM) objective. Following [14],
BERT
the concatenation of the training
sets of BookCorpus [77] and English Wikipedia". We implement our methods in PyTorch [48] and use training and evaluation pipelines from HuggingFace libraries [20, 34, 65]. We follow closely the pre-training procedure from [14]. To speed up training and experimentation, we train with maximum sequence length of 128 for the whole duration of the training. We evaluate on Wikipedia validation set and report the MLM perplexity.
OPT
We experiment with 125M sized variant of OPT [74] pre-training using the causal language modeling (CLM) objective, Due to compute constraints, we train the model on the same dataset that was used for BERT pre-training (BookCorpus Wikipedia) with a maximum sequence length of 512
Forr instance, case of BERT-base this amounts to less 0.009% of the total model size. "Specifically, we use the English subset of Wiki-40b, https: contains cleaned-up up text of English Wikipedia and training/validation splits.
ace. co/ wwwasets//wikkeee that

---

<
FP16 ppl.
Max inf. norm
Avg. kurtosis
W8A8 ppl..
1
4.49+0011
735+55
3076+262
1294=1046
(=
Vanilla)
1.003
4.48:t0.1
715
2159#2238
451
1.03
4.49:000
741 7466
1707#1249
1469:6666
0.003
1
4.46+0000
688-64
2149#110
6364:566
-0.03
1
4.41
20
80
4.55:t0.01
-0.003
1.003
4.47=t0.00
683#23
2494#1225
268=+1220
-0.03
1.03
4.43
22
73
4.56
Table 1: The impact of clipped softmax hyperparameters on BERT-base.
and batch size of 192. Similar to our BERT experiments, we use training and evaluation pipelines from HuggingFace libraries. We evaluate on Wikipedia validation set and report the CLM perplexity.
VIT Finally, we explore the effectiveness of proposed techniques on vision transformer [15] (VTT- S/16 configuration, 22M parameters) trained on ImageNet- IK [11, 52]. For these experiments, adopt the training and validation pipelines from py Torch Image models library [64]. We report top-1 accuracy the validation set of ImageNet.
In all experiments, after the model is trained, we apply 8-bit PTQ. We use uniform affine quantization - -symmetric weights, asymmetric activations - with the static activation range setting, discussed in Section 2. We quantize all weights and activations (both input and output), except the final linear layer for BERT and OPT models. We explore several choices of range estimation (see Appendix C.4) and report the best configuration for each experiment, based on the model performance. We repeat each PTQ experiment 3 times with different random seeds and report mean and standard deviation for accuracy/perplexity.
Quantization setup
We train each network two times with different random seeds and report mean and standard deviation. To assess the amount of outliers in the trained model, we use metrics: the maximum ||x||>> averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of attention layer. These metries have been shown to correlate well with the model quantizability [4, 6].
5.1 The impact of clipped softmax hyperparameters (y and (
We investigate the effect of different values of the clipped softmax stretch parameters and present the results Table We can see that most of the improvement happens when we use ? < (clipping at zero). For instance, using the value of -0.03 leads to significantly smaller infinity norm, kurtosis, and quantized model perplexity, compared to the baseline. It is also clear that in the limit hy + 0 we approach the vanilla softmax attention. Using ( > 1 (clipping at one) yields similar results to the vanilla softmax. Finally, when we combine both y < O and Ç > 1, for which the results seem similar to just clipping at O. We, therefore, conclude that for dampening outliers, only the lower-range clipping allows exact zeros matter. Going forward we use only < and in Appendix B.5 we confirm that < > 1 is not required for VIT.
These observations are in line with our hypothesis that by giving the model the mechanism for representing exact zeros in the attention, we don't 't need to learn the strong outliers.
5.2 Clipped softmax 7 vs. sequence length
As having extra parameer that needs to be tuned per model or setup is generally not desirable, we study the sensitivity of the stretch factor y and its relation with the sequence length T. Recall that the matrix of attention probabilities P has dimensions 7 T and each row sums up to one. Because of that, the average value in P is 1/T. It is reasonable to assume that if we define
Differenn random subsets of training data used for quantizer range estimation.
7

---

512
Rel. log Seppppeeeityy
1.00
0.95
Max inf norm
128
0.90
---
vanilla
T=
0.85
T-64
T=128
T-128
T-192
T-256
T-256
0.75
0.25
0.5
1.0
2.0
4.0
0.0
0.25
0.5
1.0
2.0
4,0
8.0
(a) Relative FP16 log-perplexity
(b) Maximum infinity norm
Figure 6: The performance clipped softmax using 7 = - parameterization on BERT-6L. (a) Relative (compared vanilla softmax pre-training) FP16 log-perplexity on Wikitext validation set. (b) Maximum infinity norm of the attention layer output (note the logarithmic y-axis).
Test
1n-05 0.0031 6.001
0.01
0.5
0.
0.99
300
700
Vanilla
250
Perplexity
FP16
200
accuracy
500
Max inf. norm
150
400
Max inf
6.0
100
Maa
-5
WSAS acc.
5.5
Max int nor
200
10
45
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
-10.0
-15
-3.0
-2.5
0.0
2.5
5.0
bust
bust
(a) BERT-6L
VIT
Figure 7: The performance of Linear gated attention using different bias initialization settings.
y ;= f. where a >> 0 is new hyperparameter, there might be a or a range of values of that works well across different sequence lengths.
To study this, train 6-layer variant of BERT-base (BERT-6L) for 500000 steps WikiTText- 103 [42] with a batch size of 128 with several values of maximum sequence lengths T E (32, 64, 128, 192, 256) and values of
(1/4, 1/2, 4,,,4 As using clipped softmax with a E [2,4] 4] significantly dampens the magnitude of outliers while maintaining good FP16 perplexity across all explored sequence lengths.
from Figure
5.3 The impact of bias initialization in gated attention
In all our gated attention experiments, we randomly initialize the weights G, following [22]. By initializing the bias to specific value, however, initially. More open at the start means we initialize closer to the original network, but given the exponential nature of the gate it might take many iterations for the gate to learn close. Similarly, the gates are all closed at the start, we deviate too far from the original model training, causing a potential decrease in performance. Assuming Linear G;'s 's with small initial weights, if we set the bias to the value of bimit. then Ga(-) F binit and (-)+)) sigmoid (G.((o)dGG(()) Rs sigmoid((nn. =: Tinit at the start of training.
set gates to be open more closed
We study the effect of different values of bin for Linear gated attention on BERT-6L and VIT. We set the bias for all G, 's to the same value of biiit. For BERT-6L, with fixed sequence length of 128. For VIT, we use the main setup, except we train it for 150 epochs instead of 300.
the same setup in Section 5.2,
In Figure we see both BERT and VIT cases that using bias with very high Tini generally performs similarly to the vanilla attention (comparable foating-poin performance but strong outliers and poor quantized performance) while setting bias to have very low mini dampens outliers quite well but leads to strong degradation in the floating-point and quantized performance. The reasonable ranges of Miit seems to be around (0.25, 0.9) for BERT and 10.1, 0.5) ViT. The wide range indicates the relative robustness of our method to this hyperparameter.

---

Model
Method
FP16/32
Max inf. norm
Avg. kurtosis
W8A8
Vanilla
4.49
735
3076 -722
1294-:1066
BERT
Clipped softmax
4.39=00.0
21.5
80
4.52:10.01
(ppl.4)
Gated attention
4.45#0033
39.2+26.0
201=181 =111
4.65=10.04
Vanilla
15.84=0.05
340
1778
21.18:t1.89
OPT (ppl.))
Clipped softmax
16,29=0.07
63.2#88
19728=7480
37.20++22440
Gated attention
15.55=0.05
8.7
18.9
16.02:t0.07
Vanilla
80.75 =0.10
359
1018
69.24=t6.95
ViT
Clipped softmax
80.89=0.13
73.7#149
22.9
79.77+0.25
(ace.1)
Gated attention
81.01 0.06
79.8
19.9
79.82
Table 2: summary of results for our proposed methods applied on BERT, OPT-125m, and VIT.
Model
Method
FP16
Max inf. norm
Avg. kurtosis
W8A8
OPT-350m
Vanilla
13.19
253
2689
37.52
(ppl.))
Gated attention
13.01
65.4
261
14.42:10.06
OPT-1.3B
Vanilla
12.13
428
2756
989.6+t175
(ppl.I)
Gated attention
12.21
67.2
444
29.95 1.0.2
Table 3: The performance of gated attention applied bigger variants of OPT model.
5.4
Main results
We summarize main set of results in Table 2. As see, in almost all cases, both of our proposed techniques dampen the outliers' magnitude to a great extent, reduce the kurtosis, and yield models with significantly higher quantized performance, which is close the original FP16/32 performance. In addition to that, for each model, at least one of our methods also improves the floating-pott point task performance. We hypothesize this is because the network is helped with learning the "no-op" updates more easily. However, we are cautious about the improved performance as this consistent across all syper-parrmeters and it is unclear if it generalizes more architectures and larger models.
The only where method failed perform well the clipped softmax applied to OPT. At the moment, we do not have an explanation of why this is the case and leave it for future work. We list selected paprr-parameters and show extended results in Appendix B. We also show the results of our proposed methods quantized to lower bitwidths Appendix B.7.
Results for bigger models We study the question of scalability of our methods to larger models. In Table we show the gated attention results for 350m and 1.3B variants of OPT. Due to compute constraints, we trained networks for 10t steps with batch size of 256 and the rest is the same in our main pre-training setup. As we can see, our proposed gated attention is also very effective at dampening the outliers and significantly improving the quantized model performance when applied to bigger models. We further study in Appendix B.6 how gated attention can decrease outliers when fine-tuning bigger pre-trained models with outliers.
5.5 Qualitative results
In Figure we compare the learned attention patterns using vanilla softmax and our proposed methods (more examples in Appendix A.1). As we can see, both methods can represent partial/soft no-op behavior, but case our methods this does not require strong outliers elsewhere the network. Note that we found similar patterns in multiple attention heads, but the exact head indices where we observed such patterns depend on random initialization. In the case clipped softmax, the attention probabilities are generally more diffused and smaller in magnitude (which comes from the stretching and clipping). the case of gated attention, the output of the softmax significantly different since the update of the hidden representation is now further modulated by gating probabilities.

---

(a) Vanilla softmax (Attention layer #11, head #3)
(b) Clipped softmax (Attention layer #11, head #8)
(c) Gated attention (Attention layer #11, head #5)
Figure 8: Visualization of the self-attention patterns for BERT-base trained using vanilla and our proposed techniques, computed on data sequence #5 from MNLI-m validation set. (a), (b); attention probabilities, values, and their product. (c): gating probabilities sigmoid (G (x)), attention probabilities (output of softmax), values, and their combined product.
Discussion
"No-op" behavior It is interesting to note that the identified "no-op" behavior likely not limited to transformers and that convolutional architectures likely learn something similar. We also that despite the network trying to learn a full "no-op", still small amount noise is added to each residual, which may constitute form of network regularization. Investigating this further might give us a clue as to why neural networks generalize despite being significantly overparametrized if many parameters rendered unused by not updating the representation in later layers [72].
Limitations We have studied the effect of method large-scale transformers, it would require training very expensive models from scratch. Given the fundamental understanding of the issue underlying solutions, expect the same effect large-scale models. We show very small improvement in FP16/FP32 performance due to our methods, but we do not deem our results exhaustive enough to claim that this will hold in general. Lastly, our methods do have hyperparameter each, although we show that both methods are relatively robust to its hyperparameter, having one never optimal.
Impact As our methods help transformers be more efficient, we expect only positive outcomes of our work. Making neural networks more efficient will help with their high power consumption at inference. further helps move inference from the cloud edge devices which can overcome potential privacy concerns. We cannot fathom any negative impact from our work that is not severely construed.
7 Conclusions
We have thoroughly analyzed the activation outlier problem that makes transformers difficult to quantize. We showed that transformer networks try to learn not update residuals and that by doing so, through the combination of the softmax, residual connections and LayerNorm, significant outliers appear in transformers. Based on this insight, we proposed two methods to address this the core clipped softmax and gated attention. These structural changes to transformers give similar, significantly improve the post-training
not better, floating-point performance after training quantization results. We hope that with these two architectural changes to transformers, anyone can train performacee transformers that are easy to quantize and can benefit from efficient integer inference.
10

---

References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
(eXiv::607.064/ 2016.
[2] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv: 1810.05733, 2018.
[3] Yash Bhalgat, Jinwon Lec, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[4] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort, Understanding and overcoming the chal- lenges of efficient transformer quantization. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 7947-7969, Online and Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011 .ewmleennnnnne main.627. URL https: h/tp///olaaattoogg...oooo org/2021. e021.emmlp..........mmmmm.1ppll1
emnlp- main. 627.
[5] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: novel zero shot quantization framework. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13169-13178 13178, 2020.
[6] Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, Uri Weiser, et al. Robust guantization: One model to rule them all. Advances in neural information processing systems, 33: 5308-5317, 2020.
[7] Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops, pages 3009-3018, 2019.
[8] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's s attention, In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1// 1/w33////11991/1 19-////999988.8.. W19-4828.
URL https://aclantthologg org/
[9] Kevin Clark, ThangThang Luong, Quoc Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020.
[10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quac V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703. 2020.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[12] Tim Dettmers and Luke Zettlemoyer. The case 4-bit precision: k-bit inference scaling laws, arXiv preprint arXiv:22 2022.
[13] Tim Detmmes, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q): 8-bit matrix multiplica- tion for transformers scale. In Advances in Neural Information Processing Systems, 2022.
[14] Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N19. 1423. URL https://aclanthologg org/ N19-144 1423.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11909, 2020.
[16] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR), 2020.

---

[17] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arX1v:2004.07320. 2020.
[18] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gpiq: Accurate post- training quantization for generative pre-trained transformers. arXiv preprint wxXiv22200173222 2022.
[19] Amir Cholami, Sehoon Kim, Zhen Dong. Zhewei Yao. Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2 13630, 2021.
[20] Sylvain Gugger, Lysandre Debu, Thomas Wolf, Philipp Schmid, Zachary Mueller, and Sourab Mangrulkar. Accelerate: Training and inference at scale made simple, efficient and adaptable. https:// //gtthub/// com/ Sugggingfacce/acceeeeaaa 2022.
[21] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737-1746. PMLR, 2015.
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026- 1034, 2015.
[23] M. Horowitz. 1.1 computing's 's energy problem (and what we can do about it). In 2014 IEEE International Solid-Staie Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14, 2014, doi: 10.1109/ ISSCC.2014.67577255
[24] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yaniv,,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-688) 6898, 2017
[25] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006. 10518, 2020.
[26] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pateenn 2018
Recognition, pages
[27] Minsoo Kim, Sihwa Lee, Sukjin Hong, Du-Seong Chang, and Jungwook Choi. Understanding and improving knowledge distillation for quantization-awee awwac training of large transformer encoders. arXiv preprint Xiv:2211. 1014, 2022.
[28] Schoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. arXiv preprint arX1::2001.03321, 2021.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[30] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNN) pages 4365 4374, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 18653/v1/D19 1445, URL
org/D19- 1445.
[31] Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky, Bert busters: Outlier dimen- sions that disrupt transformers. In Findings of the Association for Computational I Linguistics: ACI-IJCNLP 2021, pages 3392-3405, 2021.
[32] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv: 2018.
[33] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising requence-o--sequence pre-training for natural language generation. translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/c200 mai.:.......mmmam........ URL https:////// httpp::s///caattthhhe aalaaaaaaaaantooyy/ae .rgg/ 2020. acl-main. 703.
12

---

[34] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu. Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani. Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysan- dre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https: /ateee///caaaahhhlaaecee otg//021.emm/aeee.oo emnlp-dem dempo .21.
[35] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang. Fengwei Yu, Wei Wang, and Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05442 2021.
[36] Ji I Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for IIM compression and acceleration. arXiv preprint arXiv:2306.00978 2023.
[37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zentlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
[38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[39] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.01001. 2017.
arXiv preprint
[40] Christos Louizos, Max Welling, and Diederik Kingma. Learning sparse neural networks through l_0 regularization. arXiv preprint 1721v:17 2017.
[41] Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but diffcrent: Re- covering ncural network quantization error through weight factorization. International Conference on Machine Learning, pages 4486-4495. PMLR, 2019.
[42] Stephen Merity, Caiming Xiong, James Bradhury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv; (es9.66099075 2016.
[43] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[44] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference Machine Learning (ICML), 2020.
[45] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMIR, 2020.
[46] Markus Nagel. Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Baalen, and Blankevoort Tijmen, white paper on neural network quantization. arXiv preprint 2xXiv:2166.0822 2021.
[47] Vinod Nair and Geoffrey E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on Machine Learning, pages 807-814. Omnipress, 2010.
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-peerformncce deep learning library. In Neural Information Processing Systems (NeuRIPS). 2019.
[49] Giovanni Puccetti, Alessio Miaschi, and Felice Dell' Orletta. How do BERT embeddings organize linguistic knowledge? Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 48-57, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v deelio- 1.6. URL /tttss//acclattho// org/ 2021. deelio- 1.6.
13

---

[50] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[51]
Bita Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang. Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das. Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, and Doug Burger. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. In Neural Information Processing Systems (NeurlPS 2020). ACM, November 2020.
[52] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
[53] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
[54] Sheng Shen, Zhen Dong, Jiayu Ye. Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of hert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821, 2020.
[55] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu. Yiming Yang, and Denny Zhou. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158-2170, Online, July 2020. Association for Computational Linguistics. doi: 10. 11553/61122200.cclllll 2020. acl-main. 195.
195. httptttpp///aaaaaaaaaaaa .rygg
[56] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[57] Hugo Touvron, Matthicu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the 1EEF/CVF International Conference on Computer Vision, pages 32-42, 2021.
[58] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In Computer Vision-ECCV 2022: 17th Eumpean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 516-533. Springer. 2022.
[59] Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, and Tijmen Blankevoort. Fp8 versus int8 for efficient deep learning inference. 2023.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31xt International Conference on Neural Information Processing Systems, pages 6000-6010, 2017.
[61] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels. Belgium, November 2018. Association for Computational Linguistics. doi: URL httpp://wwwt aclweb. org/ att///nthhoo/ieeeee 5446.
10.18653/11/118-55
[62] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint Xiv.2209.13325. 2022.
[63] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145, 2023.
[64] Ross Wightman. Pytorch image models. https://gitte httpppss//gttto....o////gaaaaaaa cttp///wgggttaaa//ptooohhiiggaea 2019.
models..
[65] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformer: Sfateoof-the-ne natural language processing, In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45, 2020.
14

---

[66] Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for transformer models: Latency speedup, composability. and failure cases. 2023.
[67] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In CVPR, 2022.
[68] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XI- net: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, E. d'Alché Buc, Fox, and R. Garnett, editors, Advances Neural Information Pro- cessing Systems, volume 32. Curran Associates, Inc., 2019. URL httpp://pproceeddngg neurips. Paper. pdf.
paper www.le/spape//eee 201///111////
[69] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong Hc. Zeroquant: Efficient and affordable post-training quantization for large-scale transform- In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Ad- vances Neural Information Processing Systems, volume 35, pages 27168-27188 Curran Asso- ciates, Inc., 2022. URL https: / http:://pppooeeeigg.n.eerrraeaae http://eeeeaaaeeeeeeeeeeeeeeeeee cttppppcccppppeppppeeeepee httppppprrffeeppppeeppp //appr/ffieee//ppe////////// a
IEEE/CVF international conference computer vision, pages Conference pdff.
[70] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classificrs with localizable features. In Proceedings of the 2019.
[71] Ofr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8hit hert. arXiv preprint arXiv: 1910,06/88, 2019.
[72] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017.
[73] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0941,, 2017.
[74] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01 00000 2022.
[75] Ritchic Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552, PMLR, 2019.
[76] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.0611 060.16666...0.... 2016.
[77] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
15

---

Supplementary materials
A Additional graphs from outlier analysis
this section, we present additional graphs from our outlier investigation in Section 3 for BERT and vision transformer.
80000
-0000
50
#100
attertion layer
(a)
(c)
Figure 9: A summary of several outlier statistics recorded from ImageNet validation set VIT. (a) Average infinity norm of the output of each attention layer. (b) histogram of outlier counts in attention layer #10 hidden dimensions. We use zero-based indexing for dimensions. (c) A heatmap outlier counts in attention layer #10 vs. patch positions.
A.1 BERT
Recall from Figure 1 that all the outliers are only present in hidden dimensions # #123, #180, #225, #308, #381, #526, #720 (with the majority of them #180, #720). These hidden dimensions correspond to attention heads #2, #3, #4, #5, #6, #9, and #12. In Figures 10 and 11 we show more examples the discovered self-attention patterns for attention heads #3 and #12 (<> hidden dim #180 and #720, respectively). We also show self-attention patterns in attention heads and layers which are not associated with the outliers Figures 12 and 13, respectively. Finally, in Figures 14 and 15 we show more examples of the attention patterns learned in the network trained with clipped softmax and gated attention.
A.2 VIT
Figure 9 further shows that there are a lot of similarities in the outlier behavior the vision transformer, compared to BERT. The strongest magnitude outliers generally happen in the later layers, peaking at layers #10 and #11. The majority of outliers (> 99%) are only ever happening in only 10 hidden dimensions, primarily in dimensions #48 and #43, which corresponds to the attention head #1. Finally, averaged across the entire ImageNet validation set, the outliers seem to be concentrated at the boundaries of the image, which suggest strong correlation with the background (and negative correlation with the object, which is usually in the center of the image the ImageNet dataset).
In Figures 16 and 17, we show more examples of outlier and self-attention patterns in the attention head (++ hidden dimensions #48, #43) for a random subset of images from the ImageNet validation set (in layers #10 and #1 111, respecively).
B Detailed results
In this section, we provide extended results for each model, including the used hyperparameters and other design choices. We also present some additional ablation studies.
16

---

Memory overhead (per attention layer) # extra tokens
Configuration
G
extra parameters
Linear
Plucads
Linear - 1)
Thccuds ( (dhead + 1)
1
MLP
Thbads MLP Whiid 1)
Theads (Thial (doadd + 2) + 1)
Whid
he-hhedd--inaarr
Lincar (dmdeel - !lbeads)
Theads ( (mmodel + 1)
Nhads
Table 4: An overview of the gating function parameterizations explored in this paper and their memory overhead.
B.1 Gating architectures
We investigate the choice of several gating functions, summarized in Table 4. The configuration "MLP" parameterizes each G; with feed-forward net with one hidden layer of size Thii and ReLU non-linearity [47]. We also explore what happens if we allow the mixing of the representation from different attention heads the KAll-heass-lnnen setting, where we use single linear layer to produce the gating probabilities for all attention heads at once. All three options are tested below. Unless explicitly stated otherwise, initialize the bias of the gating function to zero (i.e., iinit
<+ Tinit = 0.5).
B.2
BERT
Method
FP16 ppl.j
Max inf norm
Avg. Kurtosis
W8A8 ppl.
Vanilla
4.49:10.1
735.0
3076
1294
CS ( = 0.005)
4.44
406.6#35.2
1963#753
75.27
CS ( = -0.01)
4.35
198.3#78.7
1581 +:39
7.06 +2.37
CS (7 00.155
4.37=
38.9#7.9
165+34
4.54=0001
CS (7 = 0.02)
4.39
31.7#6.3
90
4.56+00.22
CS (7 = 0.025)
4.39 4.39-10.00 10,00
21.5
80
4.52 :.001
CS ( 0.03)
4.41 +0.01
20.4
79
#0.01
CS ( 0..4)
4.51
19.8
85#7
4.65=0006
GA, Linear (Mini = 0.25)
4.49
139.8=62.3
739
5.05
GA, Linear (Timi 0.5)
4.48
177.3#33.2
652+81
5.13
GA, Linear (Tiii 0.75)
4.49+00.0
71.4=499
262
4,88
GA, Linear (Tinit = 0.9)
4,49
171.5+888
559+141
5.15
GA, MLP (Tiid = 4)
4..0
39.2=26.0
201
4.65=
GA, MLP (Aiid = 64)
4.49+0001
117.0#48.3
507
4.77=0.01
GA, All-heads-linear
4.49
58.3#4122
334
4.67=0003
Table 5: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to BERT-base. We report the masked language modeling perplexity (ppl. for short) on the English Wikipedia validation set for both the floating-point baseline and W8A8 quantized model. We also report the maximum |X|| averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of an attention layer.
Detailed results for BERT-base are summarized in Table 5. As can see, across of the settings, both of our methods significantly dampen the outliers' magnitude, reduce the kurtosis, drastically improve the quantized performance, while maintaining and sometimes improving the FP16 perplexity.
B.3 OPT
Detailed results for OPT-125m are summarized Table
In our early experiments on smaller OPT model, we found that applying the weight decay on LayerNorm weights 7 (which isn't the case, by default) has a strong effect on reducing the outliers' magnitude while yielding the comparable FP16 performance. Therefore, we present the results of applying our gated attention approach in both cases, with and without applying weight decay on LN 7. As can in Table 6, in both cases gated attention (further) dampens the outliers' magnitude to
17

---

Method
LN wd
FP16 ppl.
Max inf norm
Avg. Kurtosis
W8A8 ppl.
Vanilla
X
15.84
339.6 4:722
17771444
21.18
GA, Linear (Tinit 0.1)
X
15.61=00s
35,6*45
42.4#229
16.41#t0118
GA, Linear (Tinit 0.25)
X
15,50
35,8
59,0=48.3
16.25
GA, Linear (Tinit 0.5)
X
15.54
46.5
40.6 40.9
16.30
GA, All-hcads-
X
15.43
32.8
24.2
16.30
Vanilla
V
15,96*005
87.7+319
2080+1460
39.46+16.59
CS ( -1/512)
V
15.99
106.44770
5754
185.23 185.234.22000
CS (7 -2/512)
V
15,90 40002
102.0
11290
60.90
CS (y -4/512)
V
15,86+001
83,1
17174#7791
84.64+10.55
CS (7 = 8/512)
V
16. 16.133th +0000
61.5#99
19204=42844
42.62
CS (- 12/512)
16.29
63.2
19727#7479
37.22
GA, Lincar (Fiii 0.1)
15. 55.69-t008
7.3+04
25,4#110
16.23
GA, Linear (Tini 0.25)
15,55=0005
8,7t206
18,9=1
16.02
GA, Linear (Tinit 0.5)
15.63
10.8
42,0#19
16. 16.20
GA, All-heads-linear
15.53=001
7.9
13.8=1
16.09
Table 6: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to OPT-125m. We report the causal language modeling perplexity (ppl. for short) on the English Wikipedia validation
for both the floating-poin baseline and W8A8 quantized model. We also
report the maximum ||x| averaged across the validation set, and kurtosis of x averaged across all layers, where is the output of an attention layer.
great extent, reduces the kurtosis, and yields models with significantly higher quantized performance, which is close to the original FP16 performance.
B.4 VIT
Method
Patch. Embd. LN
FP32 acc.
Max inf norm
Avg. Kurtosis
W8A8
Vanilla
X
80.75
358.5+8122
1018.3
69.24
CS (7 -0,003)
X
80.24
69.3
25.6
78.71 78.71#033 2033
CS (Y -0.004)
X
80.38
74.
30.6
78. 78.6664449
GA. Linear (Tait 0.25)
X
80.62
86.0#80
23,4
79. 10:00
GA, Linear (Timi 0.5)
X
80.32
88.4
27.9+144
78. 20.
GA. MLP (Mia = 4)
X
80.62
118.2
47.8
78. 79.79
Vanilla
80.98
81.1
24.5
79.62
CS (y = 0.0001)
80.89
73.7
22.9
79.77
CS (Y = 0.0003)
80.92
78,9
23,8
79.63 79.63#005
CS (7 0.0005)
80,95
72.9
24.4407
79.73
CS (Y 0.001)
80,95
22211
24.1
79.69=00.0
CS ( 0.002)
80.80
78.0
25.8
79.32
CS (Y -0.003)
80.79
75,6
28. 28.1
79. 79.00
GA, Linear (Tiit 0.5)
81.01 +-0.0
79,8
19,9
79.82 +
GA, Linear (Tim 0.75)
81.01 t0.05
27.3
21.8:19
79. 80-00s
GA, Linear (Taii 0.9)
80.92
70.6
23.2433
79.6440009
Table 7: Main results for proposed clipped softmax (CS) and gated attention (GA) applied to ViT-S/16. We report the top-1 accuracy on ImageNet- IK validation set for floating-point baseline and W8A8 quantized model. We also report the maximum ||x] oo averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of the attention layer.
Detailed results for ViT-S/16 summarized in Table 7.
After our preliminary experiments on VIT, we noticed that distinct outliers already originate after the patch embeddings. Therefore, we experimented with adding the LayerNorm after the patch
18

---

embeddings (which was absent in the model definition, by default). As we can see in Table 6, together with this change, both our proposed methods greatly dampens the outliers' magnitude. reduces the kurtosis, and yields models with significantly higher quantized performance, which is within 1% of the original FP32 accuracy.
B.5 The impact of clipped softmax hyperparameters! (7 and ( on ViT
FP32 acc.
Max inf norm
W8A8 acc.
1
78.80+042
426#69
71.27=088
(= Vanilla)
1.001
78. 78.78++0.29
411 188
71.24
0
1.002
78. +8
420+47
70.74
1.004
78.8
377
72.31 +:0.06
1.01
78.81+0.30 +:0.0
419-77
71.35=:0.26
0.00001
1
78.81 -:0.11
432-766
69.02-10.19
-0.0011
78.81 #0.36 +0.36
380
64.04=108
-0.001
78. 42.66
282=105
68.43=66.00
-0.003
1
78.26-10.06
99
76.49=t0.48
-0.01
78. 10
391
75.83=1112
0..3
70.26
197=2
65.80=+1411
0.001
1.001
78.45 78.45+00.3
283#82
65.03=8855
-0.003
1.003
78.25-00114
119#17
76.37=19.45
Table 8: The impact of clipped softmax hyperparameters ViT-S/16.
We investigate the effect of different values of the clipped softmax stretch parameters applied the vision transformer and present the results in Table 8. To speed up training, for this experiment we trained VIT for 150 epochs instead of the usual 300 epochs. For this experiment, we did not apply LayerNorm after the patch embeddings.
We found similar observations compared to BERT. Specifically, most of the improvement happens when we use 7 < 0 (clipping at zero) whereas using Ç > 1 (clipping at one) yields similar results the vanilla softmax and combining both 7 and yields similar results compared just
clipping at zero.
B.6 Fine-tuning experiment
Method
FP16 ppl.
Max inf norm
Avg. Kurtosis
Vanilla fine-tuning
29.46
79.3
2086
Fine-tuning w/ Gated attention
29.18
50.9
665
Table 9: OPT-1.3B fine-tuning results with vanilla softmax and gated attention. We report the causal language modeling perplexity (ppl. for short) on the English Wikipedia validation set. We also report the maximum || averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of attention layer.
One of the drawbacks of our proposed framework that requires training from scratch, which could be expensive when applied to very large models. To address this, we explored whether fine-tuning using gated attention can still lead to improved performance and decreased outliers for larger models.
We used OPT-1.33 pre-trained checkpoint from HuggingFace and fine-tuned it on Bookcorpus + Wikipedia for 4000 steps with batch size 256, maximum sequence length 512, maximum learning rate 10-5 and linear LR schedule with 400 warmup steps. We use the same LR for both model parameters and gating module parameters. The rest of hyper-parameters are the same as for our pre-training setup. We adapted our gating approach as follows. We initialized bias biii O. which corresponds to the expected initial gating probability output of mia = 0.5. We multiply the gating probability by 2 so that the expected gate output and approximate the attention output of
19

---

the vanilla softmax at the start of fine-tuning, We add small activation regularization term at the output of each FFN to further encourage the reduction in the magnitude of activations, as unlike when training from scratch outliers are already present in the pre-trained model and need to be suppressed.
As can see from Table 9, fine-tuning with our proposed gated attention results in better perplexity and also reduced maximum infinity norm and the average kurtosis compared to fine-tuning with vanilla softmax.
B.7
Low-hit quantization results
Bitwidths
Weight range estimation
Vanilla
Clipped softmax
Gated attention
FP16
4.49:1001
4.39:L0.0
4.45
W8A8
min-max
1294 1294 1046
4.52-
4.65
W6A8
min-max
598 254
4.64+t0.01
4.79:t0003
W6A8
MSE
6.49
4.56
#:003
W4A8
MSE
6.52-10.02
4.90
5.02-10.03
W6A6
MSE
42.8
6.64
5.90
Table 10: A summary of results for our proposed methods applied to BERT-base and quantized to different bitwidthds for weights and activations (using the same PTQ setup in all previous experi- ments). We report the masked language modeling perplexity on the English Wikipedia validation set.
Note that our proposed methods are not limited to 8-bit quantization only and in general can be combined with other more advanced quantization and weight compression methods, including [18, 35, 36, 45, 63, 67). In Table 10, we show the results of our proposed methods applied to BERT-base and quantized to different bitwidths using our simple post-training quantization setup. Unless stated otherwise, for low-bit (<8-bit) weights and activations by [2, 7] since it gives better results.
MSE range estimator recommended
As we can see, in all cases both of our methods significantly improve the perplexity compared to the vanilla softmax pre- pre-training. We also notice that generally the performance progressively degrades as decrease the bitwidths, which is to be expected. Achieving good results with low-bit activation quantization in general a challenging problem. Further, we notice that the perplexity of the vanilla model significantly improves whenever compared to the INT8 case. This can be explained by the fact that using MSE range estimation for weights leads to implicit clipping of activations (in the same and all subsequent layers in the network), which happen to be of the right amount so that it doesn't 't hurt the perplexity. We found that by going from W8A8 to W6A8 the average kurtosis is reduced from 3406 631 + and the maximum infinity norm is reduced from 577=80 to 158=240, However, in all cases the resulting model still has significantly larger outliers and a worse performance than both of our proposed methods. Finally, as said before, if achieving good low-bit quantization performance is the goal, it is recommended to combine our methods with more advanced quantization techniques.
consider low-bit weight quantization with MSE ranges
C Experimental details
C.1 BERT
Fine-tuning on MNLI dataset We use pre-trained checkpoint BERT- base-uncased (109M param- eters) from HuggingFace repository. We follow standard fine-tuning practices from [14] and [65] Each data sequence tokenized and truncated to the maximum sequence length of 128. Shorter sequences are padded to the same length of 128 using special [PAD] token. We fine-tune for 3 epochs using Adam [29] with a batch size of 16 and no weight decay. The learning rate initially set to its maximum value of of 2 10 and is linearly decayed to zero by the end of fine-tuning.
Pre-training from scratch We follow closely the pre-training procedure from [14]. We concate- nate, tokenize, and split the training set into sequences of length 128 (to speed up training and experimentation, we do not fine-tune on longer sequences of 512). We use the masked language modeling objective with the probability of masking 0.15. We train with a batch size of 256
20

---

sequences for 10th steps, using A optimizer [39] with the maximum learning rate of 10 10-4, learning rate warm up over the first 10* steps, following by linear decay zero by the end of training. We use 1.2 weight decay of 0.01, 1.2 gradient norm clipping of 1.0, and dropout probability of 0.1 on all layers. We also use FP16 mixed- peeed-preeision from HuggingFace Accelerate library [20].
C.2 OPT pre-training
To speed up experimentation, we train OPT-125m sized model on the concatenation of Wikipedia and BookCorpus (same as BERT pre-training). We train with a batch size of 48 and gradient accumulation steps (which results in the effective batch size of 192), so that we can perform pre- training on a single A100 80GB GPU. We concatenate, tokenize, and split the training set into sequences of length 512 and train for 125000 steps (500000 forward passes).
We use the rest of the hyper- peeemparaeeters and follow pre-training practices from [74] and [65]. We initialize weights using a normal distribution with zero mean and a standard deviation of 0.006. All bias terms are initialized zero. We use A optimizer with (31. = (0.9. 0.95). We use the linear learning rate schedule, warming up from 0 to the maximum value of 4- 10 over the first 2000 steps, following by linear decay to zero by the end of training. We use L2 weight decay of 0.1, 1.2 gradient norm clipping of 1.0, and dropout probability of 0.1 on all layers. We also use FP16 mixed-precte -reed-precisoon from HuggingFace Accelerate library [20].
C.3 VTT pre-training
We use the model definition for ViT-S/16 and the training pipeline from PyTorch Image models library [64]. All training is done on resolution 224 224 and 16 x 16 patches. For data augmentation, we use Rugment [10]. Mixup [73], CutMix [70], random image cropping [56], horizontal flip, label smoothing e = 0.1, color jitter 0.4, and random (between bilinear and bicubic) interpolation during training.
We train with batch size of 512 for 300 epochs, using AdamW optimizer and the L2 weight decay of 0.03. We use the cosine learning rate schedule, warming from 10-6 to the maximum value of 10-3 over the first 20 epochs, followed by a LR decay by a factor of 10 every 30 epochs, until reaches the minimum value of 10
C.4 Quantization settings
Weights In all cases, we use symmetric uniform quantization weights. We use min-mas weight quantization for all models except the OPT model, for which we found the MSE estimator to perform better in all cases.
Activations We adopt static range estimation approach, which determines quantization parameters for the network by passing a few batches of calibration data through the model before inference. Specifically, we use a running min-max estimator [32], which uses an exponential moving average of the min and max over multiple batches. In all cases, we use running min-max with 0.9 momentum over 16 batches randomly sampled from respective training sets.
For OPT model, also experiment with using 99.99% and 99.999% percentiles instead of actual min and max. We select the best configuration for each experiment (including baseline). based on the model performance. In almost all cases, we found that setting activation quantization ranges using 99. 99.999% percentiles gives the lowest W8A8 perplexity.
D Compute cost
We compare the runtime of our proposed methods in Table 11. As we can see, the clipped softmax is only marginally more expensive compared using the vanilla softmax attention. The gated attention using the linear G adds the compute overhead between 3% and 8%, depending on the model. We found that adding weight decay on LayerNorm y for OPT and adding the LayerNorm after the patch embeddings for ViT had a negligible effect on the runtime.
In experiments, found this value to perform better compared to the value of 6 10 listed in the paper.
21

---

Model
Vanilla
Clipped softmax
Gated attention (Linear / MLP)
BERT
92,8-12
93.6 t0s
97.7 / 119.1
OPT
53.6
54.4#u4
55.7 64.7
VIT
101.8+03
104.0#07
110.8 122.9
Table 1I: An overview of the runtime the proposed methods, compared to the vanilla pre-training, measured in hours Nvidia-A100 GPUs.
We estimated that the compute cost of producing the main results in the paper is about 320 GPU days (on A A000) and the total cost of the project (including preliminary experiments and ablation studies) to be about 1400 GPU days.
22

---

(a) Attention layer #10, data sequence #16
Attention layer #11, data sequence #16
1.55 BOLL
Attention layer #10, data sequence #21
1100
(d) Attention layer #11, data sequence #21
(c) Attention layer #10, data sequence #61
sieleiin 1rle2
Attention layer #11, data sequence
Attention layer #10, data sequence #88
(h) Attention layer #1 1, data sequence #88
Figure 10: Visualization of the self-attention patterns (attention probabilities, values, and their product in left. middle and right columns, respectively) in attention head #3 (++ channel dim #1 #180) for Base-base trained with vanilla softmax, computed on several random data sequences from MNLI-m validation set.
23

---

(a) Attention layer #10, data sequence #16
Smaar Sucriti
1 In.ii.
Attention layer #11, data sequence #16
11.36 1hi.
Attention layer #10, data sequence #21
(d) Attention layer #11, data sequence #21
(c) Attention layer #10, data sequence #61
Attention layer #11, data sequence
Attention layer #10, data sequence #88
(h) Attention layer #1 1, data sequence #88
Figure 11: Visualization of the self-attention patterns (attention probabilities, values, and their product in left, middle and right columns, respectively) in attention head #12 (+> channel dim #720) for BERT-base trained with vanilla softmax, computed on several random data sequences from MNLI-m validation set.
24

---

(a) Attention layer #10, attention head #1
(b) Attention layer #11, attention head
(c) Attention layer #10, attention head #7
(d) Attention layer #11, attention head #7
(e) Attention layer #10, attention head #8
Stc.0d od.
(1) Attention layer #11, attention head #8
1n. af.
Attention layer #10, attention head #10
(h) Attention layer #11, attention head #10
Figure Visualization of the self-attention patterns (attention probabilities, values, and their product in left, middle and right columns, respectively) in attention heads that are not associated with the strong outliers for BERT-base trained with vanilla softmax, computed on data sequences #16 from MNLI-m validation set.
25

---

St.. 111 Berdedis
(a) Attention layer #1
(b) Attention layer
Stn. oel
(c) Attention layer #3
(d) Attention layer
Attention layer #5
St N
(1) Attention layer #6
Attention layer #7
Innart Birertt
Irsii 2.rrit
(h) Attention layer #8
Figure Visualization of the self-attention patterns (attention probabilities, values, and their product in left, middle and right columns, respectively) in attention head #3 (++ channel dim #1 #180) and the first eight layers BERT-base trained with vanilla softmax, computed on data sequences #16 from MNLI-m validation set.
26

---

Attention layer #10, Attention head #3, data sequence #1
(b) Attention layer #11, Attention head #3, data sequence #1
(c) Attention layer #10, Attention head #3, data sequence #5
(d) Attention layer #11, Attention head #3, data sequence #S
(e) Attention layer #10, Attention head #3, data sequence #7
(f) Attention layer #11, Attention head #3, data sequence #7
Attention layer #10, Attention head #12, data sequence
(h) Attention layer #11, Attention head #12, data sequence #1
Figure 14: Visualization of the self-attention patterns (attention probabilities, values, and their product in left, middle and right columns, respectively) for BERT-base trained with Clipped softmax, computed on several random data sequences from MNLI-m validation set.
27

---

(a) Attention layer #10, Attention head #3, data sequence
(b) Attention layer #11, Attention head #3, data sequence
Attention layer #10, Attention head #3, data sequence #5
(d) Attention layer #11, Attention head #3, data sequence #5
(c) Attention layer #10, Attention head #3, data sequence #7
(1 Attention layer #11, Attention head #3, data sequence #7
Attention layer #10, Attention head #12, data sequence
(h) Attention layer #11, Attention head #12, data sequence
Figure 15: Visualization of the self-attention patterns (from left to right: gating probabilities
sigmoid (G (x)). output of softmax, values, and their combined product) for BERT-base trained with gated attention, computed several random data sequences from MNLI-m validation
28

---

(b)
(d)
Figure 16: A summary of our outlier analysis for ViT demonstrated on random subset from
ImageNet validation set. (a) An input image. (b) Outliers in the output of layer #10. (c) Cumulative attention weight spent on every patch (matrix of attention probabilities summed over rows) in the attention head #1, in the next layer #11. (d) A corresponding matrix of attention probabilities. (e) An average magnitude of values (V) for outlier and non-outlier patches.
29

---

(a)
(b)
(d)
Figure 17: A summary of our outlier analysis for VIT demonstrated on random subset from
ImageNet validation set. (a) input image. (b) Outliers in the output of layer #11. (c) Cumulative attention weight spent on every patch (matrix of attention probabilities summed over rows) in the attention head #1, in the next layer #12. (d) corresponding matrix of attention probabilities. (e) An average magnitude of values (V) for outlier and non-outlier patches.
30


Reviews:
[{"review_id": "7Ul34r15oD", "reviewer": "Reviewer_sWJn", "summary": "This paper analyzes the activation outlier problem. It is shown that the outliers are related to the behavior of transformer networks trying to learn not to update residuals (no-op). To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger. To solve this problem, the authors propose clipped softmax method and gated attention method, which can introduce exact zeros for softmax, thus the outliers can be removed.", "questions": "How to explain the clipped softmax method no better than vanilla network on OPT?\n\nCan you provide the visualization (Figure 2) after using the proposed methods?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper explains why outliers exist in transformers. The analysis and visualization are convincing.\n\nTwo independent methods are proposed (clipped softmax and gated attention) to solve the outlier problem of transformers, so that models can be quantized easily.\n\nExperiments on language models (BERT, OPT) and vision transformers are provided.", "weaknesses": "No results on large scale models and datasets are provided. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1\nSee general response to all reviewers.\n\n## Question 1\nAt the moment, we do not have an explanation of why this is the case but will further investigate this issue.\n\n## Question 2\nThanks for the suggestion. We investigated this and included the visualization, please see Figure 1 in the attached PDF.\n\nAs we can see, both methods can represent a partial/soft no-op behavior, but in case of our methods this does not require strong outliers elsewhere in the network:\n* In the case of clipped softmax, the attention probabilities are generally more diffused and smaller in magnitude (which comes from the clipping).\n* In the case of gated attention, the output of softmax is also quite different since the update of the hidden representation is now further modulated by the gating probabilities.\n\nNote that we found similar patterns in multiple attention heads, but the exact head indices where we observed such patterns depend on random initialization.\n"}, {"review_id": "Th8xztZ2ij", "reviewer": "Reviewer_Vxgc", "summary": "In this paper, the authors show that strong outliers are related to very specific behavior of attention heads that try to learn a “no-op” or just a partial update of the residual. To reduce outliers, the authors propose two simple (independent) modifications to the attention mechanism. This enables them to quantize transformers to full INT8 quantization of the activations without any additional effort. ", "questions": "1. Could the proposed method be applied to lower bitwidths beyond INT8?  (e.g., 4/6 bit)\n2. Does the proposed method require a full retraining of the FP model? When applied to larger models, the training cost could be excessive. ", "limitations": "The authors do not include the limitations and potential negative societal impact in their paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a detailed analysis of the outliers in transformers, which well clarifies their research motivation and provides insights.\n2. The analysis of \"strong outliers are related to attention heads try to learn no-op\" is interesting.", "weaknesses": "1. The authors do not give detailed data distribution after training with the proposed method. I would expect a reduction in the magnitude and number of outliers than in the performance of INT8 quantization that validates the main target of the proposed technique.\n2. The authors mention that it is limited to small models only (125M, which is too small), but quantizing small models with less than 2.7B parameters to INT8 is less challenging compared with large models  [1]. There exists concern is the generalization ability of the proposed method on large-scale models. Analysis or experiments that demonstrate the effectiveness of proposed technique on larger scale model is recommended. \n\n[1] LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NIPS'22", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1\nPlease note, that in all result tables we report next to the floating-point and quantized perplexity/accuracy also two metrics that quantify the magnitude and the frequency of the outliers:\n1) Maximum infinity norm - measures the magnitude of the outliers (by definition).\n2) Kurtosis - the (empirical) fourth standardized moment - measures the tailedness of a distribution, which is directly related to the frequency of the outliers in the distribution (the heavier the tails of the distribution - the higher the kurtosis and vice versa).\n\nAnd as we can see from Table 2, in almost all cases, both of our proposed techniques significantly reduce both the maximum infinity norm (the magnitude) and the kurtosis (the frequency) of the outliers.\n\n## Weakness 2\nSee general response to all reviewers.\n\n## Question 1\nSee general response to all reviewers.\n\n## Question 2\n* Indeed, the idea is that the model is pre-trained from scratch using one of our proposed techniques instead of the vanilla softmax which result in a way more quantization-friendly model. We hope that gated attention and/or clipped softmax will be picked up for future LLM generation such that models are readily quantizable.\n* However, we agree that this could lead to excessive training costs when applied to very large models.\n* To address this, we explored whether fine-tuning using Gated attention can still lead to improved performance and decreased outliers for larger models.\n* Setup: \n\t- we used OPT-1.3B pre-trained checkpoint from HuggingFace and fine-tuned it on Bookcorpus + Wikipedia for 4000 steps with batch size 256, maximum sequence length 512, LR = 1e-5, and linear LR schedule with 400 warmup steps (we use the same LR for both model parameters and gating module parameters) and the rest of hyper-parameters are the same as for our pre-training setup.\n\t- we adapted our gating approach as follows: we used b_init = 0 which results in the expected initial gating probability output of pi_init = 0.5. We multiply the gating probability by 2.0 so that at initialization this value is 1 and approximately resembles the model with vanilla softmax at the start of fine-tuning.\n\t- we add a small activation regularization term (at the output of each FFN) to further encourage the reduction in the magnitude of activations (as unlike when training from scratch outliers are already present in the pretrained model and need to be suppressed).\n* We show results in Table 2 in the attached PDF. As we can see, fine-tuning with our proposed Gated attention results in a better perplexity and also reduced maximum infinity norm and the average kurtosis compared to fine-tuning with vanilla softmax.\n\n## Limitations\nPlease note that in the discussion (Section 6), we mention some limitations and a few points regarding the potential societal impact. In case we missed any limitation or potential negative societal impact, we are happy to update the section based on the reviewer’s suggestion(s)."}, {"review_id": "dxzBK7BJLn", "reviewer": "Reviewer_NwNc", "summary": "The paper proposes two architectural modifications that mitigate the activation outlier problem that makes transformers challenging to quantize. Particularly, the authors conduct a comprehensive analysis of the underlying causes of outlier issues in various pre-trained Transformer models across different tasks and datasets. They identify specific architectural limitations that lead to large outliers during the pre-training phase. To tackle this challenge, the authors propose two solutions: 1/ clipped softmax and 2/ gated attention that can effectively address the outlier problem without sacrificing model performance when compared to vanilla Transformer models across various use cases. Furthermore, the suggested modifications result in smaller outlier values and enhance the quantization performance of the Transformer models, making them more robust to quantization.", "questions": "N/A", "limitations": "Please see the weakness section", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents strong motivations and novel approaches to address outlier issues that distinguish it from prior works. Unlike existing methods that focus on circumventing the impact through post-training techniques, this paper proposes a fundamental training-time solution.\n2. The paper proposes a simple yet highly effective methodology, which directly addresses the fundamental issues and observations.", "weaknesses": "1. Evaluating the proposed methodology by comparing it solely to the naive application of (vanilla) PTQ on the plain Transformer architecture might be somewhat unfair and not informative enough. Various outlier suppression methods have been proposed [] as post-training solutions, and thus it would be better to include a comparison with these methods. Given that the proposed methodology requires from-scratch training, its practical value will be justified if it outperforms post-training methods that require no additional training.\n\n2. The paper introduces two distinct solutions; however, they are two orthogonal/competing methodologies that tackle the same problem. The paper lacks clarity regarding the specific circumstances under which each solution should be employed. That said, how can we decide which scheme to use when training a new model on a new dataset without having to try both and choose the better one?\n\n3. As also stated in the paper, the evaluations have only been conducted on smaller-scale models (100M parameters). While the proposed architectural and train-time modifications show promising results that match/outperform the vanilla Transformer performance, it is unclear if this trend can persist on a larger scale (hundreds of millions to billions of parameters). It would be more informative if the authors can provide across different model scale regimes. Moreover, since the proposed solution is a pre-training methodology, it requires training from scratch and additional risk/cost related to extra hyperparameter tuning, which can potentially limit their practical application.\n\n\n[1] Wei, Xiuying, et al. \"Outlier suppression: Pushing the limit of low-bit transformer language models.\" Advances in Neural Information Processing Systems 35 (2022): 17402-17414.\n\n[2] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" arXiv preprint arXiv:2211.10438 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1\nIndeed, in most of real-world applications the vanilla PTQ might not be good enough.\n\nAt the same time, our proposed methodology is complementary and can be combined with most of more advanced PTQ and weight compression techniques ([1],[2] as well as [3-6] and many more).\n\nOur main motivation of choosing this PTQ baseline was to show how effective our methods are at recovering the quantized network performance even with such a naive and minimal effort PTQ pipeline.\n\nPlease also note that, as stated in the main text, in most of these works, they keep certain parts of the network (often including the problematic residual connections after FFN, LayerNorm etc.) in FP16. On the contrary, we quantize all weigths and activations, including the problematic input, output and residual connections in FFN.\n\nReferences:\n* [3] Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.\n* [4] Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., & Han, S. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv preprint arXiv:2306.00978.\n* [5] Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., & Blankevoort, T. (2020, November). Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning (pp. 7197-7206). PMLR.\n* [6] Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., ... & Gu, S. (2021). Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426.\n\n## Weakness 2\nThis is a good point!\n* Given the fact that clipped softmax did not work for OPT, for the moment we would recommend starting with gated attention (that worked consistently well on all tested models so far) approach but also exploring clipped softmax for new models.\n* On top of that, we expect to further bring more clarity to this question by exploring bigger-scale models.\n\n## Weakness 3\nWe agree on risks regarding the hyper parameter tuning and with all the other points.\nPlease, see general response to all reviewers.\n\n"}, {"review_id": "egdEXL5Q01", "reviewer": "Reviewer_Akrm", "summary": "This paper has thoroughly analyzed the activation outlier problem during quantizing of the transformer model, and further indicates the limitation of existing works. Based on the comprehensive studies, the author proposes two components, namely clipped softmax and gated attention, for regularizing the activation outlier during training. Sufficient experiment results demonstrate the effectiveness and efficiency of the proposed method.\n", "questions": "$\\cdot$ The  avg. kurtosis of OPT model with clipped softmax in tab. 2 is strange, what is the possible reason?", "limitations": "$\\cdot$ The author has claimed in L290-296. I encourage the authors to finish the experiment in the larger model.\n\n$\\cdot$ I encourage the authors to release the code for reproducing.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "$\\cdot$ The proposed components, namely clipped softmax and gated attention are simple but reasonable. Further experiments also support their effectiveness.\n\n$\\cdot$ The observation is solid and further analysis is practical. The experiments include different structures of the transformer model in various tasks.\n\n$\\cdot$ I appreciate the comprehensive studies conducted by the authors. The experiment result of computing cost is convincing.\n\n$\\cdot$ This paper is well-organized and clear to understand. And The authors provide detailed information on related methods, making it easy for readers to understand the progress of the field.\n", "weaknesses": "$\\cdot$ The quantization performance of the proposed method at a lower bit (e.g. 4W4A) is worth looking forward to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness\nSee general response to all reviewers.\n\n## Question\nAt the moment, we do not have an explanation of why this is the case but will further investigate this issue.\n\n## Limitation 1\nSee general response to all reviewers.\n\n## Limitation 2\nWe agree that this will greatly improve the reproducibility and accessibility of our work. We intend to release the code with the camera-ready version."}]

Author rebuttal:
We thank all reviewers for their thoughtful and positive feedback!

We are encouraged they found our work is well-organized and easy to follow (Akrm), has comprehensive experiments (sWjn, Akrm), solid and insightful analysis (sWjn, Vxgc ,Akrm), and presents a simple yet effective methodology, which directly addresses the observations and fundamental issues of outliers (NwNc, Akrm).

One common concern (all reviewers) is the question of the scalability of our methods to larger models, including LLMs. 
* First, we would like to clarify that our methods are not limited to the smaller-scale models and we expect them to translate to larger models/LLMs. We did not include the results for larger models purely because of compute constraints.
* We acknowledge that this is a very important question and that showing that our methods remain effective at a larger scale will greatly improve the likelihood of the adoption of our methods.
* We intend to include the experiments on bigger models with the camera-ready version (up to 1.5B model size).

Several reviewers (Vxgc, Akrm) have also asked if the proposed method be applied to lower bitwidths beyond INT8 (e.g., 4/6 bits)?
* Indeed, the proposed methods can be applied to lower bitwidths and can also be combined with other more advanced PTQ techniques.
* Please, see Table 1 in the attached PDF the results for our proposed methods applied to BERT-base and quantized to different bitwidths using our simple PTQ setup. Unless stated otherwise, for low-bit (<8-bit) weights and activations we use MSE range estimation as recommended by [1,2] since it gives better results.
* As we can see, in all cases both of our methods significantly improve the perplexity compared to the vanilla softmax pre-training.
* We also notice that generally the performance progressively degrades as we decrease the bitwidths, which is as expected. Achieving good results with low-bit activation quantization in general is a challenging problem to this day.
* Finally, we notice that the perplexity of the vanilla model significantly improves whenever we consider a low-bit weight quantization with MSE ranges compared to the INT8 case. This can be explained by the fact that using MSE range estimation for weights leads to an implicit clipping of activations (in the same and all subsequent layers in the network), which happen to be of the right amount so that it doesn't hurt the perplexity. We found that by going from W8A8 to W6A8 the average kurtosis is reduced from 3406±547 to 631±94 and the maximum infinity norm is reduced from 577±80 to 158±40. However, in all cases the resulting model still has significantly larger outliers and a worse performance than both of our proposed methods.

References:
* [1] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. 2018. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv:1810.05723
* [2] Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. 2019. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops, pages 3009–3018.


Meta-review: All reviewers agree that this paper proposes an interesting hypothesis for the outlier problem in transformers. The paper presents enough evidence and further evaluates two simple approaches to mitigate the solution. Main concerns were about the scale of the experiments, and in-consistency in performance of the proposed approaches. Authors provided further experiments in the response and promised to include more in the final version, and acknowledged the inconsistency in some of the approaches, which satisfied reviewer's concerns. Overall, the paper makes clean contribution to an important problem. All reviewers agree on accepting this paper, and I am happy to suggest acceptance.
Decision: Accept (poster)