Title: Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Venue: NeurIPS 2023
Accepted: yes
Keywords: transformers

LLM

softmax

attention

outliers

quantization

post-training quantization
Abstract: Transformer models have been widely adopted in various domains over the last years and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways for reducing the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher-bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op", or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - _clipped softmax_ and _gated attention_. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers.

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

Abstract

Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a 'no-op' or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism clipped softmax and gated attention . We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers. Our source code is available at https: //github.com/qualcomm-ai-research/outlier-free-transformers .

1 Introduction

Quantization has been one of the most impactful ways to reduce the computational complexity of transformer networks. Previous work has shown that quantizing networks to 4-bit weights is possible without losing too much accuracy [66, 69]. Some research even shows 4-bit weights might be optimal when trading off model size and bit-width [12].

However, quantizing transformers is not always trivial. When quantizing the activations of a transformer, significant problems arise with outliers in specific layers. This has been noted by several researchers that suggest fixes to transformers after training to ameliorate their effect [13, 67]. These methods are frequently tedious and either require retraining the network, require implementing specific hardware for input-channel quantization [13] or require parts of the activations to still be in higher bit-widths, reducing the effectiveness of the activation quantization [67].

In this paper, we set out to solve the transformer outlier problem entirely by changing the architecture of the network itself. We hope to make transformers easy to quantize from the get-go without needing

any post-processing. To do so, we thoroughly analyze why these outliers appear. Previous work has found the existence of these outliers [4, 13], but in our work, we come to a fuller understanding of these outlying values. We find that the outliers occur because attention heads are trying not to update the hidden state, and in the process, strong outliers appear due to the softmax function. This happens for language and vision transformers and different specific transformer architectures. This understanding is the foundation for two new tweaks we suggest to transformer architectures that can remove the problem of the outliers entirely.

2 Background and related work

In this section, we briefly cover the basics of neural network quantization and discuss why modern transformers are difficult to quantize.

Quantization One of the most powerful ways to decrease the computational time and memory consumption of neural networks is quantization, which uses low-bit representations for the weights and activation tensors. On top of that, using low-bit fixed-point representations, such as INT8, one can further reduce energy consumption since the fixed-point operations are more efficient than their floating-point counterparts [23, 59].

We simulate the quantization process in floating-point according to Jacob et al. [26]. We use the following definition of the quantization function:

where x denotes the quantizer input (i.e., network weights or activations), s ∈ R + the scale factor or the step-size, z ∈ Z the zero point, and b ∈ N the bitwidth. ⌊·⌉ denotes the round-to-nearest-integer operator. This quantization scheme is called uniform affine or asymmetric quantization [24, 32, 76] and it is one of the most commonly used quantization schemes because it allows for efficient implementation of fixed-point arithmetic. In the case of symmetric quantization, we restrict the quantization grid to be symmetric around z = 0 .

In this work, we focus on post-training quantization (PTQ) methods, which take a pre-trained FP32 network and convert it directly into a fixed-point network without the need for the original training pipeline [2, 5, 7, 25, 32, 35, 41, 43, 44, 75]. These methods require either no data or only a small calibration dataset and are easier to use compared to quantization-aware training (QAT, Bhalgat et al. 3, Esser et al. 16, Gupta et al. 21, Jacob et al. 26, Krishnamoorthi 32) methods that have you train the entire network for more epochs. For more details on neural network quantization, we refer the reader to [19, 46].

Outliers in Transformers Multiple studies have shown that modern transformer-based language models tend to learn outliers in weights and activations [4, 13, 31]. These outliers are present only in a small fixed set of embedding dimensions, but they appear regularly and consistently across multiple layers and data sequences. It was also shown that those outliers play a crucial role in the model predictions and clipping them or by setting to zero the corresponding parameters significantly degrades the model task performance [31, 49]. The strongest in magnitude outliers typically appear at the output of the feed-forward network, FFN, although Dettmers et al. [13] showed that for big enough transformer-based language models they start appearing after every linear layer, including query, key, and value projection layers. This phenomenon holds for many tasks, training objectives and models (both encoder and decoder transformers), including BERT [14], RoBERTa [37], DistilBERT [53], MobileBERT [55], ELECTRA [9], BART [33], XLNet [68], GPT-2 [50], and OPT [74].

Because of these strong outliers, applying per-tensor PTQ for the FFN's output and the residual sum will likely cause a notable error because of the following trade-off between the range and the precision. On the one hand, using a large quantization range for small-ranged values leads to a loss in representation (high rounding error). On the other hand, a small quantization range for large values leads to a very high clipping error. For the case of significant transformer outliers, frequently, no good trade-off can be found between the rounding and clipping error, resulting in an overall high error.

There have been numerous attempts to fix the issue of transformer quantization [4, 12, 13, 17, 27, 28, 51, 54, 62, 63, 69, 71]. Most of these approaches resort to finer quantization granularity (row-wise,

(a) FFN output in layer #10

(b) FFN output in layer #11

40000

40000

80000

35000

50000

35000

30000

40000

30000

40000

60000

25000

20000

15000

30000

20000

25000

20000

15000

20000

10000

5000

10000

10000

5000

.[SEP]，）the

itto.and]

#180#720#123#381#225#526#308

Q

[SEP].,）the'toit and-]so;

0

#180#720#381#123#225#308#526

channel-wise, group-wise weight and activation quantization), use higher bitwidth and/or different numeric format to represent those outliers better or require extra fine-tuning (in the form of QAT and/or knowledge distillation). In other words, they adapt quantization to work with outliers, which often comes at the expense of general applicability or extra inference overhead.

In contrast, in this work, we want to address the root cause of the problem and understand why outliers are learned in the first place and suggest a new pre-training protocol that significantly reduces the magnitude of outliers yielding way more quantization-friendly models that can be effortlessly quantized using PTQ without strong degradation of performance.

3 Outlier analysis

Outliers in BERT models In Section 2 we discussed that outliers are present only in a few designated embedding dimensions but they appear regularly and consistently across multiple layers and data sequences. We also discussed that the strongest magnitude outliers in BERT typically appear at the output of FFN in the last encoder layers.

We start by taking the pre-trained BERT-base-uncased checkpoint from HuggingFace [65] and finetune it on MNLI dataset from the well-known GLUE benchmark [61] (see experimental details in C.1). To identify the outlier dimensions, we pass the MNLI-m validation set through the network and record all outliers 1 at the FFN output in layers #10 and #11 2 . As we can see in Figure 1, there are indeed only a few hidden dimensions where outliers ever occur. We also notice that the majority of outliers ( > 97% ) correlate with the position of delimiter tokens [SEP] , ' . ', and ' , '.

To better understand the role of those outliers, we analyze the attention patterns of the corresponding attention heads. BERT-base uses multi-head attention with n heads = 12 and each head operating on a consecutive subset of d head = 64 features. Therefore, the hidden dimension #180, which happens to have the highest outlier count in both layers #10 and #11, corresponds to attention head #3. In Figure 2 (and more examples in Appendix A.1) we show examples of the attention matrices, values and their product for that head.

A common pattern we found is that the attention head assigns almost all of its probability mass to [SEP] tokens, and other less informative tokens like dots/commas, while these tokens also have small values in V associated with those tokens. This results in a small magnitude product between the two (see Figure 2a). This effectively corresponds to a (soft) no-update of the hidden representation, where only small noise is added after the residual. In other cases (Figure 2b and 2c), we observe that a significant portion of attention probability is still spent on delimiter tokens. However, by allocating some of the probability mass on other tokens (together with the small values for the delimiter tokens), this results in a (soft) selective update of the hidden representation.

These patterns in self-attention seem to be a learned 'workaround' for the limitations of having the softmax and the residual connections in cases where the attention head does not want to update the representation of some or all of the tokens. These observations are in line with Clark et al. [8], Kovaleva et al. [30] that also argued that attending exclusively or almost exclusively to delimiter tokens such as [SEP] , periods/commas acts as a 'no-op' when the attention head's function is not applicable.

(a) Attention layer #11, data sequence #1

(b) Attention layer #11, data sequence #5

[CLS]

[CLS]

0.8

rights

Mew

0.6

nice

are

enough

[SEP]

[SEP]

0.0

0.2

0.4

auofuana

benefits

newest

[SEP]

[SEP]

really

the

likes

everyone

benefts

newest

[SEP}]

[SEP]

really

0.8

0.6

18

0.4

0.0

Z'0

[SEP]

[SEP}

0.0

[SEP]

0.8

0.6

583

SEP

[SEP

[SEP]

0.0

[SEP]

Aapex

SEP

(c) Attention layer #10, data sequence #5

(a)

(b)

(c)

(d)

(e)

25

0.16

0.14

2.0

50

0.12

75

0.10

100

0.08

125

0.06

1.0

150

0.04

0.5

175

0.02

25

50

75

100

125

150

175

0.00

0.0

non-outlier

outlier

Outliers in ViT We conduct a similar analysis for Vision transformer [15] trained on ImageNet [52]. For this study, we use a pre-trained checkpoint following our experimental setup from Section 5.

We highlight our findings in Figure 3 and provide more examples in Appendix A.2. Our analysis shows many similarities to the BERT case. Instead of delimiter tokens, the majority of outliers seem to correlate with some random uninformative patches (e.g., in the background). We also see that the corresponding attention head in the next layer allocates the majority of attention probabilities to the same patches. Finally, those outlier patches on average have a distinctly smaller magnitude of values compared to non-outlier ones, leading to similar no-update behavior. The fact that those values are not as close to zero as it was in the BERT case might be related to the smaller model capacity 3 , or a relatively shorter training procedure.

Hypothesis Based on these observations, we pose the following hypothesis on how this behavior of attention heads is related to outliers:

In order for an attention block to not update a representation of a token on the residual, some attention heads want to allocate most of their attention probability mass to some fixed and common set of tokens that have a low information content (e.g., delimiter tokens or background patches) that can be learned to have a small value function output.

From the definition of the softmax function 4 , it is easy to see that this would require an input of the softmax to have a relatively big dynamic range (Figure 4, 1 ). In fact, in the limit case where softmax is exactly zero, this would require an infinite dynamic range:

̸

Since Layer Normalization ([1], 2 ) normalizes the outliers, the magnitude of the FFN output in the previous layer ( 3 ) has to be very high to still produce a sufficiently big dynamic range after the LayerNorm. Note, that this is also applicable for the transformer models with LayerNorm applied prior to the self-attention or linear transformations instead, a variant adopted by GPT, OPT, and many vision transformers [15, 38, 57, 58].

Finally, as softmax will never output exact zeros, it will always back-propagate a gradient signal to grow bigger outliers 5 . The outliers will thus tend to become stronger in magnitude, the longer the network is trained.

4 Method

In this section, we introduce our proposed modifications for the softmax attention mechanism. Based on our insights from Section 3, the core idea of these modifications is to grant the model the ability to produce very small the magnitude (or even exact zeros) output of attention function, without producing outliers.

Recall that the self-attention [60] is defined as follows:

where Q , K and V are learnable linear projections of the input x . Most modern transformer models employ a multi-headed variant of self-attention, where d model features are partitioned into n heads groups of d head features, and the final output is the concatenation of the outputs of (3) applied to each group.

4.1 Clipped softmax

First, we propose to replace softmax function in (3) with the following clipped softmax:

Here x is the input and ζ ≥ 1 , γ ≤ 0 are the stretch factors which are hyper-parameters of the method. This formulation was proposed before in [40] in the context of binary stochastic gates. We can view (4) as stretching the output of the softmax from (0 , 1) to ( γ, ζ ) and then clipping back to (0 , 1) so that we can represent exact zeros if γ < 0 and exact ones if ζ > 1 . Specifically, the values of the softmax larger than 1 -γ ζ -γ are rounded to one whereas values smaller than -γ ζ -γ are rounded to zero.

With this drop-in replacement, we can achieve exact zeros (and ones) with a finite range for the softmax input. In addition to that, whenever values are clipped they will not give a gradient, preventing the outliers to grow further.

̸

4.2 Gated attention

An alternative way of architecting the model to have a small attention output without outliers is to equip it with an explicit conditional gating mechanism, as shown in Figure 5. The idea is that the model can use the gating to either keep or nullify the update to the representation of certain tokens and not rely on the attention probabilities and values to achieve the same outcome.

Specifically, we propose the following modification to the attention function:

Here G is the gating function, ⊙ is an element-wise multiplication across the token axis and everything else remains the same as in (3). The gating function G is parameterized by a small neural network that is learned jointly with the rest of the model. We replace the attention formulation with the proposed variant in every layer on the transformer network.

Gating module design Recall that the input to the attention layer x has shape ( T, d model ) that is reshaped into ( n heads , T, d head ) for the multi-headed self-attention, where T is the sequence length. We chose to define the gating function on a per-head basis. For each head i ∈ { 1 , . . . , n heads } , we specify G i : R d head → R and the output of the gating module is π i ∈ R T that is computed as follows:

note that gating modules are shared between different token positions but not shared across attention heads.

We want our gating module to be as lightweight as possible. To start with, we experiment with G i 's parameterized by a single linear layer. This gives us a gating module that is computationally inexpensive and has a memory overhead of just n heads · ( d head +1) ∼ d model extra parameters (which is equivalent to 1 extra token) per attention layer 6 . We also investigate the effect of using several other gating functions in Appendix B.1.

5 Experiments

In this section, we evaluate the proposed modifications to self-attention on several language models (BERT, OPT) and the vision transformers (ViT). We first test the different hyperparameters for the methods and provide insight into how they work. Then we set out to test our method in terms of accuracy, and the difference in quantization improvement after training. All detailed hyperparameters of our experiments are in Appendix C.

BERT We experiment with BERT-base-uncased (109M parameters) pre-training using the masked language modeling (MLM) objective. Following [14], we use the concatenation of the training sets of BookCorpus [77] and English Wikipedia 7 . We implement our methods in PyTorch [48] and use training and evaluation pipelines from HuggingFace libraries [20, 34, 65]. We follow closely the pre-training procedure from [14]. To speed up training and experimentation, we train with a maximum sequence length of 128 for the whole duration of the training. We evaluate on Wikipedia validation set and report the MLM perplexity.

OPT We experiment with a 125M sized variant of OPT [74] pre-training using the causal language modeling (CLM) objective. Due to compute constraints, we train the model on the same dataset that was used for BERT pre-training (BookCorpus + Wikipedia) with a maximum sequence length of 512

and batch size of 192 . Similar to our BERT experiments, we use training and evaluation pipelines from HuggingFace libraries. We evaluate on Wikipedia validation set and report the CLM perplexity.

ViT Finally, we explore the effectiveness of proposed techniques on vision transformer [15] ( ViTS/16 configuration, 22M parameters) trained on ImageNet-1K [11, 52]. For these experiments, we adopt the training and validation pipelines from PyTorch Image models library [64]. We report top-1 accuracy on the validation set of ImageNet.

Quantization setup In all experiments, after the model is trained, we apply 8-bit PTQ. We use uniform affine quantization - symmetric weights, asymmetric activations - with the static activation range setting, as discussed in Section 2. We quantize all weights and activations (both input and output), except the final linear layer for BERT and OPT models. We explore several choices of range estimation (see Appendix C.4) and report the best configuration for each experiment, based on the model performance. We repeat each PTQ experiment 3 times with different random seeds 8 and report mean and standard deviation for accuracy/perplexity.

We train each network two times with different random seeds and report mean and standard deviation. To assess the amount of outliers in the trained model, we use two metrics: the maximum ∥ x ∥ ∞ averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of an attention layer. These metrics have been shown to correlate well with the model quantizability [4, 6].

5.1 The impact of clipped softmax hyperparameters ( γ and ζ )

We investigate the effect of different values of the clipped softmax stretch parameters and present the results in Table 1. We can see that most of the improvement happens when we use γ < 0 (clipping at zero). For instance, using the value of γ = -0 . 03 leads to a significantly smaller infinity norm, kurtosis, and quantized model perplexity, compared to the baseline. It is also clear that in the limit | γ |→ 0 we approach the vanilla softmax attention. Using ζ > 1 (clipping at one) yields similar results to the vanilla softmax. Finally, when we combine both γ < 0 and ζ > 1 , for which the results seem similar to just clipping at 0. We, therefore, conclude that for dampening outliers, only the lower-range clipping allows exact zeros matter. Going forward we use only γ < 0 and in Appendix B.5 we confirm that ζ > 1 is not required for ViT.

These observations are in line with our hypothesis that by giving the model the mechanism for representing exact zeros in the attention, we don't need to learn the strong outliers.

5.2 Clipped softmax γ vs. sequence length

As having an extra hyper-parameter that needs to be tuned per model or setup is generally not desirable, we study the sensitivity of the stretch factor γ and its relation with the sequence length T . Recall that the matrix of attention probabilities P has dimensions T × T and each row sums up to one. Because of that, the average value in P is 1 /T . It is reasonable to assume that if we define

(a) Relative FP16 log-perplexity

(b) Maximum infinity norm

512

256

128

orm

nor

64

32

vanilla

T=32

16

T=64

8

T=128

4

T=192

T=256

2

0.25

0.5

1.0

2.0

4.0

8.0

α

log-perplexity

1.00

0.95

0.90

vanilla

T=32

9

FP16

0.85

T=64

T=128

Rel.

0.80

T=192

T=256

0.75

0.25

0.5

1.0

2.0

4.0

8.0

α

(a) BERT-6L

(b) ViT

Tlinit

1e-05 0.0001 0.001

0.01

0.1

0.5

0.9

0.99

300

7.5

Vanilla

250

7.0

FP16 ppl.

norm

Perplexity

6.5

W8A8ppl.

200

Max inf. norm

150

6.0

100

5.5

50

5.0

0

-10.0-7.5-5.0-2.5

0.0

2.5

5.0

binit

Tlinit

1e-05 0.0001 0.001

0.01

0.1

0.5

0.9

0.99

700

80

1

75

600

Top-1 accuracy

705605.50

500

Max inf. norm

Vanilla

400

FP32aCC.

300

W8A8acc.

Max inf. norm

200

100

45

-10.0-7.5-5.0-2.5

0.0

2.5

5.0

binit

γ := -α T , where α > 0 is a new hyperparameter, there might be a set or a range of values of α that works well across different sequence lengths.

To study this, we train a 6-layer variant of BERT-base (BERT-6L) for 500000 steps on WikiText103 [42] with a batch size of 128 with several values of maximum sequence lengths T ∈ { 32 , 64 , 128 , 192 , 256 } and values of α ∈ { 1 / 4 , 1 / 2 , 1 , 2 , 4 , 8 } . As we can see from Figure 6, using a clipped softmax with α ∈ [2 , 4] significantly dampens the magnitude of outliers while maintaining good FP16 perplexity across all explored sequence lengths.

5.3 The impact of bias initialization in gated attention

In all our gated attention experiments, we randomly initialize the weights of G , following [22]. By initializing the bias to a specific value, however, we can set gates to be more open or more closed initially. More open at the start means we initialize closer to the original network, but given the exponential nature of the gate it might take many iterations for the gate to learn to close. Similarly, if the gates are all closed at the start, we deviate too far from the original model training, causing a potential decrease in performance. Assuming Linear G i 's with small initial weights, if we set the bias to the value of b init, then G i ( · ) ≈ b init and π i ( · ) = sigmoid( G i ( · )) ≈ sigmoid( b init ) =: π init , at the start of training.

We study the effect of different values of b init for Linear gated attention on BERT-6L and ViT. We set the bias for all G i 's to the same value of b init. For BERT-6L, we use the same setup as in Section 5.2, with a fixed sequence length of 128. For ViT, we use the main setup, except we train it for 150 epochs instead of 300.

In Figure 7 we see in both BERT and ViT cases that using bias with very high π init generally performs similarly to the vanilla attention (comparable floating-point performance but strong outliers and poor quantized performance) while setting bias to have very low π init dampens outliers quite well but leads to strong degradation in the floating-point and quantized performance. The reasonable ranges of π init seems to be around [0 . 25 , 0 . 9] for BERT and [0 . 1 , 0 . 5] for ViT. The wide range indicates the relative robustness of our method to this hyperparameter.

5.4 Main results

We summarize our main set of results in Table 2. As we can see, in almost all cases, both of our proposed techniques dampen the outliers' magnitude to a great extent, reduce the kurtosis, and yield models with significantly higher quantized performance, which is close to the original FP16/32 performance. In addition to that, for each model, at least one of our methods also improves the floating-point task performance. We hypothesize this is because the network is helped with learning the 'no-op' updates more easily. However, we are cautious about the improved performance as this is not consistent across all hyper-parameters and it is unclear if it generalizes to more architectures and larger models.

The only case where our method failed to perform well was the clipped softmax applied to OPT. At the moment, we do not have an explanation of why this is the case and leave it for future work. We list selected hyper-parameters and show extended results in Appendix B. We also show the results of our proposed methods quantized to lower bitwidths in Appendix B.7.

Results for bigger models We study the question of scalability of our methods to larger models. In Table 3 we show the gated attention results for 350m and 1.3B variants of OPT. Due to compute constraints, we trained networks for 10 5 steps with batch size of 256 and the rest is the same as in our main pre-training setup. As we can see, our proposed gated attention is also very effective at dampening the outliers and significantly improving the quantized model performance when applied to bigger models. We further study in Appendix B.6 how gated attention can decrease outliers when fine-tuning bigger pre-trained models with outliers.

5.5 Qualitative results

In Figure 8 we compare the learned attention patterns using vanilla softmax and our proposed methods (more examples in Appendix A.1). As we can see, both methods can represent a partial/soft no-op behavior, but in case of our methods this does not require strong outliers elsewhere in the network. Note that we found similar patterns in multiple attention heads, but the exact head indices where we observed such patterns depend on random initialization. In the case of clipped softmax, the attention probabilities are generally more diffused and smaller in magnitude (which comes from the stretching and clipping). In the case of gated attention, the output of the softmax is significantly different since the update of the hidden representation is now further modulated by gating probabilities.

0.8

0.6

0.4

0.2

[SEI

[SEP]

[SEP]

0.0

[SEP]

exactly

[SEP]

,0

0.8

0.6

8

0.4

0.0

XCDy

[SEP]

[SEP]

[SEP]

[SEP]

1.0

ALS

0.8

0.6

0.4

0.0

0.2

SEP

0.0

1.0

SEF

[SEP]

[SEP]

[d3s]

[SEP]

6 Discussion

'No-op' behavior It is interesting to note that the identified 'no-op' behavior is likely not limited to transformers and that convolutional architectures likely learn something similar. We also see that despite the network trying to learn a full 'no-op', still a small amount of noise is added to each residual, which may constitute a form of network regularization. Investigating this further might give us a clue as to why neural networks generalize despite being significantly overparametrized if many parameters are rendered unused by not updating the representation in later layers [72].

Limitations We have not studied the effect of our method on large-scale transformers, as it would require training very expensive models from scratch. Given the fundamental understanding of the issue underlying our solutions, we expect the same effect on large-scale models. We show a very small improvement in FP16/FP32 performance due to our methods, but we do not deem our results exhaustive enough to claim that this will hold in general. Lastly, our methods do have a hyperparameter each, although we show that both methods are relatively robust to its hyperparameter, having one is never optimal.

Impact As our methods help transformers to be more efficient, we expect only positive outcomes of our work. Making neural networks more efficient will help with their high power consumption at inference. It further helps to move inference from the cloud to edge devices which can overcome potential privacy concerns. We cannot fathom any negative impact from our work that is not severely construed.

7 Conclusions

We have thoroughly analyzed the activation outlier problem that makes transformers difficult to quantize. We showed that transformer networks try to learn not to update residuals and that by doing so, through the combination of the softmax, residual connections and LayerNorm, significant outliers appear in transformers. Based on this insight, we proposed two methods to address this at the core clipped softmax and gated attention . These structural changes to transformers give similar, if not better, floating-point performance after training but significantly improve the post-training quantization results. We hope that with these two architectural changes to transformers, anyone can train high-performance transformers that are easy to quantize and can benefit from efficient integer inference.

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv:1810.05723 , 2018.

Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2020.

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7947-7969, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.627. URL https://aclanthology.org/2021.emnlp-main.627 .

Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13169-13178, 2020.

Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, Uri Weiser, et al. Robust quantization: One model to rule them all. Advances in neural information processing systems , 33: 5308-5317, 2020.

Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops , pages 3009-3018, 2019.

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 276-286, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4828. URL https://aclanthology.org/ W19-4828 .

Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 , 2020.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. arXiv preprint arXiv:2212.09720 , 2022.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/ N19-1423 .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR) , 2020.

Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07320 , 2020.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022.

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630 , 2021.

Sylvain Gugger, Lysandre Debu, Thomas Wolf, Philipp Schmid, Zachary Mueller, and Sourab Mangrulkar. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/ huggingface/accelerate , 2022.

Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning , pages 1737-1746. PMLR, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

M. Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) , pages 10-14, 2014. doi: 10.1109/ ISSCC.2014.6757323.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research , 18(1):6869-6898, 2017.

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.10518 , 2020.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integerarithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2704-2713, 2018.

Minsoo Kim, Sihwa Lee, Sukjin Hong, Du-Seong Chang, and Jungwook Choi. Understanding and improving knowledge distillation for quantization-aware training of large transformer encoders. arXiv preprint arXiv:2211.11014 , 2022.

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. arXiv preprint arXiv:2101.01321 , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4365-4374, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 18653/v1/D19-1445. URL https://aclanthology.org/D19-1445 .

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 3392-3405, 2021.

Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 , 2018.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.703. URL https://aclanthology.org/ 2020.acl-main.703 .

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario S a s ko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-demo.21 .

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426 , 2021.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978 , 2023.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l _ 0 regularization. arXiv preprint arXiv:1712.01312 , 2017.

Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural network quantization error through weight factorization. In International Conference on Machine Learning , pages 4486-4495. PMLR, 2019.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1325-1334, 2019.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML) , 2020.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning , pages 7197-7206. PMLR, 2020.

Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Blankevoort Tijmen. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295 , 2021.

Vinod Nair and Geoffrey E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on Machine Learning , pages 807-814. Omnipress, 2010.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeuRIPS) . 2019.

Giovanni Puccetti, Alessio Miaschi, and Felice Dell'Orletta. How do BERT embeddings organize linguistic knowledge? In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures , pages 48-57, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.deelio-1.6. URL https://aclanthology.org/ 2021.deelio-1.6 .

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Bita Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, and Doug Burger. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. In Neural Information Processing Systems (NeurIPS 2020) . ACM, November 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 2015.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8815-8821, 2020.

Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2158-2170, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.195. URL https://aclanthology.org/ 2020.acl-main.195 .

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 32-42, 2021.

Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pages 516-533. Springer, 2022.

Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, and Tijmen Blankevoort. Fp8 versus int8 for efficient deep learning inference. 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 6000-6010, 2017.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://www.aclweb.org/anthology/W18-5446 .

Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13325 , 2022.

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145 , 2023.

Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models , 2019.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages 38-45, 2020.

Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases. 2023.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In CVPR , 2022.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper_files/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf .

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 27168-27183. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ adf7fa39d65e2983d724ff7da57f00ac-Paper-Conference.pdf .

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188 , 2019.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning , pages 7543-7552. PMLR, 2019.

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV) , December 2015.

Supplementary materials

A Additional graphs from outlier analysis

In this section, we present additional graphs from our outlier investigation in Section 3 for BERT and vision transformer.

(a)

(b)

(c)

1e6

400

80000

layeroutput infnorm

1.0

350

300

0.8

60000

250

0.6

200

40000

150

0.4

100

0.2

20000

50

0

0.0

1234567

attentionlayer

6

101112

43

48

#272

#324

#133

353

#248

#11

#166

#111

A.1 BERT

Recall from Figure 1 that all the outliers are only present in hidden dimensions #123, #180, #225, #308, #381, #526, #720 (with the majority of them in #180, #720). These hidden dimensions correspond to attention heads #2, #3, #4, #5, #6, #9, and #12. In Figures 10 and 11 we show more examples of the discovered self-attention patterns for attention heads #3 and #12 ( ↔ hidden dim #180 and #720, respectively). We also show self-attention patterns in attention heads and layers which are not associated with the outliers in Figures 12 and 13, respectively. Finally, in Figures 14 and 15 we show more examples of the attention patterns learned in the network trained with clipped softmax and gated attention.

A.2 ViT

Figure 9 further shows that there are a lot of similarities in the outlier behavior in the vision transformer, compared to BERT. The strongest magnitude outliers generally happen in the later layers, peaking at layers #10 and #11. The majority of outliers ( > 99% ) are only ever happening in only 10 hidden dimensions, primarily in dimensions #48 and #43, which corresponds to the attention head #1. Finally, averaged across the entire ImageNet validation set, the outliers seem to be concentrated at the boundaries of the image, which suggest a strong correlation with the background (and a negative correlation with the object, which is usually in the center of the image in the ImageNet dataset).

In Figures 16 and 17, we show more examples of outlier and self-attention patterns in the attention head #1 ( ↔ hidden dimensions #48, #43) for a random subset of images from the ImageNet validation set (in layers #10 and #11, respecively).

B Detailed results

In this section, we provide extended results for each model, including the used hyperparameters and other design choices. We also present some additional ablation studies.

B.1 Gating architectures

We investigate the choice of several gating functions, summarized in Table 4. The configuration 'MLP' parameterizes each G i with a feed-forward net with one hidden layer of size n hid and a ReLU non-linearity [47]. We also explore what happens if we allow the mixing of the representation from different attention heads in the 'All-heads-linear' setting, where we use a single linear layer to produce the gating probabilities for all attention heads at once. All three options are tested below. Unless explicitly stated otherwise, we initialize the bias of the gating function to zero (i.e., b init = 0 ↔ π init = 0 . 5 ).

B.2 BERT

Detailed results for BERT-base are summarized in Table 5. As we can see, across most of the settings, both of our methods significantly dampen the outliers' magnitude, reduce the kurtosis, drastically improve the quantized performance, while maintaining and sometimes improving the FP16 perplexity.

B.3 OPT

Detailed results for OPT-125m are summarized in Table 6.

In our early experiments on a smaller OPT model, we found that applying the weight decay on LayerNorm weights γ (which isn't the case, by default) has a strong effect on reducing the outliers' magnitude while yielding the comparable FP16 performance. Therefore, we present the results of applying our gated attention approach in both cases, with and without applying weight decay on LN γ . As we can see in Table 6, in both cases gated attention (further) dampens the outliers' magnitude to a

Table 6: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to OPT-125m. We report the causal language modeling perplexity (ppl. for short) on the English Wikipedia validation set for both the floating-point baseline and W8A8 quantized model. We also report the maximum ∥ x ∥ ∞ averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of an attention layer.

great extent, reduces the kurtosis, and yields models with significantly higher quantized performance, which is close to the original FP16 performance.

Table 7: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to ViT-S/16. We report the top-1 accuracy on ImageNet-1K validation set for floating-point baseline and W8A8 quantized model. We also report the maximum ∥ x ∥ ∞ averaged across the validation set, and kurtosis of x averaged across all layers, where x is the output of the attention layer.

Detailed results for ViT-S/16 are summarized in Table 7.

After our preliminary experiments on ViT, we noticed that distinct outliers already originate after the patch embeddings. Therefore, we experimented with adding the LayerNorm after the patch

embeddings (which was absent in the model definition, by default). As we can see in Table 6, together with this change, both of our proposed methods greatly dampens the outliers' magnitude, reduces the kurtosis, and yields models with significantly higher quantized performance, which is within 1% of the original FP32 accuracy.

B.5 The impact of clipped softmax hyperparameters ( γ and ζ ) on ViT

We investigate the effect of different values of the clipped softmax stretch parameters applied to the vision transformer and present the results in Table 8. To speed up training, for this experiment we trained ViT for 150 epochs instead of the usual 300 epochs. For this experiment, we did not apply LayerNorm after the patch embeddings.

We found similar observations compared to BERT. Specifically, most of the improvement happens when we use γ < 0 (clipping at zero) whereas using ζ > 1 (clipping at one) yields similar results to the vanilla softmax and combining both γ < 0 and ζ > 1 yields similar results compared to just clipping at zero.

B.6 Fine-tuning experiment

One of the drawbacks of our proposed framework is that it requires training from scratch, which could be expensive when applied to very large models. To address this, we explored whether fine-tuning using gated attention can still lead to improved performance and decreased outliers for larger models.

We used OPT-1.3B pre-trained checkpoint from HuggingFace and fine-tuned it on Bookcorpus + Wikipedia for 4000 steps with batch size 256, maximum sequence length 512, maximum learning rate 10 -5 , and linear LR schedule with 400 warmup steps. We use the same LR for both model parameters and gating module parameters. The rest of hyper-parameters are the same as for our pre-training setup. We adapted our gating approach as follows. We initialized bias as b init = 0 , which corresponds to the expected initial gating probability output of π init = 0 . 5 . We multiply the gating probability by 2 so that the expected gate output is 1 and we approximate the attention output of

the vanilla softmax at the start of fine-tuning. We add a small activation regularization term at the output of each FFN to further encourage the reduction in the magnitude of activations, as unlike when training from scratch outliers are already present in the pre-trained model and need to be suppressed.

As we can see from Table 9, fine-tuning with our proposed gated attention results in a better perplexity and also reduced maximum infinity norm and the average kurtosis compared to fine-tuning with vanilla softmax.

B.7 Low-bit quantization results

Table 10: A summary of results for our proposed methods applied to BERT-base and quantized to different bitwidthds for weights and activations (using the same PTQ setup as in all previous experiments). We report the masked language modeling perplexity on the English Wikipedia validation set.

Note that our proposed methods are not limited to 8-bit quantization only and in general can be combined with other more advanced quantization and weight compression methods, including [18, 35, 36, 45, 63, 67]. In Table 10, we show the results of our proposed methods applied to BERT-base and quantized to different bitwidths using our simple post-training quantization setup. Unless stated otherwise, for low-bit (<8-bit) weights and activations we use MSE range estimator as recommended by [2, 7] since it gives better results.

As we can see, in all cases both of our methods significantly improve the perplexity compared to the vanilla softmax pre-training. We also notice that generally the performance progressively degrades as we decrease the bitwidths, which is to be expected. Achieving good results with low-bit activation quantization in general is a challenging problem. Further, we notice that the perplexity of the vanilla model significantly improves whenever we consider a low-bit weight quantization with MSE ranges compared to the INT8 case. This can be explained by the fact that using MSE range estimation for weights leads to an implicit clipping of activations (in the same and all subsequent layers in the network), which happen to be of the right amount so that it doesn't hurt the perplexity. We found that by going from W8A8 to W6A8 the average kurtosis is reduced from 3406 ± 547 to 631 ± 94 and the maximum infinity norm is reduced from 577 ± 80 to 158 ± 40 . However, in all cases the resulting model still has significantly larger outliers and a worse performance than both of our proposed methods. Finally, as said before, if achieving good low-bit quantization performance is the goal, it is recommended to combine our methods with more advanced quantization techniques.

C Experimental details

C.1 BERT

Fine-tuning on MNLI dataset We use pre-trained checkpoint BERT-base-uncased (109M parameters) from HuggingFace repository. We follow standard fine-tuning practices from [14] and [65] Each data sequence is tokenized and truncated to the maximum sequence length of 128. Shorter sequences are padded to the same length of 128 using a special [PAD] token. We fine-tune for 3 epochs using Adam [29] with a batch size of 16 and no weight decay. The learning rate is initially set to its maximum value of of 2 · 10 -5 and is linearly decayed to zero by the end of fine-tuning.

Pre-training from scratch We follow closely the pre-training procedure from [14]. We concatenate, tokenize, and split the training set into sequences of length 128 (to speed up training and experimentation, we do not fine-tune on longer sequences of 512 ). We use the masked language modeling objective with the probability of masking p = 0 . 15 . We train with a batch size of 256

sequences for 10 6 steps, using AdamW optimizer [39] with the maximum learning rate of 10 -4 , learning rate warm up over the first 10 4 steps, following by a linear decay to zero by the end of training. We use L2 weight decay of 0 . 01 , L2 gradient norm clipping of 1 . 0 , and dropout probability of 0 . 1 on all layers. We also use FP16 mixed-precision from HuggingFace Accelerate library [20].

C.2 OPT pre-training

To speed up experimentation, we train OPT-125m sized model on the concatenation of Wikipedia and BookCorpus (same as BERT pre-training). We train with a batch size of 48 and 4 gradient accumulation steps (which results in the effective batch size of 192 ), so that we can perform pretraining on a single A100 80GB GPU. We concatenate, tokenize, and split the training set into sequences of length 512 and train for 125000 steps ( 500000 forward passes).

We use the rest of the hyper-parameters and follow pre-training practices from [74] and [65]. We initialize weights using a normal distribution with zero mean and a standard deviation of 0 . 006 . All bias terms are initialized to zero. We use AdamW optimizer with ( β 1 , β 2 ) = (0 . 9 , 0 . 95) . We use the linear learning rate schedule, warming up from 0 to the maximum value † of 4 · 10 -4 over the first 2000 steps, following by a linear decay to zero by the end of training. We use L2 weight decay of 0 . 1 , L2 gradient norm clipping of 1 . 0 , and dropout probability of 0 . 1 on all layers. We also use FP16 mixed-precision from HuggingFace Accelerate library [20].

C.3 ViT pre-training

We use the model definition for ViT-S/16 and the training pipeline from PyTorch Image models library [64]. All training is done on resolution 224 × 224 and 16 × 16 patches. For data augmentation, we use RandAugment [10], Mixup [73], CutMix [70], random image cropping [56], horizontal flip, label smoothing ε = 0 . 1 , color jitter 0 . 4 , and random (between bilinear and bicubic) interpolation during training.

We train with a batch size of 512 for 300 epochs, using AdamW optimizer and the L2 weight decay of 0 . 03 . We use the cosine learning rate schedule, warming up from 10 -6 to the maximum value of 10 -3 over the first 20 epochs, followed by a LR decay by a factor of 10 every 30 epochs, until it reaches the minimum value of 10 -5 .

C.4 Quantization settings

Weights In all cases, we use symmetric uniform quantization of weights. We use min-max weight quantization for all models except the OPT model, for which we found the MSE estimator to perform better in all cases.

Activations We adopt static range estimation approach, which determines quantization parameters for the network by passing a few batches of calibration data through the model before inference. Specifically, we use a running min-max estimator [32], which uses an exponential moving average of the min and max over multiple batches. In all cases, we use running min-max with 0 . 9 momentum over 16 batches randomly sampled from respective training sets.

For OPT model, we also experiment with using 99 . 99% and 99 . 999% percentiles instead of actual min and max. We select the best configuration for each experiment (including baseline), based on the model performance. In almost all cases, we found that setting activation quantization ranges using 99 . 999% percentiles gives the lowest W8A8 perplexity.

D Compute cost

We compare the runtime of our proposed methods in Table 11. As we can see, the clipped softmax is only marginally more expensive compared to using the vanilla softmax attention. The gated attention using the linear G adds the compute overhead between 3% and 8% , depending on the model. We found that adding weight decay on LayerNorm γ for OPT and adding the LayerNorm after the patch embeddings for ViT had a negligible effect on the runtime.

We estimated that the compute cost of producing the main results in the paper is about 320 GPU days (on A100) and the total cost of the project (including preliminary experiments and ablation studies) to be about 1400 GPU days.

SEP

[SEP]

you

[SEP]

[SEP]

the

[SEP]

[SEP]

[43s

[SEP]

[SEP]

[SEP]

ooats

SEP

[SEP]

[SEP]

[SEP]

[SEP]

[SEP]

never

[d3s]

[43s

[e3s

[SEP]

ooats

[SEP]

ooats

bar

gt

SEP

[SEP]

[SEP]

CLS

SEP

SEP]

[CLS!

88

SEP

[SEP]

[SEP]

员

SEP

SEP]

[SEP]

at

[SEP]

SEP)

SEP

SEP

SEP

SEP]

(h) Attention layer #8

SEP

[SEP]

CLS

SEP

[SEP]

[SEP]

9t

[SEP]

SEP

SEP]

[SEP]

[SEP]

0.8

0.6

0.4

0.2

[SEP)

[d3s]

8°0

0.6

SEP

0.2

[SEP]

[SEP]

1.0

0.8

0.6

0.4

0.2

SEP

[SEP]

[SEP]

[SEP]

8°0

0.6

838

[d3s]

[SEP]

[SEP

8'0

tdd

[SEP]

[SEP]

orke

8°0

0.6

0.2

[SEP]

[SEP]

0.0

0.4

0.2

8'0-

0.6

1.0

everyone

enough

rights

really

[CL5]

[SEP]

likes

new

nice

the

the

are

everyone

enough

rights

really

[CLS]

[SEP]

IRkes

new

nice

the

are

the

S'0-

0.0

1.0

0.5

[SEP]

[SEP]

0.0

benefits

newest

[SEP]

benefits

newest

[SEP]

1.0

8°0-

0.6

0.4

0.2

1.0

everyone

enough

rights

[CLS]

[SEP]

really

Eikes

new

nice

the

the

are

everyone

enough

rights

[SEP]

really

[CLS]

Iikes

new

nice

the

are

the

0.0

1.0

0.5

0.5

0.0

benefits

newest

benefits

newest

1.0

[SEP]

[SEP]

[SEP]

[SEP]

(b) Attention layer #11, Attention head #3, data sequence #1

(c) Attention layer #10, Attention head #3, data sequence #5

(e) Attention layer #10, Attention head #3, data sequence #7

(g) Attention layer #10, Attention head #12, data sequence #1

ghts

CW

[SEP]

[das]

[SEP]

SEPI

d35

[SEP]

[SEP]

SEP]

S$'0

0.50

[SEP]

[SEP]

[SEP]

year

ea

no

ISE

[SEP]

[SEP]

"ISEP

PSEPC

rea

[SEP]

[SEP]

WOSEP

everyone

benefits

enough

newest

rghts

[SEP]

[CL5]

really

[SEP]

Ies

new

nice

Bhe

are

the

0.25

0.30

0.20

0.15

[SEP]

[SEP]

0.0

8'0-

0.6

0.4

0.2

1.0

everyone

benefits

enough

newest

rights

really

[sT5]

[SEP]

[SEP]

Ikes

nice

new

the

are

auoAaAa

benefits

enough

newest

rights

[CL5]

[SEP]

really

[SEP]

likes

new

nice

the

the

are

0.0

1.0

0.5

0.5

-1.0

everyine

benefits

emough

newet

rights

[CLS]

[SEP]

really

[SEP]

IBes

new

nice

the

are

he

0.24

0.22

0.20

0.18

0.16

0.14

0.12

0.10

[SEP]

[SEP]

0.0

0.8

1.0

0.6

0.4

0.2

everyone

enough

benefits

newest

rights

really

[CLS]

[SEP]

[SEP]

Ikes

new

nice

the

the

are

2

everyone

benefits

enough

newest

rights

[CL5]

[SEP]

really

[SEP]

likes

new

nice

the

are

the

0.0

0.5

1.0

0.5

-1.0

(h) Attention layer #11, Attention head #12, data sequence #1

(a)

(b)

(c)

(d)

(e)

non-outlier

outlier

non-outlier

outlier

non-outlier

outlier

non-outlier outlier

non-outlier

outlier

non-outlieroutlier

non-outlier

outlier

non-outlier

outlier

(a)

(b)

(c)

(d)

(e)

0.175

0.150

0.125

0.100

0.075

0.050

0.025

non-outlier

outlier

0.200

0.175

0.150

0.125

0.100

0.075

0.050

0.025

non-outlier

outlier

non-outlier

outlier

nnn-nutlier

ntlier

0.200

0.175

0.150

0.125

0.100

0.075

0.050

0.025

non-outlier

outlier

0.20

non-outlier

outlier

.15

non-outlier

outlier

Peer reviews:
comments: SUMMARY:
This paper has thoroughly analyzed the activation outlier problem during quantizing of the transformer model, and further indicates the limitation of existing works. Based on the comprehensive studies, the author proposes two components, namely clipped softmax and gated attention, for regularizing the activation outlier during training. Sufficient experiment results demonstrate the effectiveness and efficiency of the proposed method.

STRENGTHS:
$\cdot$ The proposed components, namely clipped softmax and gated attention are simple but reasonable. Further experiments also support their effectiveness.

$\cdot$ The observation is solid and further analysis is practical. The experiments include different structures of the transformer model in various tasks.

$\cdot$ I appreciate the comprehensive studies conducted by the authors. The experiment result of computing cost is convincing.

$\cdot$ This paper is well-organized and clear to understand. And The authors provide detailed information on related methods, making it easy for readers to understand the progress of the field.

WEAKNESSES:
$\cdot$ The quantization performance of the proposed method at a lower bit (e.g. 4W4A) is worth looking forward to.

QUESTIONS:
$\cdot$ The  avg. kurtosis of OPT model with clipped softmax in tab. 2 is strange, what is the possible reason?

LIMITATIONS:
$\cdot$ The author has claimed in L290-296. I encourage the authors to finish the experiment in the larger model.

$\cdot$ I encourage the authors to release the code for reproducing.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two architectural modifications that mitigate the activation outlier problem that makes transformers challenging to quantize. Particularly, the authors conduct a comprehensive analysis of the underlying causes of outlier issues in various pre-trained Transformer models across different tasks and datasets. They identify specific architectural limitations that lead to large outliers during the pre-training phase. To tackle this challenge, the authors propose two solutions: 1/ clipped softmax and 2/ gated attention that can effectively address the outlier problem without sacrificing model performance when compared to vanilla Transformer models across various use cases. Furthermore, the suggested modifications result in smaller outlier values and enhance the quantization performance of the Transformer models, making them more robust to quantization.\n\nSTRENGTHS:\n1. The paper presents strong motivations and novel approaches to address outlier issues that distinguish it from prior works. Unlike existing methods that focus on circumventing the impact through post-training techniques, this paper proposes a fundamental training-time solution.\n2. The paper proposes a simple yet highly effective methodology, which directly addresses the fundamental issues and observations.\n\nWEAKNESSES:\n1. Evaluating the proposed methodology by comparing it solely to the naive application of (vanilla) PTQ on the plain Transformer architecture might be somewhat unfair and not informative enough. Various outlier suppression methods have been proposed [] as post-training solutions, and thus it would be better to include a comparison with these methods. Given that the proposed methodology requires from-scratch training, its practical value will be justified if it outperforms post-training methods that require no additional training.\n\n2. The paper introduces two distinct solutions; however, they are two orthogonal/competing methodologies that tackle the same problem. The paper lacks clarity regarding the specific circumstances under which each solution should be employed. That said, how can we decide which scheme to use when training a new model on a new dataset without having to try both and choose the better one?\n\n3. As also stated in the paper, the evaluations have only been conducted on smaller-scale models (100M parameters). While the proposed architectural and train-time modifications show promising results that match/outperform the vanilla Transformer performance, it is unclear if this trend can persist on a larger scale (hundreds of millions to billions of parameters). It would be more informative if the authors can provide across different model scale regimes. Moreover, since the proposed solution is a pre-training methodology, it requires training from scratch and additional risk/cost related to extra hyperparameter tuning, which can potentially limit their practical application.\n\n\n[1] Wei, Xiuying, et al. \"Outlier suppression: Pushing the limit of low-bit transformer language models.\" Advances in Neural Information Processing Systems 35 (2022): 17402-17414.\n\n[2] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" arXiv preprint arXiv:2211.10438 (2022).\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nPlease see the weakness section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors show that strong outliers are related to very specific behavior of attention heads that try to learn a “no-op” or just a partial update of the residual. To reduce outliers, the authors propose two simple (independent) modifications to the attention mechanism. This enables them to quantize transformers to full INT8 quantization of the activations without any additional effort.\n\nSTRENGTHS:\n1. The authors provide a detailed analysis of the outliers in transformers, which well clarifies their research motivation and provides insights.\n2. The analysis of \"strong outliers are related to attention heads try to learn no-op\" is interesting.\n\nWEAKNESSES:\n1. The authors do not give detailed data distribution after training with the proposed method. I would expect a reduction in the magnitude and number of outliers than in the performance of INT8 quantization that validates the main target of the proposed technique.\n2. The authors mention that it is limited to small models only (125M, which is too small), but quantizing small models with less than 2.7B parameters to INT8 is less challenging compared with large models  [1]. There exists concern is the generalization ability of the proposed method on large-scale models. Analysis or experiments that demonstrate the effectiveness of proposed technique on larger scale model is recommended. \n\n[1] LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NIPS'22\n\nQUESTIONS:\n1. Could the proposed method be applied to lower bitwidths beyond INT8?  (e.g., 4/6 bit)\n2. Does the proposed method require a full retraining of the FP model? When applied to larger models, the training cost could be excessive.\n\nLIMITATIONS:\nThe authors do not include the limitations and potential negative societal impact in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper analyzes the activation outlier problem. It is shown that the outliers are related to the behavior of transformer networks trying to learn not to update residuals (no-op). To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger. To solve this problem, the authors propose clipped softmax method and gated attention method, which can introduce exact zeros for softmax, thus the outliers can be removed.

STRENGTHS:
The paper explains why outliers exist in transformers. The analysis and visualization are convincing.

Two independent methods are proposed (clipped softmax and gated attention) to solve the outlier problem of transformers, so that models can be quantized easily.

Experiments on language models (BERT, OPT) and vision transformers are provided.

WEAKNESSES:
No results on large scale models and datasets are provided.

QUESTIONS:
How to explain the clipped softmax method no better than vanilla network on OPT?

Can you provide the visualization (Figure 2) after using the proposed methods?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07