SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
{ybond, markusn, tijmen}@qti.qualcomm.com
Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op" or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - clipped softmax and gated attention. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers. Our source code is available at https://github.com/qualcomm-ai-research/outlier-free-transformers.
Quantization has been one of the most impactful ways to reduce the computational complexity of transformer networks. Previous work has shown that quantizing networks to 4-bit weights is possible without losing too much accuracy [66, 69]. Some research even shows 4-bit weights might be optimal when trading off model size and bit-width [12].
However, quantizing transformers is not always trivial. When quantizing the activations of a transformer, significant problems arise with outliers in specific layers. This has been noted by several researchers that suggest fixes to transformers after training to ameliorate their effect [13, 67]. These methods are frequently tedious and either require retraining the network, require implementing specific hardware for input-channel quantization [13] or require parts of the activations to still be in higher bit-widths, reducing the effectiveness of the activation quantization [67].
In this paper, we set out to solve the transformer outlier problem entirely by changing the architecture of the network itself. We hope to make transformers easy to quantize from the get-go without needing
*Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
any post-processing. To do so, we thoroughly analyze why these outliers appear. Previous work has found the existence of these outliers [4, 13], but in our work, we come to a fuller understanding of these outlying values. We find that the outliers occur because attention heads are trying not to update the hidden state, and in the process, strong outliers appear due to the softmax function. This happens for language and vision transformers and different specific transformer architectures. This understanding is the foundation for two new tweaks we suggest to transformer architectures that can remove the problem of the outliers entirely.
In this section, we briefly cover the basics of neural network quantization and discuss why modern transformers are difficult to quantize.
Quantization One of the most powerful ways to decrease the computational time and memory consumption of neural networks is quantization, which uses low-bit representations for the weights and activation tensors. On top of that, using low-bit fixed-point representations, such as INT8, one can further reduce energy consumption since the fixed-point operations are more efficient than their floating-point counterparts [23, 59].
We simulate the quantization process in floating-point according to Jacob et al. [26]. We use the following definition of the quantization function:
$$
\widehat {\mathbf {x}} := q (\mathbf {x}; s, z, b) = s \cdot \left(\operatorname {c l i p} \left(\left\lfloor \frac {\mathbf {x}}{s} \right] + z; 0, 2 ^ {b} - 1\right) - z\right), \tag {1}
$$
where  $\mathbf{x}$  denotes the quantizer input (i.e., network weights or activations),  $s \in \mathbb{R}_+$  the scale factor or the step-size,  $z \in \mathbb{Z}$  the zero point, and  $b \in \mathbb{N}$  the bitwidth.  $\lfloor \cdot \rfloor$  denotes the round-to-nearest-integer operator. This quantization scheme is called uniform affine or asymmetric quantization [24, 32, 76] and it is one of the most commonly used quantization schemes because it allows for efficient implementation of fixed-point arithmetic. In the case of symmetric quantization, we restrict the quantization grid to be symmetric around  $z = 0$ .
In this work, we focus on post-training quantization (PTQ) methods, which take a pre-trained FP32 network and convert it directly into a fixed-point network without the need for the original training pipeline [2, 5, 7, 25, 32, 35, 41, 43, 44, 75]. These methods require either no data or only a small calibration dataset and are easier to use compared to quantization-aware training (QAT, Bhalgat et al. 3, Esser et al. 16, Gupta et al. 21, Jacob et al. 26, Krishnamoorthi 32) methods that have you train the entire network for more epochs. For more details on neural network quantization, we refer the reader to [19, 46].
Outliers in Transformers Multiple studies have shown that modern transformer-based language models tend to learn outliers in weights and activations [4, 13, 31]. These outliers are present only in a small fixed set of embedding dimensions, but they appear regularly and consistently across multiple layers and data sequences. It was also shown that those outliers play a crucial role in the model predictions and clipping them or by setting to zero the corresponding parameters significantly degrades the model task performance [31, 49]. The strongest in magnitude outliers typically appear at the output of the feed-forward network, FFN, although Dettmers et al. [13] showed that for big enough transformer-based language models they start appearing after every linear layer, including query, key, and value projection layers. This phenomenon holds for many tasks, training objectives and models (both encoder and decoder transformers), including BERT [14], RoBERTa [37], DistilBERT [53], MobileBERT [55], ELECTRA [9], BART [33], XLNet [68], GPT-2 [50], and OPT [74].
Because of these strong outliers, applying per-tensor PTQ for the FFN's output and the residual sum will likely cause a notable error because of the following trade-off between the range and the precision. On the one hand, using a large quantization range for small-ranged values leads to a loss in representation (high rounding error). On the other hand, a small quantization range for large values leads to a very high clipping error. For the case of significant transformer outliers, frequently, no good trade-off can be found between the rounding and clipping error, resulting in an overall high error.
There have been numerous attempts to fix the issue of transformer quantization [4, 12, 13, 17, 27, 28, 51, 54, 62, 63, 69, 71]. Most of these approaches resort to finer quantization granularity (row-wise,
(a) FFN output in layer #10
Figure 1: Histograms of outlier counts vs. token positions (blue) and hidden dimensions (green), recorded from the MNLI-m validation set on BERT-base. We use zero-based indexing for dimensions.
(b) FFN output in layer #11
channel-wise, group-wise weight and activation quantization), use higher bitwidth and/or different numeric format to represent those outliers better or require extra fine-tuning (in the form of QAT and/or knowledge distillation). In other words, they adapt quantization to work with outliers, which often comes at the expense of general applicability or extra inference overhead.
In contrast, in this work, we want to address the root cause of the problem and understand why outliers are learned in the first place and suggest a new pre-training protocol that significantly reduces the magnitude of outliers yielding way more quantization-friendly models that can be effortlessly quantized using PTQ without strong degradation of performance.
3 Outlier analysis
Outliers in BERT models In Section 2 we discussed that outliers are present only in a few designated embedding dimensions but they appear regularly and consistently across multiple layers and data sequences. We also discussed that the strongest magnitude outliers in BERT typically appear at the output of FFN in the last encoder layers.
We start by taking the pre-trained BERT-base-uncased checkpoint from HuggingFace [65] and fine-tune it on MNLI dataset from the well-known GLUE benchmark [61] (see experimental details in C.1). To identify the outlier dimensions, we pass the MNLI-m validation set through the network and record all outliers<sup>1</sup> at the FFN output in layers #10 and #11<sup>2</sup>. As we can see in Figure 1, there are indeed only a few hidden dimensions where outliers ever occur. We also notice that the majority of outliers ( $>97\%$ ) correlate with the position of delimiter tokens - [SEP], ".", and ".".
To better understand the role of those outliers, we analyze the attention patterns of the corresponding attention heads. BERT-base uses multi-head attention with  $n_{\mathrm{heads}} = 12$  and each head operating on a consecutive subset of  $d_{\mathrm{head}} = 64$  features. Therefore, the hidden dimension #180, which happens to have the highest outlier count in both layers #10 and #11, corresponds to attention head #3. In Figure 2 (and more examples in Appendix A.1) we show examples of the attention matrices, values and their product for that head.
A common pattern we found is that the attention head assigns almost all of its probability mass to [SEP] tokens, and other less informative tokens like dots/commas, while these tokens also have small values in  $V$  associated with those tokens. This results in a small magnitude product between the two (see Figure 2a). This effectively corresponds to a (soft) no-update of the hidden representation, where only small noise is added after the residual. In other cases (Figure 2b and 2c), we observe that a significant portion of attention probability is still spent on delimiter tokens. However, by allocating some of the probability mass on other tokens (together with the small values for the delimiter tokens), this results in a (soft) selective update of the hidden representation.
These patterns in self-attention seem to be a learned "workaround" for the limitations of having the softmax and the residual connections in cases where the attention head does not want to update the representation of some or all of the tokens. These observations are in line with Clark et al. [8], Kovaleva et al. [30] that also argued that attending exclusively or almost exclusively to delimiter tokens such as [SEP], periods/commas acts as a "no-op" when the attention head's function is not applicable.
<sup>1</sup>We follow Bondarenko et al. [4] and consider outliers as values that exceed 6 standard deviations from the mean of the corresponding activation tensor.
2We use 1-based indexing for encoder layers and attention heads throughout the paper.
(a) Attention layer #11, data sequence #1
(b) Attention layer #11, data sequence #5
Figure 2: Visualization of the patterns in the self-attention, specifically the attention probabilities, values, and their product (left, middle and right columns, respectively), in attention head #3 for BERT-base, computed on several data sequences from MNLI-m validation set.
(c) Attention layer #10, data sequence #5
(d)
Figure 3: A summary of our outlier analysis for ViT demonstrated on a random image from ImageNet validation set. (a) An input image. (b) Outliers in the output of layer #11. (c) Cumulative attention weight spent on every patch (matrix of attention probabilities summed over rows) in the attention head #1, layer #12. (d) A corresponding matrix of attention probabilities. (e) An average magnitude of values for outlier and non-outlier patches.
Outliers in ViT We conduct a similar analysis for Vision transformer [15] trained on ImageNet [52]. For this study, we use a pre-trained checkpoint following our experimental setup from Section 5.
We highlight our findings in Figure 3 and provide more examples in Appendix A.2. Our analysis shows many similarities to the BERT case. Instead of delimiter tokens, the majority of outliers seem to correlate with some random uninformative patches (e.g., in the background). We also see that the corresponding attention head in the next layer allocates the majority of attention probabilities to the same patches. Finally, those outlier patches on average have a distinctly smaller magnitude of values compared to non-outlier ones, leading to similar no-update behavior. The fact that those values are not as close to zero as it was in the BERT case might be related to the smaller model capacity<sup>3</sup>, or a relatively shorter training procedure.
Hypothesis Based on these observations, we pose the following hypothesis on how this behavior of attention heads is related to outliers:
1. In order for an attention block to not update a representation of a token on the residual, some attention heads want to allocate most of their attention probability mass to some fixed and common set of tokens that have a low information content (e.g., delimiter tokens or background patches) that can be learned to have a small value function output.
3We use ViT/S-16 configuration that has only 22M parameters.
2. From the definition of the softmax function<sup>4</sup>, it is easy to see that this would require an input of the softmax to have a relatively big dynamic range (Figure 4, 1). In fact, in the limit case where softmax is exactly zero, this would require an infinite dynamic range:
$$
\operatorname {s o f t m a x} (\mathbf {x}) _ {i} = 0 \quad \Leftrightarrow \quad \exists j \neq i, \mathbf {x} _ {j} - \mathbf {x} _ {i} = + \infty \tag {2}
$$
3. Since Layer Normalization ([1], 2) normalizes the outliers, the magnitude of the FFN output in the previous layer (3) has to be very high to still produce a sufficiently big dynamic range after the LayerNorm. Note, that this is also applicable for the transformer models with LayerNorm applied prior to the self-attention or linear transformations instead, a variant adopted by GPT, OPT, and many vision transformers [15, 38, 57, 58].
4. Finally, as softmax will never output exact zeros, it will always back-propagate a gradient signal to grow bigger outliers<sup>5</sup>. The outliers will thus tend to become stronger in magnitude, the longer the network is trained.
In this section, we introduce our proposed modifications for the softmax attention mechanism. Based on our insights from Section 3, the core idea of these modifications is to grant the model the ability to produce very small the magnitude (or even exact zeros) output of attention function, without producing outliers.
Recall that the self-attention [60] is defined as follows:
$$
\operatorname {A t t e n t i o n} (\mathbf {x}) := \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} (\mathbf {x}) \boldsymbol {K} (\mathbf {x}) ^ {T}}{\sqrt {d _ {\text {h e a d}}}}\right) \boldsymbol {V} (\mathbf {x}) \tag {3}
$$
where  $Q, K$  and  $V$  are learnable linear projections of the input  $\mathbf{x}$ . Most modern transformer models employ a multi-headed variant of self-attention, where  $d_{\mathrm{model}}$  features are partitioned into  $n_{\mathrm{heads}}$  groups of  $d_{\mathrm{head}}$  features, and the final output is the concatenation of the outputs of (3) applied to each group.
4.1 Clipped softmax
First, we propose to replace softmax function in (3) with the following clipped softmax:
$$
\operatorname {c l i p p e d \_ s o f t m a x} (\mathbf {x}; \zeta , \gamma) :=
$$
$$
\operatorname {c l i p} \left(\left(\zeta - \gamma\right) \cdot \operatorname {s o f t m a x} (\mathbf {x}) + \gamma , 0, 1\right). \tag {4}
$$
Here  $\mathbf{x}$  is the input and  $\zeta \geq 1, \gamma \leq 0$  are the stretch factors which are hyper-parameters of the method. This formulation was proposed before in [40] in the context of binary stochastic gates. We can view (4) as stretching the output of the softmax from  $(0,1)$  to  $(\gamma, \zeta)$  and then clipping back to  $(0,1)$  so that we can represent exact zeros if  $\gamma < 0$  and exact ones if  $\zeta > 1$ . Specifically, the values of the softmax larger than  $\frac{1 - \gamma}{\zeta - \gamma}$  are rounded to one whereas values smaller than  $\frac{-\gamma}{\zeta - \gamma}$  are rounded to zero.
With this drop-in replacement, we can achieve exact zeros (and ones) with a finite range for the softmax input. In addition to that, whenever values are clipped they will not give a gradient, preventing the outliers to grow further.
Figure 4: A schematic illustration of the attention layer in BERT. Hidden activation tensor is denoted by  $\mathbf{x}$ .  $\oplus$  is an element-wise addition. A problematic output of the FFN that generates largest in magnitude outliers is highlighted in red. Notice how those outliers in the previous layer influence the behavior in the attention mechanism in the next layer.
${}^{4}$  softmax  ${\left( \mathbf{x}\right) }_{i} = \frac{\exp\left( {\mathbf{x}}_{i}\right) }{\mathop{\sum }\limits_{{j = 1}}^{d}\exp \left( {\mathbf{x}}_{j}\right) }$
Let  $\mathbf{y} = \mathrm{softmax}(\mathbf{x})$ . Then  $\frac{\partial \mathbf{y}_i}{\partial \mathbf{x}_j} \neq 0 \forall i, j$ .
4.2 Gated attention
An alternative way of architecting the model to have a small attention output without outliers is to equip it with an explicit conditional gating mechanism, as shown in Figure 5. The idea is that the model can use the gating to either keep or nullify the update to the representation of certain tokens and not rely on the attention probabilities and values to achieve the same outcome.
Specifically, we propose the following modification to the attention function:
$$
\operatorname {G a t e d} _ {\text {a t t e n t i o n}} (\mathbf {x}) := \operatorname {s i g m o i d} \left(\boldsymbol {G} (\mathbf {x})\right) \odot \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} (\mathbf {x}) \boldsymbol {K} (\mathbf {x}) ^ {T}}{\sqrt {d _ {\text {h e a d}}}}\right) \boldsymbol {V} (\mathbf {x}). \tag {5}
$$
Here  $G$  is the gating function,  $\odot$  is an element-wise multiplication across the token axis and everything else remains the same as in (3). The gating function  $G$  is parameterized by a small neural network that is learned jointly with the rest of the model. We replace the attention formulation with the proposed variant in every layer on the transformer network.
Gating module design Recall that the input to the attention layer  $\mathbf{x}$  has shape  $(T, d_{\mathrm{model}})$  that is reshaped into  $(n_{\mathrm{heads}}, T, d_{\mathrm{head}})$  for the multi-headed self-attention, where  $T$  is the sequence length. We chose to define the gating function on a per-head basis. For each head  $i \in \{1, \dots, n_{\mathrm{heads}}\}$ , we specify  $\pmb{G}_i: \mathbb{R}^{d_{\mathrm{head}}} \to \mathbb{R}$  and the output of the gating module is  $\pmb{\pi}_i \in \mathbb{R}^T$  that is computed as follows:
$$
\widehat {\boldsymbol {\pi}} _ {i, t} = \boldsymbol {G} _ {i} \left(\mathbf {x} _ {i, t,:}\right) \forall t \in \{1, \dots , T \} \tag {6}
$$
$$
\pi_ {i,:} = \operatorname {s i g m o i d} \left(\widehat {\pi} _ {i,:}\right), \tag {7}
$$
note that gating modules are shared between different token positions but not shared across attention heads.
We want our gating module to be as lightweight as possible. To start with, we experiment with  $G_{i}$ 's parameterized by a single linear layer. This gives us a gating module that is computationally inexpensive and has a memory overhead of just  $n_{\mathrm{heads}} \cdot (d_{\mathrm{head}} + 1) \sim d_{\mathrm{model}}$  extra parameters (which is equivalent to 1 extra token) per attention layer<sup>6</sup>. We also investigate the effect of using several other gating functions in Appendix B.1.
Figure 5: A schematic illustration of our proposed gated attention.
In this section, we evaluate the proposed modifications to self-attention on several language models (BERT, OPT) and the vision transformers (ViT). We first test the different hyperparameters for the methods and provide insight into how they work. Then we set out to test our method in terms of accuracy, and the difference in quantization improvement after training. All detailed hyperparameters of our experiments are in Appendix C.
BERT We experiment with BERT-base-uncased (109M parameters) pre-training using the masked language modeling (MLM) objective. Following [14], we use the concatenation of the training sets of BookCorpus [77] and English Wikipedia<sup>7</sup>. We implement our methods in PyTorch [48] and use training and evaluation pipelines from HuggingFace libraries [20, 34, 65]. We follow closely the pre-training procedure from [14]. To speed up training and experimentation, we train with a maximum sequence length of 128 for the whole duration of the training. We evaluate on Wikipedia validation set and report the MLM perplexity.
OPT We experiment with a 125M sized variant of OPT [74] pre-training using the causal language modeling (CLM) objective. Due to compute constraints, we train the model on the same dataset that was used for BERT pre-training (BookCorpus + Wikipedia) with a maximum sequence length of 512
For instance, in case of BERT-base, this amounts to less than  $0.009\%$  of the total model size.
<sup>7</sup>Specifically, we use the English subset of Wiki-40b, https://huggingface.co/datasets/wiki40b, that contains cleaned-up text of English Wikipedia and training/validation splits.
γ	ζ	FP16 ppl.↓	Max inf. norm	Avg. kurtosis	W8A8 ppl.↓
0 (= Vanilla)	4.49±0.01	735±55	3076±262	1294±1046
0	4.48±0.01	715±335	2159±238	451±57
0	4.49±0.00	741±66	1707±1249	1469±646
-0.003	4.46±0.00	688±64	2149±110	636±566
-0.03	4.41±0.01	20±1	80±6	4.55±0.01
-0.003	4.47±0.00	683±23	2494±1205	268±120
-0.03	4.43±0.03	22±3	73±8	4.56±0.05
Table 1: The impact of clipped softmax hyperparameters on BERT-base.
and batch size of 192. Similar to our BERT experiments, we use training and evaluation pipelines from HuggingFace libraries. We evaluate on Wikipedia validation set and report the CLM perplexity.
ViT Finally, we explore the effectiveness of proposed techniques on vision transformer [15] (ViTS/16 configuration, 22M parameters) trained on ImageNet-1K [11, 52]. For these experiments, we adopt the training and validation pipelines from PyTorch Image models library [64]. We report top-1 accuracy on the validation set of ImageNet.
Quantization setup In all experiments, after the model is trained, we apply 8-bit PTQ. We use uniform affine quantization – symmetric weights, asymmetric activations – with the static activation range setting, as discussed in Section 2. We quantize all weights and activations (both input and output), except the final linear layer for BERT and OPT models. We explore several choices of range estimation (see Appendix C.4) and report the best configuration for each experiment, based on the model performance. We repeat each PTQ experiment 3 times with different random seeds<sup>8</sup> and report mean and standard deviation for accuracy/perplexity.