SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We train each network two times with different random seeds and report mean and standard deviation. To assess the amount of outliers in the trained model, we use two metrics: the maximum  $\| \mathbf{x}\|_{\infty}$  averaged across the validation set, and kurtosis of  $\mathbf{x}$  averaged across all layers, where  $\mathbf{x}$  is the output of an attention layer. These metrics have been shown to correlate well with the model quantizability [4, 6].
5.1 The impact of clipped softmax hyperparameters  $(\gamma$  and  $\zeta)$
We investigate the effect of different values of the clipped softmax stretch parameters and present the results in Table 1. We can see that most of the improvement happens when we use  $\gamma < 0$  (clipping at zero). For instance, using the value of  $\gamma = -0.03$  leads to a significantly smaller infinity norm, kurtosis, and quantized model perplexity, compared to the baseline. It is also clear that in the limit  $|\gamma| \to 0$  we approach the vanilla softmax attention. Using  $\zeta > 1$  (clipping at one) yields similar results to the vanilla softmax. Finally, when we combine both  $\gamma < 0$  and  $\zeta > 1$ , for which the results seem similar to just clipping at 0. We, therefore, conclude that for dampening outliers, only the lower-range clipping allows exact zeros matter. Going forward we use only  $\gamma < 0$  and in Appendix B.5 we confirm that  $\zeta > 1$  is not required for ViT.
These observations are in line with our hypothesis that by giving the model the mechanism for representing exact zeros in the attention, we don't need to learn the strong outliers.
5.2 Clipped softmax  $\gamma$  vs. sequence length
As having an extra hyper-parameter that needs to be tuned per model or setup is generally not desirable, we study the sensitivity of the stretch factor  $\gamma$  and its relation with the sequence length  $T$ . Recall that the matrix of attention probabilities  $\pmb{P}$  has dimensions  $T\times T$  and each row sums up to one. Because of that, the average value in  $\pmb{P}$  is  $1 / T$ . It is reasonable to assume that if we define
Different random subsets of training data are used for quantizer range estimation.
(a) Relative FP16 log-perplexity
(b) Maximum infinity norm
Figure 6: The performance of clipped softmax using  $\gamma = -\alpha /T$  parameterization on BERT-6L. (a) Relative (compared to vanilla softmax pre-training) FP16 log-perplexity  $\uparrow$  on Wikitext validation set. (b) Maximum infinity norm of the attention layer output (note the logarithmic y-axis).
(a) BERT-6L
(b) ViT
Figure 7: The performance of Linear gated attention using different bias initialization settings.
$\gamma \coloneqq -\frac{\alpha}{T}$ , where  $\alpha > 0$  is a new hyperparameter, there might be a set or a range of values of  $\alpha$  that works well across different sequence lengths.
To study this, we train a 6-layer variant of BERT-base (BERT-6L) for 500000 steps on WikiText-103 [42] with a batch size of 128 with several values of maximum sequence lengths  $T \in \{32, 64, 128, 192, 256\}$  and values of  $\alpha \in \{1/4, 1/2, 1, 2, 4, 8\}$ . As we can see from Figure 6, using a clipped softmax with  $\alpha \in [2, 4]$  significantly dampens the magnitude of outliers while maintaining good FP16 perplexity across all explored sequence lengths.
5.3 The impact of bias initialization in gated attention
In all our gated attention experiments, we randomly initialize the weights of  $G$ , following [22]. By initializing the bias to a specific value, however, we can set gates to be more open or more closed initially. More open at the start means we initialize closer to the original network, but given the exponential nature of the gate it might take many iterations for the gate to learn to close. Similarly, if the gates are all closed at the start, we deviate too far from the original model training, causing a potential decrease in performance. Assuming Linear  $G_{i}$ 's with small initial weights, if we set the bias to the value of  $b_{\mathrm{init}}$ , then  $G_{i}(\cdot) \approx b_{\mathrm{init}}$  and  $\pi_i(\cdot) = \mathrm{sigmoid}(G_i(\cdot)) \approx \mathrm{sigmoid}(b_{\mathrm{init}}) =: \pi_{\mathrm{init}}$ , at the start of training.
We study the effect of different values of  $b_{\mathrm{init}}$  for Linear gated attention on BERT-6L and ViT. We set the bias for all  $G_{i}$ 's to the same value of  $b_{\mathrm{init}}$ . For BERT-6L, we use the same setup as in Section 5.2, with a fixed sequence length of 128. For ViT, we use the main setup, except we train it for 150 epochs instead of 300.
In Figure 7 we see in both BERT and ViT cases that using bias with very high  $\pi_{\mathrm{init}}$  generally performs similarly to the vanilla attention (comparable floating-point performance but strong outliers and poor quantized performance) while setting bias to have very low  $\pi_{\mathrm{init}}$  dampens outliers quite well but leads to strong degradation in the floating-point and quantized performance. The reasonable ranges of  $\pi_{\mathrm{init}}$  seems to be around [0.25, 0.9] for BERT and [0.1, 0.5] for ViT. The wide range indicates the relative robustness of our method to this hyperparameter.
Model	Method	FP16/32	Max inf. norm	Avg. kurtosis	W8A8
BERT (ppl.↓)	Vanilla	4.49±0.01	735±55	3076±262	1294±1046
Clipped softmax	4.39±0.00	21.5±1.5	80±6	4.52±0.01
Gated attention	4.45±0.03	39.2±26.0	201±181	4.65±0.04
OPT (ppl.↓)	Vanilla	15.84±0.05	340±47	1778±444	21.18±1.89
Clipped softmax	16.29±0.07	63.2±8.8	19728±7480	37.20±2.40
Gated attention	15.55±0.05	8.7±0.6	18.9±0.9	16.02±0.07
ViT (acc.↑)	Vanilla	80.75±0.10	359±81	1018±471	69.24±6.93
Clipped softmax	80.89±0.13	73.7±14.9	22.9±1.6	79.77±0.25
Gated attention	81.01±0.06	79.8±0.5	19.9±0.3	79.82±0.11
Table 2: A summary of results for our proposed methods applied on BERT, OPT-125m, and ViT.
Model	Method	FP16	Max inf. norm	Avg. kurtosis	W8A8
OPT-350m (ppl.↓)	Vanilla	13.19	253	2689	37.52±3.84
Gated attention	13.01	65.4	261	14.42±0.06
OPT-1.3B (ppl.↓)	Vanilla	12.13	428	2756	989.6±175
Gated attention	12.21	67.2	444	29.95±0.42
Table 3: The performance of gated attention applied on bigger variants of OPT model.
5.4 Main results
We summarize our main set of results in Table 2. As we can see, in almost all cases, both of our proposed techniques dampen the outliers' magnitude to a great extent, reduce the kurtosis, and yield models with significantly higher quantized performance, which is close to the original FP16/32 performance. In addition to that, for each model, at least one of our methods also improves the floating-point task performance. We hypothesize this is because the network is helped with learning the "no-op" updates more easily. However, we are cautious about the improved performance as this is not consistent across all hyper-parameters and it is unclear if it generalizes to more architectures and larger models.
The only case where our method failed to perform well was the clipped softmax applied to OPT. At the moment, we do not have an explanation of why this is the case and leave it for future work. We list selected hyper-parameters and show extended results in Appendix B. We also show the results of our proposed methods quantized to lower bitwidths in Appendix B.7.
Results for bigger models We study the question of scalability of our methods to larger models. In Table 3 we show the gated attention results for  $350\mathrm{m}$  and 1.3B variants of OPT. Due to compute constraints, we trained networks for  $10^{5}$  steps with batch size of 256 and the rest is the same as in our main pre-training setup. As we can see, our proposed gated attention is also very effective at dampening the outliers and significantly improving the quantized model performance when applied to bigger models. We further study in Appendix B.6 how gated attention can decrease outliers when fine-tuning bigger pre-trained models with outliers.
In Figure 8 we compare the learned attention patterns using vanilla softmax and our proposed methods (more examples in Appendix A.1). As we can see, both methods can represent a partial/soft no-op behavior, but in case of our methods this does not require strong outliers elsewhere in the network. Note that we found similar patterns in multiple attention heads, but the exact head indices where we observed such patterns depend on random initialization. In the case of clipped softmax, the attention probabilities are generally more diffused and smaller in magnitude (which comes from the stretching and clipping). In the case of gated attention, the output of the softmax is significantly different since the update of the hidden representation is now further modulated by gating probabilities.
(a) Vanilla softmax (Attention layer #11, head #3)
(b) Clipped softmax (Attention layer #11, head #8)
Figure 8: Visualization of the self-attention patterns for BERT-base trained using vanilla and our proposed techniques, computed on data sequence #5 from MNLI-m validation set. (a), (b): attention probabilities, values, and their product. (c): gating probabilities  $\pi =$  sigmoid  $(G(x))$ , attention probabilities (output of softmax), values, and their combined product.
(c) Gated attention (Attention layer #11, head #5)
"No-op" behavior It is interesting to note that the identified "no-op" behavior is likely not limited to transformers and that convolutional architectures likely learn something similar. We also see that despite the network trying to learn a full "no-op", still a small amount of noise is added to each residual, which may constitute a form of network regularization. Investigating this further might give us a clue as to why neural networks generalize despite being significantly overparametrized if many parameters are rendered unused by not updating the representation in later layers [72].
Limitations We have not studied the effect of our method on large-scale transformers, as it would require training very expensive models from scratch. Given the fundamental understanding of the issue underlying our solutions, we expect the same effect on large-scale models. We show a very small improvement in FP16/FP32 performance due to our methods, but we do not deem our results exhaustive enough to claim that this will hold in general. Lastly, our methods do have a hyperparameter each, although we show that both methods are relatively robust to its hyperparameter, having one is never optimal.
Impact As our methods help transformers to be more efficient, we expect only positive outcomes of our work. Making neural networks more efficient will help with their high power consumption at inference. It further helps to move inference from the cloud to edge devices which can overcome potential privacy concerns. We cannot fathom any negative impact from our work that is not severely construed.
We have thoroughly analyzed the activation outlier problem that makes transformers difficult to quantize. We showed that transformer networks try to learn not to update residuals and that by doing so, through the combination of the softmax, residual connections and LayerNorm, significant outliers appear in transformers. Based on this insight, we proposed two methods to address this at the core - clipped softmax and gated attention. These structural changes to transformers give similar, if not better, floating-point performance after training but significantly improve the post-training quantization results. We hope that with these two architectural changes to transformers, anyone can train high-performance transformers that are easy to quantize and can benefit from efficient integer inference.
[66] Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases. 2023.
[67] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In CVPR, 2022.
[68] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf.
[69] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27168-27183. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ADF7fa39d65e2983d724ff7da57f00ac-Paper-Conference.pdf.
[70] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[71] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188, 2019.
[72] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017.
[73] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[74] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[75] Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552. PMLR, 2019.
[76] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bandwidth convolutional neural networks with low bandwidth gradients. arXiv preprint arXiv:1606.06160, 2016.
[77] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
A Additional graphs from outlier analysis
In this section, we present additional graphs from our outlier investigation in Section 3 for BERT and vision transformer.
(b)
Figure 9: A summary of several outlier statistics recorded from ImageNet validation set on ViT. (a) Average infinity norm of the output of each attention layer. (b) A histogram of outlier counts in attention layer #10 vs. hidden dimensions. We use zero-based indexing for dimensions. (c) A heatmap of outlier counts in attention layer #10 vs. patch positions.
A.1 BERT
Recall from Figure 1 that all the outliers are only present in hidden dimensions #123, #180, #225, #308, #381, #526, #720 (with the majority of them in #180, #720). These hidden dimensions correspond to attention heads #2, #3, #4, #5, #6, #9, and #12. In Figures 10 and 11 we show more examples of the discovered self-attention patterns for attention heads #3 and #12 ( $\leftrightarrow$  hidden dim #180 and #720, respectively). We also show self-attention patterns in attention heads and layers which are not associated with the outliers in Figures 12 and 13, respectively. Finally, in Figures 14 and 15 we show more examples of the attention patterns learned in the network trained with clipped softmax and gated attention.
A.2 ViT
Figure 9 further shows that there are a lot of similarities in the outlier behavior in the vision transformer, compared to BERT. The strongest magnitude outliers generally happen in the later layers, peaking at layers #10 and #11. The majority of outliers ( $>99\%$ ) are only ever happening in only 10 hidden dimensions, primarily in dimensions #48 and #43, which corresponds to the attention head #1. Finally, averaged across the entire ImageNet validation set, the outliers seem to be concentrated at the boundaries of the image, which suggest a strong correlation with the background (and a negative correlation with the object, which is usually in the center of the image in the ImageNet dataset).
In Figures 16 and 17, we show more examples of outlier and self-attention patterns in the attention head #1 ( $\leftrightarrow$  hidden dimensions #48, #43) for a random subset of images from the ImageNet validation set (in layers #10 and #11, respectively).
B Detailed results
In this section, we provide extended results for each model, including the used hyperparameters and other design choices. We also present some additional ablation studies.
Configuration	G	Memory overhead (per attention layer)
# extra parameters	# extra tokens
Linear	n_heads × Linear(d_head → 1)	n_heads(d.head + 1)	~ 1
MLP	n_heads × MLP(d.head → n HID → 1)	n_heads(n HID(d.head + 2) + 1)	~ n HID
All-heads-linear	Linear(d_model → n_heads)	n_heads(d_model + 1)	~ n_heads
B.1 Gating architectures
We investigate the choice of several gating functions, summarized in Table 4. The configuration "MLP" parameterizes each  $G_{i}$  with a feed-forward net with one hidden layer of size  $n_{\mathrm{hid}}$  and a ReLU non-linearity [47]. We also explore what happens if we allow the mixing of the representation from different attention heads in the "All-heads-linear" setting, where we use a single linear layer to produce the gating probabilities for all attention heads at once. All three options are tested below. Unless explicitly stated otherwise, we initialize the bias of the gating function to zero (i.e.,  $b_{\mathrm{init}} = 0 \leftrightarrow \pi_{\mathrm{init}} = 0.5$ ).
B.2 BERT
Table 4: An overview of the gating function parameterizations explored in this paper and their memory overhead.
Method	FP16 ppl.↓	Max inf norm	Avg. Kurtosis	W8A8 ppl.↓
Vanilla	4.49±0.01	735.0±54.9	3076±262	1294±1046
CS (γ = -0.005)	4.44±0.02	406.6±35.2	1963±753	75.27±39.57
CS (γ = -0.01)	4.35±0.01	198.3±78.7	1581±839	7.06±2.37
CS (γ = -0.015)	4.37±0.01	38.9±7.9	165±34	4.54±0.01
CS (γ = -0.02)	4.39±0.02	31.7±6.3	90±20	4.56±0.02
CS (γ = -0.025)	4.39±0.00	21.5±1.5	80±6	4.52±0.01
CS (γ = -0.03)	4.41±0.01	20.4±0.2	79±6	4.55±0.01
CS (γ = -0.04)	4.51±0.05	19.8±9.0	85±7	4.65±0.06
GA, Linear (πinit = 0.25)	4.49±0.00	139.8±62.3	739±412	5.05±0.27
GA, Linear (πinit = 0.5)	4.48±0.00	177.3±33.2	652±81	5.13±0.15
GA, Linear (πinit = 0.75)	4.49±0.00	71.4±49.9	262±147	4.88±0.22
GA, Linear (πinit = 0.9)	4.49±0.00	171.5±8.8	559±141	5.15±0.03
GA, MLP (nhid = 4)	4.45±0.03	39.2±26.0	201±181	4.65±0.04
GA, MLP (nhid = 64)	4.49±0.01	117.0±48.3	507±167	4.77±0.01
GA, All-heads-linear	4.49±0.01	58.3±41.2	334±321	4.67±0.03
Table 5: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to BERT-base. We report the masked language modeling perplexity (ppl. for short) on the English Wikipedia validation set for both the floating-point baseline and W8A8 quantized model. We also report the maximum  $\| \mathbf{x}\|_{\infty}$  averaged across the validation set, and kurtosis of  $\mathbf{x}$  averaged across all layers, where  $\mathbf{x}$  is the output of an attention layer.
Detailed results for BERT-base are summarized in Table 5. As we can see, across most of the settings, both of our methods significantly dampen the outliers' magnitude, reduce the kurtosis, drastically improve the quantized performance, while maintaining and sometimes improving the FP16 perplexity.
B.3 OPT
Detailed results for OPT-125m are summarized in Table 6.
In our early experiments on a smaller OPT model, we found that applying the weight decay on LayerNorm weights  $\gamma$  (which isn't the case, by default) has a strong effect on reducing the outliers' magnitude while yielding the comparable FP16 performance. Therefore, we present the results of applying our gated attention approach in both cases, with and without applying weight decay on LN  $\gamma$ . As we can see in Table 6, in both cases gated attention (further) dampens the outliers' magnitude to a
Method	LN γ wd	FP16 ppl.↓	Max inf norm	Avg. Kurtosis	W8A8 ppl.↓
Vanilla	×	15.84±0.05	339.6±47.2	1777±444.	21.18±1.89
GA, Linear (πinit = 0.1)	×	15.61±0.05	35.6±4.5	42.4±22.9	16.41±0.18
GA, Linear (πinit = 0.25)	×	15.50±0.04	35.8±0.5	59.0±48.3	16.25±0.08
GA, Linear (πinit = 0.5)	×	15.54±0.01	46.5±5.0	40.6±8.9	16.30±0.01
GA, All-heads-linear	×	15.43±0.01	32.8±1.7	24.2±3	16.30±0.12
Vanilla	✓	15.96±0.03	87.7±31.9	2080±1460	39.46±16.59
CS (γ = -1/512)	✓	15.99±0.02	106.4±7.0	5764±2150	185.23±220.00
CS (γ = -2/512)	✓	15.90±0.02	102.0±27.0	11290±4372	60.90±52.70
CS (γ = -4/512)	✓	15.86±0.01	83.1±20.6	17174±7791	84.64±10.55
CS (γ = -8/512)	✓	16.13±0.09	61.5±9.9	19204±4284	42.62±3.64
CS (γ = -12/512)	✓	16.29±0.07	63.2±8.8	19727±7479	37.22±2.39
GA, Linear (πinit = 0.1)	✓	15.69±0.05	7.3±0.4	25.4±10	16.23±0.08
GA, Linear (πinit = 0.25)	✓	15.55±0.05	8.7±0.6	18.9±1	16.02±0.07
GA, Linear (πinit = 0.5)	✓	15.63±0.00	10.8±0.7	42.0±19	16.20±0.01
GA, All-heads-linear	✓	15.53±0.01	7.9±0.3	13.8±1	16.09±0.08
Table 6: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to OPT-125m. We report the causal language modeling perplexity (ppl. for short) on the English Wikipedia validation set for both the floating-point baseline and W8A8 quantized model. We also report the maximum  $\| \mathbf{x}\|_{\infty}$  averaged across the validation set, and kurtosis of  $\mathbf{x}$  averaged across all layers, where  $\mathbf{x}$  is the output of an attention layer.
great extent, reduces the kurtosis, and yields models with significantly higher quantized performance, which is close to the original FP16 performance.
B.4 ViT
Method	Patch. Embd. LN	FP32 acc.	Max inf norm	Avg. Kurtosis	W8A8 acc.
Vanilla	×	80.75±0.10	358.5±81.2	1018.3±471.5	69.24±6.93
CS (γ = -0.003)	×	80.24±0.05	69.3±20.7	25.6±8.6	78.71±0.33
CS (γ = -0.004)	×	80.38±0.01	74.9±10.6	30.6±4.9	78.66±0.49
GA, Linear (πinit = 0.25)	×	80.62±0.01	86.0±8.0	23.4±2.7	79.16±0.05
GA, Linear (πinit = 0.5)	×	80.32±0.02	88.4±17.9	27.9±14.0	78.90±0.25
GA, MLP (nhid = 4)	×	80.62±0.05	118.2±40.5	47.8±29.8	78.79±0.29
Vanilla	✓	80.98±0.08	81.1±2.5	24.5±1.8	79.62±0.06
CS (γ = -0.0001)	✓	80.89±0.13	73.7±14.9	22.9±1.6	79.77±0.25
CS (γ = -0.0003)	✓	80.92±0.07	78.9±5.5	23.8±0.5	79.63±0.05
CS (γ = -0.0005)	✓	80.95±0.08	72.9±11.8	24.4±0.7	79.73±0.08
CS (γ = -0.001)	✓	80.95±0.16	80.8±2.1	24.1±0.7	79.69±0.03
CS (γ = -0.002)	✓	80.80±0.07	78.0±0.5	25.8±0.7	79.32±0.07
CS (γ = -0.003)	✓	80.79±0.02	75.6±7.9	28.1±4.0	79.00±0.10
GA, Linear (πinit = 0.5)	✓	81.01±0.06	79.8±0.5	19.9±0.3	79.82±0.11
GA, Linear (πinit = 0.75)	✓	81.01±0.05	77.8±0.3	21.8±1.9	79.80±0.08
GA, Linear (πinit = 0.9)	✓	80.92±0.11	70.6±8.0	23.2±3.7	79.64±0.09
Table 7: Main results for our proposed clipped softmax (CS) and gated attention (GA) applied to ViT-S/16. We report the top-1 accuracy on ImageNet-1K validation set for floating-point baseline and W8A8 quantized model. We also report the maximum  $\| \mathbf{x} \|_{\infty}$  averaged across the validation set, and kurtosis of  $\mathbf{x}$  averaged across all layers, where  $\mathbf{x}$  is the output of the attention layer.
Detailed results for ViT-S/16 are summarized in Table 7.
After our preliminary experiments on ViT, we noticed that distinct outliers already originate after the patch embeddings. Therefore, we experimented with adding the LayerNorm after the patch
embeddings (which was absent in the model definition, by default). As we can see in Table 6, together with this change, both of our proposed methods greatly dampens the outliers' magnitude, reduces the kurtosis, and yields models with significantly higher quantized performance, which is within  $1\%$  of the original FP32 accuracy.