SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Performance on language and vision tasks. Metric: NLU=Mean Accuracy/Correlation. CLS=Accuracy. NLG=BLEU. QA=F1. MT=SacreBleu. †: do not quantize optimizer states for embedding layers; ‡:  $\beta_{1} = 0$ . See more results in App. A.
Optimizer	NLU RoBERTa-L	CLS Swin-T	NLG GPT-2 M	QA RoBERTa-L	MT Transformer
32-bit AdamW	88.9 ± 0.01	81.2 ± 0.05	67.7 ± 0.67	94.6 ± 0.13	26.61 ± 0.08
32-bit Adafactor	89.1 ± 0.00	80.0 ± 0.03	67.2 ± 0.81	94.6 ± 0.14	26.52 ± 0.02
32-bit Adafactor‡	89.3 ± 0.00	79.5 ± 0.05	67.2 ± 0.63	94.7 ± 0.10	26.45 ± 0.16
32-bit SM3	87.5 ± 0.00	79.0 ± 0.03	66.9 ± 0.58	91.7 ± 0.29	22.72 ± 0.09
8-bit AdamW†	89.1 ± 0.00	81.0 ± 0.01	67.5 ± 0.87	94.5 ± 0.04	26.66 ± 0.10
4-bit AdamW (ours)	89.1 ± 0.01	80.8 ± 0.02	67.8 ± 0.51	94.5 ± 0.10	26.28 ± 0.05
4-bit Factor (ours)	88.9 ± 0.00	80.9 ± 0.06	67.6 ± 0.33	94.6 ± 0.20	26.45 ± 0.05
$\beta_{1} > 0$  and the  $\beta_{1} = 0$  (no first moment) configuration. For our 4-bit optimizers, we report two versions both based on 32-bit AdamW: (1) "4-bit AdamW" quantizes first moment with B128/DE and second moment with Rank-1/Linear. (2) the more memory efficient "4-bit Factor" quantizes first moment with B128/DE, factorizes second moment when the dimension of tensor is greater than 1, and quantizes lefted 1-dimensional second moment with Rank-1/Linear. See App. D for details about the experiments.
Models, datasets and hyperparameters We report performance metrics on standard benchmarks, including image classification (CLS) with Swin-T [31] on ImageNet-1k [12], natural language understanding (NLU) by fine-tuning RoBERTa-L [30] fine-tuning on GLUE [51], question answering (QA) by fine-tuning RoBERTa-L on SQuAD [42, 43], natural language generation (NLG) by fine-tuning GPT-2 Medium [39] on E2E-NLG [35], machine translation (MT) by training Transformer-Base [50] on WMT14 en-de [3] and LLaMA [49] fine-tuning. We fine-tune LLaMA-7B, LLaMA-13B and LLaMA-33B [49] on the Alpaca dataset [48]** and evaluate them on MMLU [21] and standard common sense reasoning benchmarks: HellaSwag [56], ARC easy and challenge [10] and OpenBookQA [33].
We mainly follow the hyperparameters in the original paper or/and codebase. In each benchmark, we keep same hyperparameters in one optimizer on different quantization schemes, which gives an out-of-box transfer from full-precision optimizer to low-bit optimizer without extra hyperparameter tuning. See App. D for hyperparameters and training details.
Accuracy of 4-bit Optimizers We first check whether our memory efficient 4-bit optimizers could retain accuracy. According to Tab. 2, our 4-bit optimizers can match or exceed 32-bit AdamW performance on all fine-tuning tasks (NLU, QA, and NLG) and are comparable on all pretraining tasks (CLS and MT). Sublinear memory optimizers Adafactor  $(\beta_{1} = 0)$  and SM3 could have better memory efficiency, but they suffer from performance degradation, particularly on the CLS task. According to Tab. 3, our 4-bit AdamW will not destroy the capability of pretrained models while enabling them to obtain instruction-following ability. 4-bit AdamW is comparable with 32-bit AdamW on all tasks and does not get worse when the model size grows. Moreover, their convergence curves closely align (Fig. 4).
Memory and Computing Efficiency We evaluate the memory and computation efficiency of proposed 4-bit optimizers on instruction tuning, NLU, and NLG tasks, in Tab. 4. Our 4-bit optimizer offers more memory saving compared to 8-bit optimizers, reducing the training memory consumption by up to  $57.7\%$ . It may look like the memory saving saturates when the bitwidth goes down. This is because we report the total memory consumption (including data, activations, and memory fragments)
$\frac{1}{2}$  https://github.com/microsoft/Swin-Transformer   
 $\S$  https://github.com/huggingface/transformers   
 $\mathbb{I}$  https://github.com/microsoft/LoRA   
 $\parallel$  https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer   
\*\*  $\mathrm{https://}$  //github.com/tatsu-labStanford_alpaca
Table 3: Performance on LLaMA fine-tuning on MMLU and commonsense reasoning tasks across different sizes.
Model	Optimizer	MMLU (5-shot)	HellaSwag	ARC-e	ARC-c	OBQA
LLaMA-7B	Original	33.1	73.0	52.4	40.9	42.4
32-bit AdamW	38.7	74.6	61.5	45.1	43.4
4-bit AdamW	38.9	74.7	61.2	44.4	43.0
LLaMA-13B	Original	47.4	76.2	59.8	44.5	42.0
32-bit AdamW	46.5	78.8	63.6	48.3	45.2
4-bit AdamW	47.4	79.0	64.1	48.0	45.2
LLaMA-33B	Original	54.9	79.3	58.9	45.1	42.2
32-bit AdamW	56.4	79.2	62.6	47.1	43.8
4-bit AdamW	54.9	79.2	61.6	46.6	45.4
rather than the optimizer memory consumption alone. In principle, the optimizer states is 2x smaller for 4-bit AdamW than 8-bit AdamW, and about 4x smaller for 4-bit Factor.
The instruction tuning task uses two Nvidia A100 80GB GPUs, while the model is sharded across GPUs with PyTorch's FSDP. In this case, our 4-bit optimizer speeds up training due to reduced communication cost. For the smaller RoBERTa-L and GPT-2 Medium, our 4-bit optimizers appear to be slower than 8-bit AdamW. This is because we have not yet optimize our implementation with fused operators. The speed of our 4-bit optimizers should match or surpass 8-bit optimizers after operator fusion.
We report the largest OPT and LLaMA models trainable under a given memory budget with full-precision and our 4-bit optimizers in Tab. 5. Our optimizer allows for the training of  $4\mathrm{x}$  large OPT models, and enables the training of LLaMA-7B model with a single 80GB GPU.
Figure 4: Training loss curve of LLaMA-7B fine-tuning on Alpaca dataset (averaged over 3 runs). Only result of 4-bit AdamW is reported since all parameters are 1-dimension via FSDP packing. See more details in App. D.
Ablation Study Finally, we investigate the effectiveness of our proposed quantization schemes and the sensitivity of each moment to quantization in Tab. 6. We see that quantizing both the first and second moment brings a marginal drop in accuracy. Smaller block size on first moment improves accuracy and takes a step towards lossless performance. Factorizing second moment improves accuracy while leads to better memory efficiency.
Compression-based memory efficient optimizers There have been some works trying to approximate the gradient statistics with sublinear memory cost relative to the number of parameters. Adafactor [46] achieves memory reduction by approximating the second-moment of matrix-shaped
Table 4: Memory and Time of 4-bit optimizers compared with 32-bit AdamW and 8-bit Adam [15].
Task	Optimizer	Time	Total Mem.	Saved Mem.
LLaMA-7B	32-bit AdamW	3.35 h	75.40 GB	0.00 GB (0%)
4-bit AdamW	3.07 h	31.87 GB	43.54 GB (57.7%)
4-bit AdamW (fused)	3.11 h	31.88 GB	43.53 GB (57.7%)
RoBERTa-L	32-bit AdamW	3.93 min	5.31 GB	0.00 GB (0%)
8-bit AdamW	3.38 min	3.34 GB	1.97 GB (37.1%)
4-bit AdamW	5.59 min	3.02 GB	2.29 GB (43.1%)
4-bit AdamW (fused)	3.17 min	3.00 GB	2.31 GB (43.5%)
4-bit Factor	4.97 min	2.83 GB	2.48 GB (46.7%)
GPT-2 Medium	32-bit AdamW	2.13 h	6.89 GB	0.00 GB (0%)
8-bit AdamW	2.04 h	4.92 GB	1.97 GB (28.6%)
4-bit AdamW	2.43 h	4.62 GB	2.37 GB (34.4%)
4-bit AdamW (fused)	2.11 h	4.62 GB	2.37 GB (34.4%)
4-bit Factor	2.30 h	4.44 GB	2.45 GB (35.6%)
Table 5: Largest trainable model under given memory budget. We use a batch size of 1 and max length of 512 for this comparison. FSDP is enabled at GPUs of 80 GB.
GPU Mem.	Largest fine-tunable Model
32-bit AdamW	4-bit AdamW
24 GB	OPT-350M	OPT-1.3B
80 GB	OPT-1.3B	OPT-6.7B
80 GB	-	LLaMA-7B
Table 6: Ablation study on the impact of compressing different moments to Swin-T pretraining on ImageNet1k.
Quant. 1st	Quant. 2nd	Factor. 2nd	Acc.
-	-	X	81.2 ± 0.05
B2048/DE	-	X	80.9 ± 0.04
B128/DE	-	X	81.0 ± 0.06
B128/DE	Rank-1/Linear	X	80.8 ± 0.02
B128/DE	Rank-1/Linear	✓	80.9 ± 0.06
parameters with the outer product of "row" and "column". SM3 [2] considers the cover of parameters and maintain one statistics for each element in the cover. Experimentally, cover composed of slices of co-dimension 1 for each tensor has been adopted. Extreme tensoring [8], compared with SM3, has a different motivation but similar formulation for the experimental choice. These memory efficient methods only focus on memory reduction on second moment and are applicable to most of the second moment based optimizers, like Adagrad [19], Adam [26], AdamW [32], etc.
Another line of work achieves memory reduction by using compression-based method and maintaining coordinate-wise low-bit precision optimizer states. The stability of 16-bit Adam is firstly studied in DALL-E [44]. Dettmers et al. [15] uses block-wise and dynamic exponent quantization to reduce the coordinate-wise optimizer states from 16-bit to 8-bit and is applicable to SGDM and Adam/AdamW. Compression-based methods only reduce memory by a fixed percentage, irrelevant to the number and shape of parameters. Compression-based methods are less memory efficient compared with sublinear memory methods, but exhibit superior performance across a wider range of benchmarks.
Other memory efficient techniques Some works focus on the memory efficiency of activations. Activation compressed training [6, 29] keeps low-precision activations in GPU memory via quantization in forward phase and dequantize the low-precision activations to full-precision layer-wisely in backward phase. Gradient checkpointing [7] only keeps activation in a small number of layers in forward phase and recompute the activations in all layers when gradient computation is needed, which leads a trade-off between storage overhead of activations and additional computation cost. These methods can be combined with our optimizer for better memory efficiency. LoRA [24] freezes the pretrained weights and only tunes new initialized low-rank parameters, which greatly reduce the size of computation graph but is only applicable to language model finetuning.
Sharding [40] divides model parameters and optimizer states among multiple devices, allowing for more efficient model training through the use of additional GPUs. Offloading [41, 45] reduces GPU memory consumption by transferring data to CPU memory. Our optimizer can be utilized by these methods to reduce the communication overhead.
7 Conclusions, Limitations, and Broader Impact
We propose compression-based memory efficient 4-bit optimizers using quantization and factorization. We observe the heterogeneous outlier patterns in optimizer states and identify the zero-point problem in quantizing second moment as the main bottleneck. 4-bit optimizers achieve lossless performance in finetuning and comparable accuracy in pretraining on a wide range of tasks.
Limitations The optimal quantization setting probably depends on task, datasets, and training details. While rank-1 normalization and linear mapping for second moment quantization performs consistently well in our experiments, task-specific quantization settings not in the scope of the study might perform better and be helpful to achieve lossless performance. Our evaluation is currently limited to language and vision tasks, while the applicability of our method to reinforcement learning, audios, and graph learning tasks still needs further study.
Broader Impact Our work can facilitate the access to large models for pretraining and finetuning, which were previously constrained by GPU memory limitations. This could help democratizing large models and opens up new avenues for research that were previously unattainable due to restricted GPU memory, especially benefiting researchers with limited resources. However, our work can also exaggerate the abuse large models.
[1] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems, 30, 2017.
[2] Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. Advances in Neural Information Processing Systems, 32, 2019.
[3] Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12-58, 2014.
[4] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947-7969, 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[6] Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael Mahoney, and Joseph Gonzalez. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, pages 1803-1813. PMLR, 2021.
[7] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.
[57] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In International Conference on Learning Representations, 2023.
[58] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
A Additional Experiment Results
In this section, we show additional experiment results beyond Tab. 2. Tab. 7 shows the results of RoBERTa-L finetuning on each task in GLUE datasets. Tab. 8 shows the results of GPT-2 Medium finetuning on E2E-NLG via different metrics. Tab. 9 shows the EM and F1 of RoBERTa-L finetuning on SQuAD and SQuAD 2.0 datasets.
Table 7: Performance of RoBERTa-Large finetuning on GLUE with diverse optimizers. Medians and std over 5 runs are reported on all tasks. †: do not quantize optimizer states for embedding layers; ‡:  $\beta_{1} = 0$ .
Optimizer	MNLI	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B
32-bit AdamW	90.2 ± 0.00	94.9 ± 0.00	92.2 ± 0.00	85.2 ± 0.14	96.3 ± 0.00	93.2 ± 0.01	66.9 ± 0.01	92.3 ± 0.00
32-bit Adafactor	90.4 ± 0.00	94.7 ± 0.00	92.2 ± 0.00	85.9 ± 0.02	96.3 ± 0.00	92.8 ± 0.00	67.3 ± 0.01	92.3 ± 0.00
32-bit Adafactor†	90.5 ± 0.00	94.8 ± 0.00	92.2 ± 0.00	87.0 ± 0.03	96.3 ± 0.00	92.9 ± 0.00	68.2 ± 0.01	92.2 ± 0.00
32-bit SM3	90.6 ± 0.00	94.2 ± 0.00	89.5 ± 0.00	85.2 ± 0.02	96.0 ± 0.00	90.5 ± 0.01	62.3 ± 0.04	91.4 ± 0.01
8-bit AdamW†	90.4 ± 0.00	94.8 ± 0.00	92.2 ± 0.00	84.8 ± 0.02	96.2 ± 0.00	93.2 ± 0.00	68.0 ± 0.00	92.2 ± 0.00
4-bit AdamW	90.2 ± 0.00	94.5 ± 0.00	92.0 ± 0.00	85.2 ± 0.12	96.3 ± 0.00	92.8 ± 0.00	67.3 ± 0.01	92.5 ± 0.00
4-bit Factor	90.1 ± 0.00	94.7 ± 0.00	92.2 ± 0.00	85.9 ± 0.00	96.4 ± 0.00	92.7 ± 0.00	68.1 ± 0.00	92.3 ± 0.00
Table 8: Performance of GPT-2 Medium finetuning on E2E-NLG Challenge with diverse optimizers. Means and std over 3 runs are reported.
Optimizer	BLEU	NIST	METEOR	ROUGE-L	CIDEr
32-bit AdamW	67.7 ± 0.67	8.60 ± 0.08	45.7 ± 0.28	68.7 ± 0.61	2.35 ± 0.04
32-bit Adafactor	67.2 ± 0.81	8.61 ± 0.60	45.3 ± 0.08	68.3 ± 0.22	2.35 ± 0.01
32-bit Adafactor‡	67.2 ± 0.63	8.54 ± 0.09	45.6 ± 0.32	68.5 ± 0.30	2.32 ± 0.02
32-bit SM3	66.9 ± 0.58	8.59 ± 0.04	45.4 ± 0.32	68.2 ± 0.49	2.33 ± 0.03
8-bit AdamW†	67.5 ± 0.87	8.59 ± 0.08	45.7 ± 0.52	68.7 ± 0.97	2.34 ± 0.06
4-bit AdamW	67.8 ± 0.51	8.61 ± 0.08	45.8 ± 0.23	68.9 ± 0.33	2.35 ± 0.07
4-bit Factor	67.6 ± 0.33	8.59 ± 0.03	45.6 ± 0.43	68.6 ± 0.60	2.34 ± 0.06
Table 9: Performance of RoBERTa-Large on SQuAD and SQuAD 2.0 with diverse optimizers. Medians and std over 5 runs are reported.
Optimizer	SQuAD	SQuAD 2.0
EM	F1	EM	F1
32-bit AdamW	89.0 ± 0.10	94.6 ± 0.13	85.8 ± 0.18	88.8 ± 0.15
32-bit Adafactor	88.8 ± 0.12	94.6 ± 0.14	85.8 ± 0.44	88.7 ± 0.21
32-bit Adafactor‡	89.0 ± 0.18	94.7 ± 0.10	85.9 ± 0.15	88.8 ± 0.15
32-bit SM3	84.2 ± 0.49	91.7 ± 0.29	77.2 ± 0.71	81.1 ± 0.66
8-bit AdamW†	88.8 ± 0.15	94.5 ± 0.04	86.1 ± 0.26	89.0 ± 0.26
4-bit AdamW	88.8 ± 0.08	94.5 ± 0.10	85.4 ± 0.28	88.4 ± 0.26
4-bit Factor	88.8 ± 0.38	94.6 ± 0.20	85.9 ± 0.36	88.9 ± 0.18
B Outlier Patterns of Moments
In this section, we give a comprehensive visualization about the outlier pattern of optimizer states. [2] did similar analysis for Adagrad's second moment but here we give a better demonstration about various patterns in optimizer states. The same technique has been applied to parameters and activations in [54].
The outlier pattern of first moment depends on many factors such as data and training hyperparameters. Here we mainly focus on different transformer models and different layers inside. In one transformer block, there is one Attention module and one MLP module, including 6 main parameter matrices. We do not focus on additional parameters including bias and parameters in layer normalization (LN) since they only account a small portion. We denote the 6 matrices by  $\mathbf{W}^Q$ ,  $\mathbf{W}^K$ ,  $\mathbf{W}^V$ ,  $\mathbf{W}^O$ ,  $\mathbf{W}^1$ ,  $\mathbf{W}^2$ , respectively. When the matrices across different layers are involved at the same time, we add a subscript indicating layer index. Note that  $\mathbf{W}$  has shape  $\mathbb{R}^{C_{\mathrm{o}} \times C_{\mathrm{i}}}$  in a linear layer, we call the output and input dimension by  $\dim0/\text{row}$  and  $\dim1/\text{column}$ , respectively. The per-channel quantization used in other works actually correspond to per-row  $(\dim0)$  normalization here.
Swin Transformer ImageNet pretraining In Fig. 5,6,7, the magnitude of first moment in transformer blocks at different depths are shown. It can be seen that the 1-dimensional structure in all parameter matrices are vague at the initial layer. At layer 2, the pattern in  $\mathbf{W}^O$  and  $\mathbf{W}^1$ , that outliers occur at fixed columns, becomes obvious while other parameter matrices remain noisy. At layer 3, the patterns in  $\mathbf{W}^O$ ,  $\mathbf{W}^1$ ,  $\mathbf{W}^2$  are quite obvious. In  $\mathbf{W}^O$  and  $\mathbf{W}^2$ , the outliers occur at fixed rows while the pattern in  $\mathbf{W}^1$  remains unchanged. The 1-dimensional structure in  $\mathbf{W}^Q$ ,  $\mathbf{W}^K$ ,  $\mathbf{W}^V$  also seem to appear even though not remarkable. It is notable that the pattern of same parameter matrix at different depths are not necessarily same. See  $\mathbf{W}^O$  in Fig. 6,7.
Figure 5: Outlier patterns of first moment in transformer block layers.0. blocks.0 of Swin-T at epoch 210.
Figure 6: Outlier patterns of first moment in transformer block layers.2 blocs.0 of Swin-T at epoch 210.
Figure 7: Outlier patterns of first moment in transformer block layers.3. blocks.0 of Swin-T at epoch 210.
RoBERTa-Large GLUE finetuning In Fig. 8,9,10,11,12,13, the magnitude of first moment in transformer blocks of RoBERTa-Large at different depths are shown. At layer 0 and layer 1 (initial layers), patterns in  $\mathbf{W}^O$ ,  $\mathbf{W}^2$  are obvious. At layer 11 and layer 12 (intermediate layers), patterns are all noisy. At layer 22 and layer 23 (last layers), patterns in  $\mathbf{W}^Q$ ,  $\mathbf{W}^K$  are obvious. Patterns in other matrices are weak.
W2
Figure 8: Outlier patterns of first moment in transformer block layer-0 of RoBERTa-Large at epoch 8.
WO
Figure 9: Outlier patterns of first moment in transformer block layer-1 of RoBERTa-Large at epoch 8.
Figure 10: Outlier patterns of first moment in transformer block layer-11 of RoBERTa-Large at epoch 8.
Figure 11: Outlier patterns of first moment in transformer block layer-12 of RoBERTa-Large at epoch 8.
Figure 12: Outlier patterns of first moment in transformer block layer-22 of RoBERTa-Large at epoch 8.
Figure 13: Outlier patterns of first moment in transformer block layer-23 of RoBERTa-Large at epoch 8.
GPT-2 Medium E2E-NLG finetuning In Fig. 14,15,16,17,18,19, the magnitude of first moment in transformer blocks of GPT-2 Medium at different depths are shown. At layer 1 and layer 2 (initial layers), patterns in  $\mathbf{W}^{\mathrm{O}}$  are obvious. At layer 13 and layer 14 (intermediate layers), patterns in  $\mathbf{W}^{K},\mathbf{W}^{O}$  are obvious. At layer 21 and layer 22 (last layers), patterns in  $\mathbf{W}^{Q},\mathbf{W}^{K},\mathbf{W}^{V},\mathbf{W}^{O}$  are obvious. First moment of  $\mathbf{W}^{1},\mathbf{W}^{2}$  are consistently noisy throughout layers. It is notable that the rows(or columns) that gather outliers are different across different layers.
W1
Figure 14: Outlier patterns of first moment in transformer block layer-1 of GPT-2 Medium at epoch 2.
WO
Figure 15: Outlier patterns of first moment in transformer block layer-2 of GPT-2 Medium at epoch 2.
Figure 16: Outlier patterns of first moment in transformer block layer-13 of GPT-2 Medium at epoch 2.
Figure 17: Outlier patterns of first moment in transformer block layer-14 of GPT-2 Medium at epoch 2.
Figure 18: Outlier patterns of first moment in transformer block layer-21 of GPT-2 Medium at epoch 2.
Figure 19: Outlier patterns of first moment in transformer block layer-22 of GPT-2 Medium at epoch 2.
C Quantization Quality via Histogram
C.1 Zero-point Problem
In Fig. 20,21,22, we show the effect of zero-point on quantization error for second moment via histogram. All those figures show the negative impact of zero-point on quantizing second moment. After removing zero-point, the quantization quality improves at a great scale.
Figure 20: Histogram of second moment of the attention layer  $(\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V)$  in transformer block-wise layer-2 of GPT-2 Medium at epoch 2. In one horizontal line, the first figure is the original second moment. The second figure is the tensor after normalization. The third figure is the quantized tensor. The last figure is the dequantized object. Both the first and last figure is at log10 scale. Both the second and third take values in [0, 1]. All y-axis represents density. Good quantization methods try to make the third figure identical to the second figure and make the last figure identical to the first figure. Top: B128/DE quantization. Bottom: B128/DE-0 quantization.
Figure 21: Histogram of second moment of the  $\mathbf{W}^V$  in transformer block layer-10 of RoBERTa-Large at epoch 8. Top: B128/DE quantization. Bottom: B128/DE-0 quantization.
Figure 22: Histogram of second moment of the attention layer  $(\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V)$  in transformer block layers.0 blocs.0 of Swin-T at epoch 210. Top: B128/DE quantization. Bottom: B128/DE-0 quantization.
C.2 Comparison between Block-wise and Rank-1 Normalization
To show the differences in quantization error for second moment between block-wise normalization and rank-1 normalization, some cases where rank-1 normalization approximates better than blockwise normalization are shown in Fig. 23, 25, 27. Also, some cases where rank-1 normalization approximates worse than block-wise normalization is shown in Fig. 24, 26, 28. Empirically, it has been observed that rank-1 normalization yields superior results when the distribution exhibits long-distance multimodal characteristics. On the other hand, block-wise normalization tends to outperform when the distribution displays short-distance multimodal patterns and/or intricate local structures.
Figure 23: Histogram of second moment of  $\mathbf{W}^1$  in transformer block layer-23 of GPT-2 Medium at epoch 2. A case where rank-1 normalization is better than block-wise normalization with block size 128. In this case, the tail in the right side of distribution is captured by rank-1 normalization but lost in block-wise normalization. Top: B128/DE-0 quantization. Bottom: Rank-1/DE-0 quantization.