SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Outlier Analysis As a potential proxy measure of PTQ degradation, we measure the number of outlier dimensions as it's been shown to strongly effect activation quantization sensitivity of LLMs(Dettmers et al., 2022). Activation feature dimensions are classified as outlier dimensions when the activation values (denoted as  $\alpha$ ) are greater than 6.0 at more than  $25\%$  of layers and  $6\%$  of tokens. However, we find that a threshold of 6.0 is too high to classify a feature dimension as an outlier for all the variants we consider. After correspondence with the authors, we also explored various adaptations of this outlier detection recipe presented in Dettmers et al. (2022) to make it generalizable. However, we did not observe a clear correlation between these measures and sensitivity to quantization. We refer to Appendix B.3 for detailed treatment of these replication efforts.
Scaling Insights to 52B scale To validate our experimental findings at scale and with fully trained models, we pre-train 410M, 6B, 13B, and 52B parameter models for more than 200k steps, using the best optimization choices with respect to robustness in post-training quantization: weight decay of 0.1, no dropout, gradient clipping of 1, and bf16 as the half-precision format. Figure 1 shows mean zero-shot accuracy for the non-quantized and quantized model using INT8 weights and activations. Compared with OPT models (Zhang et al., 2022), our fully-trained models are significantly more robust to post-training quantization starting from 6B parameter size. Our largest scale model with 52B parameters, even shows a  $0.08\%$  average improvement across the evaluation suite and  $0\%$  degradation across LAMBADA, HellaSwag, and PIQA where OPT-66B which is the closest OPT model in terms of size, has an extreme drop in performance of  $42\%$  as reported in Dettmers et al. (2022).
Transferability of Results All the models that we experimented with are architecturally very similar to the OPT & BLOOM family as they are both decoder-only, pre-layernorm (where the layernorm is placed before the linear layers), and use vanilla attention. They differ in the number of blocks and the hidden dimensions of models at different sizes. Since we thoroughly experimented with different model sizes, we believe that our insights directly translate to the OPT& BLOOM families' architectures.
To evaluate our models' performance under a different quantization recipe in addition to INT8, we also test 4-bit integer (INT4) column-wise weight-only quantization, similar to Du et al. (2021). Our 52B parameter model exhibited only a  $3.6\%$  relative drop in mean zero-shot performance across the 8 evaluation tasks. It is worth noting that this quantization scheme does not require any fine-tuning or optimization and hence these results highlight the high robustness of our trained models. In comparison, when applying the same quantization scheme to BLOOM, BLOOM-176B and BLOOM-7B show  $29.5\%$  and  $18.7\%$  degradation respectively on LAMBADA as reported in the GLM-130B technical report (Du et al., 2021), while our 52B model only has  $8.6\%$  degradation.
Figure 4: Weight distribution of attn-kqv-proj and layernorm gain (g) parameter in an example block (Block 14) for both fp16/bf16 variants and our final 6B model in comparison with OPT and BLOOM. Weight distributions for all blocks are shown in Appendix B.1
5 Weight and Activation Analysis
Our results in Section 4 find that sensitivity to quantization at scale is not an inherent emergent property. Rather, it can be avoided at scales as large as 52B given the right optimization choices. We also found that outlier dimensions are not always a consistent proxy for quantization sensitivity. Thus as an attempt to address the short-comings of outlier dimensions as a proxy and to reconcile our observations on degradation trends, we perform a fine-grained analysis of activations and weights to understand how the trained distribution of our models differs from models like OPT that are far more sensitive to quantization. We hope that the metrics we propose and evaluate in this section, help further discussion about useful proxy metrics for guiding pre-training optimization choices to improve robustness to quantization. For all the metrics proposed below, we include the complete analysis for all layers in Appendix B.
Activations As a first step, we analyze input activations of the attention projection (attn-kqv-proj) as it is the earliest point for INT8 multiplication in a decoder block. Here, we measure root-mean-square error  $\mathrm{RMSE}(\mathbf{X},\hat{\mathbf{X}})$  where  $\hat{\mathbf{X}}$  denotes the de-quantized activations. Additionally, we report the mean standard deviation of the input activations measured per token. While  $\mathrm{RMSE}(\mathbf{X},\hat{\mathbf{X}})$  directly indicates the quantization error, standard deviation (STD) has been shown to be closely related to the expected quantization error of a normally distributed input (Kuzmin et al., 2022). Figure 5 compares the bf16 and fp16 variants. We observe that the RMSE and STD of fp16 are far higher than the bf16 variant – the RMSE for the fp16 variant is  $6.9\mathrm{x}$  the RMSE for the bf16 variant. This difference is even more pronounced if we compare our model to the OPT: the RMSE and STD of the OPT are  $27.7\mathrm{x}$  and  $1.8\mathrm{x}$  higher respectively relative to our model (Figure 5; Bottom row).
LayerNorm Since we use a pre-norm architecture, the input activations to attn-kqv-proj and outer-expander-mlp are both preceded by a layernorm. The layernorm gain parameter,  $\mathbf{g} \in \mathbb{R}^h$ , directly influences the output activation tokens' spread, and can significantly vary in distribution shape as seen in Figure 4. We generally observe that within our experimental axes, the standard deviation of the gain parameters for both the first and second layernorms are higher in a significant number of layers for variants with higher degradation compared to others in the same axis. In Figure 5, we compare the standard deviation of  $\mathbf{g}$  for self-attention layernorm and we observe that  $\mathrm{STD}(\mathbf{g})$  is  $2x$  higher for the fp16 variant relative to bf16. Our findings add further support to previous work which suggests that the gain parameters act as an outlier amplifier and thus further quantization degradation through scaling (Wei et al., 2022b).
In the bottom row of Figure 5, we also compare  $\mathrm{STD}(\mathbf{g})$  of our model relative to OPT and BLOOM. We observe that even the BLOOM model that is relatively robust to quantization has a far larger  $\mathrm{STD}(\mathbf{g})$  than our model with a multiplier of  $5\mathrm{x}$ . Interestingly, we find that OPT-6B layernorm gain parameters are all set to 1.0 while biases varied as expected. Hence, given the gain parameters appear to be hardcoded, the  $\mathrm{STD}(\mathbf{g})$  of the OPT model is 0. We were not able to find any mention of such design decision either in Zhang et al. (2022) or the github repository: https://github.com/facebookresearch/metaseq.
Attention Projection Weights Finally, we compare the weight distribution of attn-kqv-proj layers. As seen in Figure 4, the fp16 variant has a significantly wider distribution compared to bf16.
Figure 5: Average input token activation STD, preceding layernorm gain STD, and spectral norm of linear weights, follow trends similar to those observed in  $\mathrm{RMSE}(\mathbf{X},\hat{\mathbf{X}})$ . Plots correspond to the attn-kqv-proj layer in an example block (Block 14). Comparisons for all other blocks are given in Appendix B.2. Top row: Comparison of fp16 and bf16 variants. Bottom row: Our 6B model trained with optimal hyper-parameters compared against similar sized OPT-6B and BLOOM-7.1B models
Additionally, inspired by Lin et al. (2019), we use spectral norm to measure the maximum degree of noise amplification for each token activation.
For a given weight matrix  $\mathbf{W} \in \mathbb{R}^{h \times o}$ , and input token activation noise  $\mathbf{x}_{\delta} \in \mathbb{R}^{\mathbf{h}}$ , the spectral norm  $\| \cdot \|_2$  is defined as
$$
\left\| \mathbf {W} \right\| _ {2} = \sup  _ {\mathbf {x} _ {\delta} \neq 0} \frac {\left\| \mathbf {W} \mathbf {x} _ {\delta} \right\| _ {2}}{\left\| \mathbf {x} _ {\delta} \right\| _ {2}} = \sigma_ {\max } \tag {3}
$$
where  $\sigma_{max}$  is the largest singular value of  $\mathbf{W}$ .
As seen in Figure 5, we observe that the spectral norm of the fp16 variant is  $4\mathrm{x}$  higher than the bf16. In addition, on the bottom row of Figure 5, we observe that both BLOOM and our model have generally lower spectral norm than OPT 6B that is far more sensitive to quantization.
Challenges of Quantization at Scale Recently, there have been several studies to characterize the emergence of outliers at scale, and relate this to the difficulties in post-training quantization of both weights and activations (Dettmers et al., 2022; Wei et al., 2022b; Puccetti et al., 2022). Dettmers et al. (2022) depict a phenomenon of emerging outliers by observing that large outlier dimensions systematically emerge at a certain scale (6.7B parameters) which hamper quantization attempts. Extreme outliers at scale was also empirically confirmed in follow-up works (Zeng et al., 2022; Xiao et al., 2022). The causes of outliers have also been the subject of recent work. Puccetti et al. (2022) observe that in Masked Language Models (MLMs) the magnitude of hidden state coefficients corresponding to outlier dimensions correlates with the frequency of encoded tokens in pre-training data. Wei et al. (2022b) observe that LayerNorm scaling (g) amplifies the outliers and can be suppressed using a modified LayerNorm and token-wise clipping. Wortsman et al. (2023) consider large-scale vision-language models and show that quantization techniques are more stable if the network is trained and initialized so that large feature magnitudes are discouraged.
Most mitigation strategies to quantize in the presence of outliers has required more complex quantization techniques. For example, Dettmers et al. (2022) propose selective mixed-precision computation by only computing the outliers at higher precision. However, such a setup proves difficult to map to hardware, limiting the inference speedup. Xiao et al. (2022) propose to smoothen out these outliers by migrating some of the activation variances into the model weights with appropriate scaling.
Although the authors demonstrate the ability of this framework to scale to large models, additional rescaling is required for activations which leads to additional latency overhead without specialized kernels. Another limitation of Xiao et al. (2022) is that it relies on the assumption that outliers exist in activations, and that weights can bear additional outliers and still be easy to quantize.
Our work is the first to show that outliers are not inherent to scaling large language models. Rather than an emerging property, they are a result of particular training methods. Compared to previous methods using extensive quantization schemes with custom kernels (Dettmers et al., 2022), our work applies PTQ using simple, one-shot linear weight and activation quantizations which can take advantage of NVIDIA-provided CUTLASS kernels, leading to a significant decrease in latency and memory footprint (Table 6).
We present a rigorous study of the effect of how various optimization choices affect INT8 PTQ with the goal of reconciling the recent contradictory observations regarding emergent properties in Large Language Models. We show that regularization directly impacts PTQ performance and that higher levels of regularization through common techniques such as weight-decay, and gradient-clipping leads to lower post-training quantization degradation. We further demonstrate that the choice of half-precision training data type has a significant impact on PTQ performance – emergent features are significantly less pronounced when training with bf16.
Broader Impact Our work serves as a useful counter-example to scholarship which has advanced the notion that certain properties depend only on model scale (Wei et al., 2022a). Rather, our results support the conclusion that optimization choices play a large role in whether emergent properties are present. We believe there is more work to be done here. We also hope that the insights gained from our work illustrate the significant impact the underlying hardware can have on PTQ. Currently, bf16 training is possible on TPUs and only very recently introduced to A100 & H100 GPUs. Finally, we delve our results present an impactful formula for training models which are inherently easier to quantize at scale, making these models more accessible for deploying in a variety of deployment environments.
Limitations and Future Work We do not vary the architectural design (except for the positional embedding type) and training objective in our experiments given our goal of a controlled experimental set-up and the large computational cost of each variant. We leave this to the future work. In addition, it is worthwhile to explore the impact of finetuning with the recommended optimization choices on make a pre-trained model more quantization-friendly, as well as the theory behind why these optimization choices correlate with better compression results.
We thank João Araujo, Milad Alizadeh and other colleagues in Cohere & Cohere For AI for helpful feedback and support. Additionally, we thank Jimmy Ba for helpful discussions surrounding the topic of this paper. We also thank Tim Dettmers for assisting in replicating the outlier dimension definition and results in int8.LLM().
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2022. URL https://arxiv.org/abs/2211.10438.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-27183, 2022.
Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. A comprehensive study on post-training quantization for large language models, 2023.
Joanna Yoo, Kuba Perlin, Siddhartha Rao Kamalakara, and João G. M. Araujo. Scalable training of language models using jax pjit and tpuv4, 2022. URL https://arxiv.org/abs/2204.06514.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pp. 36-39. IEEE, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. Glm-130b: An open bilingual pre-trained model, 2022. URL https://arxiv.org/abs/2210.02414.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/abs/2205.01068.
Shaojie Zhuo, Hongyu Chen, Ramchalam Kinattinkara Ramakrishnan, Tommy Chen, Chen Feng, Yicheng Lin, Parker Zhang, and Liang Shen. An empirical study of low precision quantization for tinyml. arXiv preprint arXiv:2203.05492, 2022.