RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Post-Training Quantization
Model quantization can be mainly categorized into quantization-aware training (QAT) and post-training quantization (PTQ), depending on whether it involves additional training for weight fine-tuning or not. Most PTQ studies are primarily focused on convolutional neural networks (CNNs) Nagel et al. ( 2020 ); Li et al. ( 2021 ); Wu et al. ( 2020 ); Cai et al. ( 2020 ); Nagel et al. ( 2019 ) . However, with the growing popularity of transformer-based models, only a limited number of works Bondarenko et al. ( 2021 ); Yuan et al. ( 2022 ); Ding et al. ( 2022 ) have been conducted to realize PTQ on transformers. Moreover, the existing works primarily focus on visual transformer models and exhibit inferior performance when the bit width is below 8. Therefore, in this work, we delve into the challenges of the low-bit PTQ for language transformers.
Floating-Point Quantization
Floating-point (FP) quantization has emerged as a promising alternative to integer quantization due to its ability to handle long-tail distributions, and offers increased flexibility Kuzmin et al. ( 2022 ) . Additionally, modern GPUs such as H100 Micikevicius et al. ( 2022 ) now support FP quantization. Nonetheless, minimal research has been conducted on FP quantization. Only Kuzmin et al. ( 2022 ) proposes a general FP8 quantization scheme primarily for vision tasks, and Zhang et al. ( 2023 ) adopts a mixture of FP and INT formats quantization for LLMs. In this work, we propose FPQ baseline as a general guideline for low-bit floating-point PTQ to compress language transformer models.
Formulation of Floating-Point Variables
A standard floating-point number is represented as:

X_{\rm{FP}}=(-1)^{s}2^{p-b}(1+\frac{d_{1}}{2}+\frac{d_{2}}{2^{2}}+...+\frac{d_{m}}{2^{m}}) (1)

where s\in\{0,1\} is the sign bit. d_{i}\in\{0,1\} is i^{th} mantissa bit, m denoted number of mantissa bits. p is an integer in [0,2^{e}-1] , and e denotes number of exponent bits. b is an integer exponent bias. A floating point with j number exponent bits and k mantissa bits is denoted as FP format \rm{EjMk} .
Floating-Point Matrix Multiplication
With the floating-point quantized variables, the matrix multiplication is formulated as:

\vspace{-0.4em}\mathbf{O}_{out}^{i,k}=\mathbf{X}_{\rm FP}^{i,:}\mathbf{W}_{\rm FP}^{:,k}=\tilde{\alpha}_{{}_{\mathbf{X}}}\tilde{\alpha}_{{}_{\mathbf{W}}}^{k}\mathbf{\tilde{X}}^{i,:}_{\rm FP}\mathbf{\tilde{W}}^{:,k}_{\rm FP} (12)

Here in per-tensor activation quantization and per-channel weight quantization, \mathbf{X}_{\rm FP}^{i,:} denotes i^{th} row in the activation matrix and \mathbf{W}_{\rm FP}^{:,k} denotes k^{th} column in the weight matrix, such that each element \mathbf{O}_{out}^{i,k} in the output matrix is computed by the product of two real-valued scalars \tilde{\alpha}_{{}_{\mathbf{X}}} and \tilde{\alpha}_{{}_{\mathbf{W}}}^{k} times the corresponding quantized activation and weight vectors. We depict all the possible quantization granularity options that support such efficient matrix multiplication in Appendix D .
Method
In this section, we begin by introducing our joint format and max value search, which establishes our strong baseline and already achieves state-of-the-art results at 8-bit and 6-bit quantization. Then we present an efficient pre-shifted exponent bias to tackle the catastrophic high inter-channel activation variance in transformer models and push the quantization limit to 4-bit.
Experiments
To validate the effectiveness of the proposed method, we conduct experiments on LLaMA Touvron et al. ( 2023 ) and BERT Devlin et al. ( 2019 ) models in 5.2.1 and Sections 5.2.2 . Further, in Section 5.2.3 we show that our method also generalizes well to vision transformer architectures. We present ablation studies on the calibration size and search range in Section 5.3 , and analyze the hardware costs of implementing FP operators in Section 5.4 .
Experiments Details
We adopt per-tensor quantization for activation and per-channel quantization for weight. We employ layer reconstruction following the settings of Yuan et al. ( 2022 ); Nagel et al. ( 2020 ) , and parallel quantization based on the approach outlined in Bai et al. ( 2022 ); Yuan et al. ( 2022 ) . A more detailed discussion regarding our implementation decisions can be found in Appendix F . For LLaMA models, we quantize all the weight and activation tensors in fully-connected layers for a fair comparison with previous work Xiao et al. ( 2022 ); Liu et al. ( 2023 ) . For BERT and ViT models, both fully-connected layers and activation-activation multiplication tensors in the self-attention module are quantized. Note that for FPQ on BERT Devlin et al. ( 2019 ) and ViTs models, the reconstruction metric Eq. 14 is substituted with a Hessian approximation loss metric. This substitution is further detailed in Appendix A .
LLM Zero-Shot Reasoning
We evaluate the effectiveness of FPQ for LLaMA-7B/ LLaMA-13B Touvron et al. ( 2023 ) on common sense zero-shot reasoning tasks. For the calibration data, we sample 32 random segments with 2048 tokens length from the C4 Raffel et al. ( 2020 ) dataset following the setting of GPTQ Frantar et al. ( 2023 ) . The data preprocessing and score calculation are based on EleutherAI evaluation harness 1 1 1 https://github.com/EleutherAI/lm-evaluation-harness . In Table 1 , we compare FPQ to the floating-point PTQ baselines, and state-of-the-art PTQ and QAT methods, including SmoothQuant Xiao et al. ( 2022 ) and GPTQ Frantar et al. ( 2023 ) , and LLM-QAT Liu et al. ( 2023 ) .

In general, all methods, except for the naïve MinMax INT Quantization, produce comparable outcomes in the 8-bit setting on both LLaMA-7B and LLaMA-13B. Additionally, we observe that the naïve MinMax FP Quantization achieves nearly lossless results and even surpasses the state-of-the-art integer post-training quantization method, SmoothQuant (Xiao et al., 2022), which indicates that floating-point quantization naturally has a strong capability in handling the distributions in transformers. However, both MinMax FP Quant and FPQ baseline fail when pushing the quantization precision to ultra-low 4/4/4 bit setting, with 28.9\% and 23.8\% accuracy degradation on LLaMA-7B, respectively. In this extreme case, the previous state-of-the-art PTQ and QAT methods, SmoothQuant Xiao et al. ( 2022 ) and LLM-QAT Liu et al. ( 2023 ) also suffer severe accuracy downgrade. In comparison, FPQ demonstrates a strong capability of handling extra-low bit settings and achieves only 8.2 / 5.8\% accuracy drop on LLaMA-7B/13B with 4/4/4 bit-width, outperforming SmoothQuant Xiao et al. ( 2022 ) by a large margin, yet with less bit-width and smaller calibration size. Moreover, FPQ even achieves 5.3% accuracy improvements compared to LLM-QAT Liu et al. ( 2023 ) in the 4/4/4 setting and 1.5% over GPTQ Frantar et al. ( 2023 ) in the 4/4/16 configuration on LLaMA-7B.

For practitioners, a crucial consideration is determining the appropriate quantization methods for various bit-widths. Therefore, based on our findings, we offer two recommendations that balance the trade-off between accuracy and search/optimization efficiency. First of all, since the difference between MinMax FP Quant and the rest of the methods is marginal for the 8/8/8 setting, we recommend simply using the MinMax FP Quant method for the 8/8/8 setting as the MinMax method does not involve search process. However, for more demanding scenarios, especially with activation quantization to 4 bits, we recommend employing FPQ for minimizing accuracy degradation with negligible inference overhead.
BERT Model
We evaluate the proposed quantization techniques for BERT model on GLUE tasks Wang et al. ( 2019 ) . Full-precision BERT-base models fine-tuned on GLUE datasets are obtained from Huggingface public repository 2 2 2 https://huggingface.co/textattack/bert-base-uncased-{TASK_NAME} . We randomly sample 128 data from the training set as the calibration set. In Table 2 , FPQ demonstrates remarkable performance, achieving absolute average accuracy improvements of 44.3\% compared to BrecQ Li et al. ( 2021 ) and 7.9\% over QDrop Wei et al. ( 2022 ) with 4/4/4 bit setting. Further, with 4-bit weight and 8-bit activation, MREM-S/MREM-P Bai et al. ( 2022 ) present a 1.6/1.5% accuracy gap to the full-precision model with 4096 calibration data, while FPQ achieves almost no accuracy loss with only 128 calibration data points.
Generalizability on Vision Transformer
Based on our findings that vision transformers also exhibit a consistent activation distribution pattern as language transformers, characterized by high inter-channel variance and low intra-channel variance, as detailed in Fig. 2 , we extended our proposed methods to ViT and compared FPQ with floating-point PTQ baselines and state-of-the-art PTQ method for ViT on the ImageNet classification task. Table 3 shows that findings on ViT are consistent with that on language models: previous state-of-the-art integer-based methods struggled to maintain reasonable accuracy when quantizing the transformer to lower bits. In comparison, the proposed FPQ outperformed both PTQ4ViT and APQ-ViT on 6 bits, and also achieved 40.9% and 31.5% absolute accuracy improvement over PTQ4ViT and APQ-ViT on DeiT-S in the 4-bit configuration.
Ablation Study
In this section, we first compare the influence of different calibration sizes on FPQ . We vary the calibration size in \{32,64,128,256\} and test on MNLI, QQP, and CoLA. Table 4 shows that the evaluation on MNLI and QQP is more robust to different settings, and the variance is more significant on CoLA. We observe that FPQ performs well with a calibration set size of 128 data points. However, we also find that it remains robust and maintains competitive accuracy even with limited access to calibration data, such as when using as few as 32 data points.

We investigate the robustness of FPQ to different search ranges (\gamma_{1},\gamma_{2}) . Table 5 presents the results of FPQ using three sets of (\gamma_{1},\gamma_{2}) : {(0.01,1.2),(0.1,1.2),(0.5,1.5)} , on MNLI, QQP, and CoLA. It is observed that no single search range outperforms the others consistently across all tasks. For instance, the search range (0.01,1.2) performs better than (0.5,1.5) on MNLI and QQP, but slightly worse on CoLA in the 4-bit configuration. Overall, FPQ exhibits robustness to various \gamma_{1} and \gamma_{2} , as long as the search range is not overly aggressive.
Hardware Cost
We further examine the hardware utilization of low-bit INT, FP, and mixed-format FP multiplication operators, including adder, multiplier, and multiply-accumulate (MAC) units, in terms of hardware area. Mixed-format FP refers to the multiplication of floating-point numbers with different formats, e.g. , E2M1 multiplies with E1M2. We implemented the MAC operator by Verilog HDL and utilized Cadence Genus to obtain the synthesized area under TSMC 40nm technology and 0.5GHz clock frequency.

Table 6 illustrates the hardware cost of the INT and FP operators, with the multiplier being the primary cost for INT and the adder for FP. Notably, the disparity between FP4 and INT4 adders is small, while INT has twice the hardware cost for the multiplier. Moreover, the mixed-format FP4 operator has comparable hardware area as the standard FP4 operator. These findings indicate that the proposed FPQ approach imposes negligible overhead in terms of hardware implementation when compared to the standard FP operators and the hardware cost for FP is comparable with INT.
Conclusion
This paper presents the first successful demonstration of 4-bit floating-point post-training quantization for weights, activations, and embeddings in natural language transformer architectures, including both large language models and BERT model. We also extend our method to vision transformers and observe its robust generalization ability. Our approach involves a practical search-based technique
which establishes a strong baseline and achieves state-of-the-art results for 6-bit and 8-bit quantization. Furthermore, we address the challenge of high inter-channel variance in transformers by proposing pre-shifted exponent bias , which proves highly effective in achieving accurate 4-bit quantization.
Acknowledgement
This research is supported by National Natural Science Foundation of China/ HKSAR Research Grants Council Joint Research Scheme under Grant NHKUST627/20 , and Foshan HKUST Projects under Grant FSUST21-HKUST10E .
Limitations
Our experiments were conducted on publicly available datasets with finite sentence lengths, and the generalizability of our method to extremely long sequences or streaming data has not been verified and may require further investigation. In addition, it remains to be seen how our proposed method can generalize to other domains beyond language and vision, such as audio. It would also be interesting to see the applicability of our method to generative tasks and other applications.
Figure 1: An illustration of floating-point (FP) quantization process using FP5 (E2M2) positive axis. The real-valued clipped X_{\rm R}^{\prime\prime} in Eq. 5 is rescaled by the real-valued scaling factor \tilde{\alpha} . Then, the quantization step-size v is determined by the range [2^{p},2^{p}+1) in which \frac{X_{\rm R}^{\prime\prime}}{\tilde{\alpha}} falls (Eq. 11 ). Here, p\in\{0,1,...,2^{e-1}\} is the exponent bit value. Lastly, X can be quantized to low-bit floating point values simply by X_{\rm{FP}}=\tilde{\alpha}\cdot v\cdot\left\lfloor\frac{X_{\rm R}^{\prime\prime}}{\tilde{\alpha}\cdot v}\right\rceil (Eq. 8 ).; An illustration of floating-point (FP) quantization process using FP5 (E2M2) positive axis. The real-valued clipped X_{\rm R}^{\prime\prime} in Eq. 5 is rescaled by the real-valued scaling factor \tilde{\alpha} . Then, the quantization step-size v is determined by the range [2^{p},2^{p}+1) in which \frac{X_{\rm R}^{\prime\prime}}{\tilde{\alpha}} falls (Eq. 11 ). Here, p\in\{0,1,...,2^{e-1}\} is the exponent bit value. Lastly, X can be quantized to low-bit floating point values simply by X_{\rm{FP}}=\tilde{\alpha}\cdot v\cdot\left\lfloor\frac{X_{\rm R}^{\prime\prime}}{\tilde{\alpha}\cdot v}\right\rceil (Eq. 8 ).
Figure 2: Magnitude of the output activations of the feed-forward network blocks in LLaMA-7B, BERT, and DeiT.; Magnitude of the output activations of the feed-forward network blocks in LLaMA-7B, BERT, and DeiT.
Figure 3: Overview of pre-shifted exponent bias method : (a) Search phase : The real-valued channel-wise scaling exponent bias for activations ( \tilde{\mathbf{b}}_{j} ) is partitioned into a real-valued tensor-wise exponent bias ( \rho ), and the integer-based channel-wise exponent bias ( \tilde{\mathbf{b}}^{ori}_{j} ). (b) Reparameterization and weight pre-computation : Once the optimal values are determined on the calibration set, \tilde{\mathbf{b}}^{ori}_{j} are re-parameterized into the weight tensor. The weights are pre-computed to apply the bias, therefore this is a one-time cost. (c) Inference phase : The method leverages efficient matrix multiplication between low-bit floating-point matrices.; Overview of pre-shifted exponent bias method : (a) Search phase : The real-valued channel-wise scaling exponent bias for activations ( \tilde{\mathbf{b}}_{j} ) is partitioned into a real-valued tensor-wise exponent bias ( \rho ), and the integer-based channel-wise exponent bias ( \tilde{\mathbf{b}}^{ori}_{j} ). (b) Reparameterization and weight pre-computation : Once the optimal values are determined on the calibration set, \tilde{\mathbf{b}}^{ori}_{j} are re-parameterized into the weight tensor. The weights are pre-computed to apply the bias, therefore this is a one-time cost. (c) Inference phase : The method leverages efficient matrix multiplication between low-bit floating-point matrices.
Figure 4: Quantization error of different formats for BERT layers.; Quantization error of different formats for BERT layers.
Figure 5: Magnitude of the output activations of different modules in BERT (left column), and DeiT-S (right column).; Magnitude of the output activations of different modules in BERT (left column), and DeiT-S (right column).
Figure 6: Magnitude of the output activations of different modules in LLaMA-7B.; Magnitude of the output activations of different modules in LLaMA-7B.
Figure 7: Quantization granularity options that support efficient matrix multiplication. The dimensions that share the same scaling factor are indicated with red dotted frames; Quantization granularity options that support efficient matrix multiplication. The dimensions that share the same scaling factor are indicated with red dotted frames
Zero-shot performance on common sense reasoning tasks with LLaMA Touvron et al. ( 2023 ) models. We denote E/W/A as the bit-width of word embeddings, model weight and activations, respectively.
Results on the GLUE development set with BERT Bai et al. ( 2022 ) model. We denote E/W/A as the bit-width of word embeddings, model weight and activations, respectively.
Comparison on the ImageNet dataset with vision transformer structures.
Ablation studies of different calibration sizes.
Ablation studies of different search range.
Area differences of INT, FP and mixed Format FP operators across different bit-widths.