LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Authors: Shih-yang Liu
Zechun Liu
Xijie Huang
Pingcheng Dong
Kwang-Ting Cheng
Venue: EMNLP 2023 Main (2023)
Keywords: Model Compression
Model Quantization
TL;DR: 4-Bit Floating-Point Quantized Transformers

Abstract: We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values, in a post-training manner. Existing post-training quantization (PTQ) solutions are primarily integer-based and struggle with bit widths below 8 bits. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail or bell-shaped distributions, and it has emerged as a default choice in many hardware platforms. One characteristic of FP quantization is that its performance largely depends on the choice of exponent bits and clipping range. In this regard, we construct a strong FP-PTQ baseline by searching for the optimal quantization parameters. Furthermore, we observe a high inter-channel variance and low intra-channel variance pattern in activation distributions, which adds activation quantization difficulty. We recognize this pattern to be consistent across a spectrum of transformer models designed for diverse tasks such as LLMs, BERT, and Vision Transformer models. To tackle this, we propose per-channel activation quantization and show that these additional scaling factors can be reparameterized as exponential biases of weights, incurring a negligible cost. Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63.1 on the common sense zero-shot reasoning tasks, which is only 5.8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12.7 points. Code is available at: https://github.com/nbasyl/LLM-FP4.

LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Shih-yang Liu"l, Zechun Liu"?, Xijie Huang', Pingcheng Dong', Kwang-Ting Cheng'
'Hong Kong University of Science and Technology. Meta Reality Labs {sliuau, xhuangbs, wrngcheng.domge. @ cos.nect.ust.hk zechunliu @ meta.com timcheng ust.hk
Abstract
Toutanova, 2019; Lewis al., 2020). The trans- formative impact of the transformer has been fur- ther propelled by the emergence of models like GPT (Brown et al., 2020; OpenAI, 2023), cata- pulting the popularity of this architecture to new heights. Meanwhile, the versatility of transformers extends beyond NLP, encompassing diverse do- mains such vision (Dosovitskiy al.; Touvron et al., 2021), audio (Akbari et al., 2021), etc. This trend towards a unified urchitecture for different modalities represents a groundbreaking develop- ment within the realm of deep learning.
We propose LLM-FP4 for quantizing both weights and activations in large language mod- els (LLMs) down to 4-bit floating-point val- ues, in a post-training manner. Existing post- training quantization (PTQ) solutions pri- marily integer-based and struggle with bit widths below bits. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail bell-shaped distributions, and it has emerged a default choice in many hardware platforms. One characteristic of FP quantization is that its performance largely depends on the choice of exponent bits and clipping range. In this re- gard, we construct a strong FP-PTQ baseline by searching for the optimal quantization pa- rameters. Furthermore, we observe a high inter- channel variance and low intra-channel vari- ance pattern in activation distributions, which adds activation quantization difficulty. We rec- ognize this pattern to be consistent across spectrum of transformer models designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models. To tackle this, propose per-channel activation quantization and show these additional scaling factors can repa- rameterized as exponential biases of weights, incurring negligible cost. Our method, for the first time, can quantize both weights and acti- vations in the LLaMA-133 1BB to only 4-bit and achieves an average score of 63. on the com- mon sense zero-shor shot reasoning tasks, which is only 5.8 lower than the full-preed full-precision model, significantly outperforming the previous state- of-the-art by 12.7 points. Code is available at: https:///ithhu cww/ Conm/nbasylLLLM-FFP
However, the advancements transformer per- formance are accompanied by a corresponding in- crease in model size and computational costs (Ka- plan et al., 2020). This poses significant challenges when attempting to leverage the full potential of transformer models in use cases where memory or computational resources are limited. Despite the extensive research and widespread adoption of transformers, the field of transformer compres- sion remains relatively underexplored. To address this gap, our study focuses on the compression of transformers, especially through floating- point post-training quantization techniques.
Post-training quantization (PTQ) offers the ad- vantages of simple to use with minimal fine-tuning requirements (Nagel et al., 2020; Cai et al., 2020). Existing PTQ solutions for transformers primar- ily focus on integer (INT) quantization (Liu et al.. 2021; Yuan et al., 2022), which can be effective in certain scenarios but often break down when bit widths are below 8 bit. On the other hand, floating- point (FP) quantization has gained significant trac- tion as a more flexible alternative, capable better accommodating various activation and weight dis- tributions. In fact, FP8 has emerged as the default choice in various hardware platforms, including the NVIDIA H100.
1 Introduction
Since the introduction of transformer architec- ture (Vaswani et al., 2017), transformers have superseded recursive neural networks, emerging the dominant architecture in numerous natu- ral language processing (NLP) tasks (Kenton and
Different from integer (INT) quantization, a par- ticular challenge in doating-point (FP) quantiza-
'These authors contributed equally to this work

---

tion is how to select appropriate exponent bits and scale parameters. Improper parameter choices can lead to subpar or divergent quantization results. To tackle this challenge, we introduce a robust recipe for FP quantization, which leverage layer-wise re- construction to jointly search for optimal exponent bits and maximum values. Compared to previous approaches that utilize gradient updates for expo- nent bits (Kuzmin et al., 2022), our search-based method proves to be more stable and consistently delivers desirable quantization results, which estab- lishes a strong baseline for FP-PTQ.
We further extend our method BERT and vi- sion transformers. surpasses the previous best 4- bit quantized BERT by 7.8 points on GLUE dataset and achieves 31.4 points higher accuracy compared to the previous SoTA ViT quantization method for 4-bit DeiT-S on ImageNet dataset.
2 Related Works
2.1 Post-Training Quantization
Model quantization can be mainly categorized into quantization-awar training (QAT) and post- training quantization (PTQ), depending on whether it involves additional training for weight fine- tuning or not. Most PTQ studies are primarily focused on convolutional neural networks (CNNs) (Nagel al., 2020; Li al., 2021; Wu et al., 2020; Cai et al., 2020; Nagel et al., 2019). However, with mod-
Furthermore, our investigation uncovers an in- triguing pattern of activation distributions in trans- formers, characterized by high inter-channel vari- ance and low intra-channel variance. Similar pat- terns are also observed in previous works (Xiao et al., 2022; Dettmers et al., 2022), while we argue that this pattern is inherent to transformer architec- tures and not limited to specific tasks, observed consistent patterns not only in large lan- guage models but also in BERT model and even vision transformers. Motivated by these findings, we introduce a novel pre-shifted exponent bias for FP quantization of transformers. Concretely, we leverage the per-channel activation variance com- puted from calibration data and reparameterize these scales the exponential bias of the corre- sponding FP quantized weight vectors. This ap- proach effectively addresses the challenge posed by high inter-channel variance while incurring neg- ligible computational cost.
the growing popularity of transformer- els, only limited number of works (Bondarenko et al., 2021; Yuan et al., 2022: Ding et al., 2022) have been conducted to realize PTQ on transform- ers. Moreover, the existing works primarily focus on visual transformer models and exhibit inferior performance when the bit width is below 8. There- fore, in this work, we delve into the challenges of the low-bit PTQ for language transformers.
have
2.2 Floating-Poii Point Quantization
Floating-poin (FP) quantization has emerged as a promising alternative to integer quantization due to its ability to handle long-tail distributions, and offers increased flexibility (Kuzmin et al., 2022). Additionally, modern GPUs such as H100 (Micike- vicius et al., 2022) now support FP quantization. Nonetheless, minimal research has been conducted on FP quantization. Only (Kuzmin et al., 2022) pro- poses general FP8 quantization scheme primarily for vision tasks, and (Zhang et al., 2023) adopts a mixture of FP and INT formats quantization for LLMs. In this work, we propose FPQ baseline as a general guideline for low-bit floating-poit PTQ to compress language transformer models.
In summary, we study floating-poin post- training quantization (PTQ) for transformer archi- tectures, and the contribution of this paper includes:
We propose a search-based framework for de- termining the optimal exponent bias and maximal quantization value. This method outperforms ex- isting techniques in terms of stability and perfor- mance, establishing strong baseline for floating- point post-training quantization.
We propose a novel technique, shi-siifted expo- nent bias, which effectively addresses the challenge of high inter-channel variance in the transformer with negligible computational overhead.
3 Preliminaries
3.1 Formulation of Floating-Poin Variables
Experimental results demonstrate that the pro- posed method yields the first usable FP4 weight and activation quantized LLaMA-131 model with mere 5.8-point degradation in zero-shot reasoning tasks against the full- precisonn model, reducing the gap by ~70% compared to the previous SoTA.
standard floating- point number is represented as:
d1 dg
dm
XFP = (-1)²2" 6 11
(1)
2
22
2m
where E {0, 1} is the sign bit. d; E (0, 1} is tth mantissa bit, m denoted number of mantissa bits.

---

Xg when < z
[xg
when logz
2
1xg
2
loga
loga [x
logo (2a1)
2
X
Qmax (2 2-"1)2 11= 14
Figure I: An illustration of floating-poin (FP) quantization process using FP5 (F2M2) positive axis. The real-valued clipped X in Eq. 5 is rescaled by the real-valued scaling factor Then, the quantization step-size e is determined
by the range [2P, 2 1) in which Xg falls (Eq. 9). Here, E {0. 1..,,.... 2 is the exponent bit value. Lastly, X can be quantized to low-bit floating point values simply by Xpp = ā 0.
Xa
(Eq. 8).
p is an integer [0, 2 1], and e denotes number exponent bits. is am integer exponent bias. A floating point with j number exponent bits and k mantissa bits is denoted as FP format EjMk.
Note that we combine the tensor- wise real-valued scaling factor a with integer exponent bias to form new scaling factor à = 2 2- Here ô denotes relaxed tensor-wise real-valued exponent, and we can derive from the desired clipping value Qmaz from Eq. as:
3.2 Floating-Point Quantization Process
In integer quantization, the real-valued variable XR is quantized to an integer XINT with the following formula:
2 log2amms + log, -2-2-
-- (7)
(2) Compare and quantize. Different from inte- ger quantization, which simply utilizes the round- ing function to convert the real-valued variables to quantized ones, in floating-poins quantization, there is an additional step of comparing X with quantization levels and then quantize:
XR
XINT = Clip
Qmin Qmar
(2)
where 1.1 is the rounding function. XR is the real-valued variable, a represents the full-precision scaling factor, and Qmin, Qmer are the min/max value of the quantization range. Similarly, real- valued variable XR can be converted to floating- point XFP in two steps.
X
XFP = ā. v
(8)
ā
(1) Scale and clip. In FP quantization, we also scale and clip the real-valued variable before quan- tization as:
where X is clipped real-valued variable (Eq. 5), is the tensor-wise doating-point scaling factor, and v is an integer power of 2.
X Clip(XR, Qmin: Qmax)
2 20g [[]b]] 2 m
|loga | + otherwise
where the min/max value range of signed floating- point quantization can be calculated from 1:
(9)
Here we select the quantization level v according to the magnitude of Xi which equals to X 25. Then the floating triiingg-onnttoe quantized variables can be derived with Eq.8. The illustration of the quantiza- tion process is in Fig. 1, detailed explanation can also be found in (Micikevicius et al., 2022).
Qmar -0-mii min = (2 2 2--2" b-1
(4)
a
Here the integer exponent bias is another ad- justable hyperparameter controlling Qmar and Qmin- which has similar functionality as a. There- fore, for simplicity, we reformulate Eq. as:
X = Clip XR: Qmin: Qmearr
(5)
3.3
Floating- Point Matrix Multiplication
With the floating-pon quantized variables, the ma- trix multiplication formulated as:
where
Qmac aQmaz a (2 2-
= a 2 b (2 - 2 m)2"
0
FP ā á W.kk (10)
= b, (2 2 2 0-1
Here in per-tensor activation quantization and per- channel weight quantization, X denotes 4th row
(6)

---

in the activation matrix and W denotes hthh col- FP umn in the weight matrix, such that each element O in the output matrix is computed by the prod- uct of two vellev scalars 0x and o times the corresponding quantized activation and weight vectors. We depict all the possible quantization granularity options that support such efficient ma- trix multiplication in Appendix D.
method, details can be found in Appendix E. In- stead, we propose a search-based algorithm that jointly determines the optimal format and its asso- ciated clipping range to address this challenge.
The searching process is conducted layer by layer with the metric of minimizing Eq. 12. The output of matrix multiplication corresponding to each sub-module is denoted as O XY, where Y can be either a weight tensor W or another acti- vation tensor.
4 Method
The search space of q-bit FP format includes all formats except for the format with an exponent bit equal to 0, as the quantization of the format with an exponent bit equal to 1 already degenerates to INT quantization. We search for the ponent bias b, which equals to the logarithm of the scaling factor. We initialize bx and by from Eq. 7 with Qmaz equals the maximum value of [Xn] and YR.. respectively. We then define the search space of xx and by by linearly dividing [7, Jinät 72 bmit and ly, Eonit, 72 (2 into & intervals, where 7, and Y/2 are empirically set to 0.01 and 1.2, and k: = 100. The search process is outlined in Alg. We search the quantization scheme in all the matrix multiplication layers in parallel following (Yuan et al., 2022: Bai et al., 2022). The algorithm can be divided into two parts. (1) Do forward propagation to store the intermediate raw output of each layer (2) Iteratively update the optimal format and bi- ases for each layer for three rounds by minimizing the reconstruction metric (Eq, 12). We name this search-based framework as Floating Point Quanti- zation Baseline (FPQ baseline), and it can already achieve state -tate-of-the-art results on both 8-bit and 6- bit settings.
In this section, we begin by introducing our joint format and max value search, which establishes our strong baseline and already achieves state-of- the-art results at 8-bit and 6-bit quantization. Then we present an efficient pre-shifted exponent bias to tackle the catastrophic high inter-channel activa- tion variance in transformer models and push the quantization limit to 4-bit.
ex-
4.1 Joint Format and Max Value Search
The objective of post-training quantization is to minimize the perturbation (6X = XFP - XR) introduced by quantization to the pre-trained real-
valued network:
min E[C(XR + oX) - [(XR)
(11)
In this study, we adopt the setting presented in (Choukroun al., 2019; Wu et al., 2020), which assumes a positive correlation between the change in the intermediate output of the quantized model and Eq. 11. Therefore, minimizing the distance between the intermediate output of the quantized layer (Ô) and the output of the original layer (O) leads to minimize Eq. 11. Hence, the objective loss metric is formulated as:
4.2 Pre-Shifted Exponent Bias
min (Ô O0-
(12)
In transformer architectures, we observed an in- triguing phenomenon of high inter-channel vari- ance. As shown in Fig.2, the magnitudes of values within the same channel are close to each other but exhibit significant differences across different channels. This phenomenon is not only observed in language models (i.e., LLaMA and BERT) but also significant in vision transformer models. Since outlier channels are often orders of magnitude big- ger than the rest, they will dominate the quantiza- tion precision of the quantized tensor, resulting in less representation capacity for those channels with smaller magnitudes (Xiao et al., 2022). This makes tensor-wind wise or token-wise scaling factor insufficient for accurate activations quantization.
which is used search for the optimal FP quantiza- tion function in the following proposed framework.
The challenges in FP quantization arise from its sensitivity to the quantization format and clipping range. Undesirable format selection will result in a catastrophic error rate. In addition, we observe that the optimal clipping range varies depending on the format used. Previous work (Kuzmin et al., 2022) on floating-point (FP) quantization-aware training (QAT) proposed learn both the IPP format and maximum value with gradients. However, we find this method suffers from over-fitting in PTQ, with accuracy being even worse than naive MinMax

---

Algorithm FPQ baseline
LLaMA
BERT
Input: Calibration dataset, Full-preeision Model M, Quantization format search space Rx (e.g. Rx (E3MO, E2M1,,
for FP4), number of round
Y 3,
2: Output: FP Quantized model
3: for I in 1 to 1.th layer in M do
4:
Forward & collect raw output O' = X'y' of layer 5: end for
Chacoll
6: for in se to 1 layer in M do
7: Initialize the FP format search space w.r.t X and y
and Ry fry. " ry..l 8: Initialize bias 6g by with Eq.7 for each format can-
as Rx fr>x "xx .... "x....
didate r'x E Rx and r Ry-.
9:
Generate search space of 6x in formats to be by: fanse .%2 "r and by to be [r, (7,8& Kémit -2
10:
for to
11:
Scarch for by w.r.t each x that minimizes Eq.12
12:
Scarch for rxx E Hx that minimizes 12
13:
Search for by w.r.t each rt that minimizes Eq.12
Figure 2: Magnitude of the output activations of the feed-forward network blocks LLaMA-7B, BERT, and DeiT.
14:
Search for ry E Ry that minimizes Eq.
15: end
16:
end
values:
However, applying per-channel scaling factors for activations poses challenges to efficient matrix multiplication, because the scaling factor is not a shared constant along the multiplication direc- tion and cannot be extracted Lqq 10. To address this challenge, introduce pre-shifted exponent bias, which allows calculate per-channel scal- ing factors from activations. These scaling factors are then re-parameterized as the exponent biases of the corresponding weights. This method effec- tively handles high inter-channel variance while maintaining nearly identical efficiency to per-tensor quantization.
by = 2 1og X Hog, 1-1---11------ (14)
Here X denotes the j channel in the activation matrix. Then we separate b tensor-wise real- valued scaling factor plus a channel-wise integer scaling factor:
b = ò bori
(15)
= p [.tp((bbp,,,,2-0
where ` C R.. bori c Z. Then the formula for one of the entries in the jth channel of X can be rewrote as follows:
Recalling in Eq. 7, we extracted the tensor-wise integer exponent bias and times it with real- valued scaling factor a and becomes new scaling factor ã = 2 2 2 a. Then, the floating point quantization formula in Eq. 13 becomes:
XFP -2 (-1))22 0 di
dm
2m
=2 --12(-1)- byri (1
di
dm
2m
(16)
Note that the bias bori is constrained to integers within [0, 2" 11, compatible with the standard floating-point number calculation. Nevertheless, adding different biascs for each channel during inference may still cause some extra hardware operations. Thus, we re-parameterized the per- channel activation bias into weight tensor and pre-computed the weights using the calibration set. This way, the exponent biases shifting only hap- pens in the calibration stage. Then, an element in jth channel of activation tensors X becomes:
Xpp - --1)2-0(11)-2-- di d2
dm
(13)
2
2m
We note that after the bias is absorbed in the scal- ing factor, the original bias term (bort) in the FP formula is always zero. In dealing with the inter- channel variance, we devise an innovative usage of this integer exponent bias: we set it to be a per- channel variant (bori- c ZZ).
Then the calculation of the channel-wise integer bias vector (bor)) is very straightforward. We first calculate the initial per-channel real-valued scal-
XFP = 2 (1+ d1 +...+ dm
ing factor (2 by) from the per-channel maximum
(17)
2

---

b.
Pre-compute weights to store by in low-bit format
Tensor-wise scaling factor P (Eq. 17)
Decompase b (Eq.
X
W
W
W
W
X
%2,1
2
W
2,1
W 2.
by
b
Pre-shifted exponent blas bg" (Eq.
Efficient matrix multiplization Ry 2- Ciyy 2--/ (Eq. 22)
W..
W W.
x
X
W
1lth b
R
Wai
W.2
b
P ER
W 2,1
aa)
(c)
Figure 3: Overview of pre-shifted exponent bias method: (a) Search phase: The real-valued channel-wise scaling exponent bias for activations (b,) is partitioned into a real-valued tensor-wise exponent bias (p), and the maseer-bbased channel-wise exponent bias (bşri). ). (b) Reparameterization and weight pre-computation: Once the optimal values are determined on the calibration set, bigr are re-parameterized into the weight tensor. The weights are pre-computed to apply the bias, therefore this is one-time cost. (c) Inference phase: The method leverages efficient matrix multiplication between low-bit foating-poine matrices.
and the corresponding weight element in jth row of the weight tensor W becomes:
and search range in Section 5.3, and analyze the hardware costs of implementing FP operators in Section 5.4.
d1
dm
Wep 11pp2--b (-1)$2p (-1)22 birh (1+
(18)
2
2m
5.1 Experiments Details
As result, efficient matrix multiplication in Eq.10 is reformulated as:
We adopt per-tensor quantization for activation and per-channel quantization for weight. We employ layer reconstruction following the settings of (Yuan et al., 2022; Nagel et al., 2020), and parallel quanti- zation based on the approach outlined in (Bai et al.. 2022; Yuan et al., 2022). A more detailed discus- sion regarding our implementation decisions can be found in Appendix F. For LLaMA models, we quantize all the weight and activation tensors in fully-connected layers for fair comparison with previous work (Xiao et al., 2022; Liu et al., 2023). For BERT and ViT models, both fully-connected layers and activation-activatiio multiplication ten- sors in the self-attention module are quantized. Note that for FPQ on BERT (Devlin et al., 2019) and VITs models, the reconstruction metric Eq. 12 is substituted with Hessian approximation loss metric. This substitution further detailed in Ap- pendix A.
i.k out Pp Fp = ãx t Xi: ( (BC ) (19)
X
with
where @ is the element-wiss multiplication, & FP) be pre-calculated and stored in low-bit FP format. We depict the
2-b and (B @ wish)
overall pre-shifted exponent bias method in Fig.3. This method applies to quantizing all the fully- connected layers. During the search process, we initialize 7x as the miny(b;). Then, we fixed bx to be the bias calculated from the Eq. 14 and search for the optimal 7x from 1y, init fmit 72px initj. Combining pre-shifted exponent bias method with the joint format and max-value search framework(FPQ baseline), we name our method as (FPQ), short for Floating Point Quantization.
5 Experiments
5.2 Main Results
To validate the effectiveness of the proposed method, conduct experiments LLaMA (Tou- vron al., 2023) and BERT (Devlin al., 2019) models in 5.2.1 and Sections 5.2.2. Further, in Section 5.2.3 we show that our method also gen- cralizes well to vision transformer architectures. We present ablation studies on the calibration size
5.2.1 LLM Zero-Shot Reasoning
We evaluate the effectiveness of FPQ for LLaMA- 7B/ LLaMA-13B (Touvron et al., 2023) on com- mon sense zero-shot reasoning tasks. For the cali- bration data, we sample 32 random segments with 2048 tokens length from the C4 (Raffel et al., 2020)

---

Quant Method
#Bits (E/WA) # Calib BoolQ
PIQA
HellaSwag
WinoGrande ARC- ARC-c Avg.
LLaMA-7B Full-precc precision
16/16/16
75.1
78.7
56.9
69.9
75.3
MinMax INT Quant
8/8/8
64.3
66.8
40.5
57.4
59.0
29.6
52.9
MinMax FP Quant (E4M3)
8/8/8
32
74.9
78.6
56.8
69.5
75.5
41.6
66.1
SmoothQuant (Xiao et al., 2022)
16/8/
74.0
77.5
55.0
69.6
74.4
37.4
64.6
FPQ baseline
8/8/8
75.8
78.3
55.9
69.5
75.6
41.3
66.1
FPQ
8/8/8
32
75.6
78.2
56.6
70.2
74.6
40.7
66.0
MinMax INT Quant
4/4/16
32
64.1
76.
51.6
66.3
72.4
40.0
61.7
MinMax Quant (E2MI)
4/4/16
73.0
77.9
55.2
69.1
73.6
40,9
64.9
GPTQ (Frantar ct al., 2023)
4/4/16
128
73.3
77.9
54.9
67.9
72.7
37.4
64.0
FPQ baseline
4/4/16
77.9
55.6
75.2
FPQ
4/4/16
32
74.2
77.8
55.8
69,9
74.9
40.4
65.5
MinMax INT Quant
4/4/8
32
50.4
56.5
27.9
46.5
36.1
21.2
39.7
MinMax FP Quant (E2M1/E4M3)
4/4/8
73.0
77.5
55.0
69.3
73.6
40,9
FPQ baseline
4/4/8
32
75.0
77.6
55.9
69.9
74.3
39.4
65.3
FPQ
4/4/8
32
75.0
77.7
55.5
69.8
74.5
39,9
65.4
MinMax INT Quant
4/4/4
54.1
51.7
25.6
49.8
24.7
22.9
38.1
MinMax Quant (E2MI)
4/4/4
47.3
53.1
25.7
50.7
25.1
22.4
37.4
SmoothQuant (Xiao et al., 2022)
16/44
512
54.1
62.8
41.5
52.6
50.6
32.9
49.1
L.M-QAT (Liiu al., 2023)
16/4/
(QAT)
63.5
64.3
55.6
52,9
30.2
52.8
FPQ baseline
4/4/4
32
57.4
56.6
30.2
51.1
37.7
23.2
42.7
FPQ
4/4/
64.2
73.5
47.8
63.7
65.9
33.6
1.I.MA-13B Full- Full-precision
16/16//6
77.9
79.2
59.9
72.6
77.4
46.4
68.9
MinMax INT Quant
8/8/8
32
60.6
69.6
46.0
61.5
63.3
32.8
55.6
MinMax FP Quant (E4M3)
8/8/8
78.0
79.1
60.0
72.3
77.2
47.1
68.9
SmoothQuant (Xiao
2022)
16/8/8
512
76.5
78.0
58.0
72.1
76.3
45.5
68
FPQ baseline FPQ
8/8/8
32
78.0
79.1
59.9
72.3
77.2
47.1
68.9
8/8/8
78.5
72.4
MinMax INT Quant
4/4/8
52.1
65.0
36.4
53.9
52.3
29.0
48.1
MinMax FP Quant (F2M1/FAM3)
4/4/8
32
78.0
78.9
58.0
71.6
76.0
44.8
67.9
FPQ baseline FPQ
4/4/8
76.2
78.2
57.9
71.9
75.1
43.9
67.2
4/4/8
32
76.4
78.5
58.2
72.1
75.2
44.7
67.5
MinMax INT Quant
4/4/4
32
54.5
52.7
25.5
51.1
25.3
22.1
38.5
MinMax Quant (F2MI)
4/4/4
45.8
51.7
25.5
49.5
25.0
22.8
SmuothQuant (Xiao 2022)
16/4/4
512
57.6
61.3
56.0
52.6
49.9
25.1
50.4
FPQ baseline
4/4/4
32
54.3
57.7
35.7
52.2
41.1
25.7
44.5
FPQ
4/4/4
71.9
74.8
53.3
66.7
71.7
39.9
63.1
Table Zero-shot performance on common sense reasoning tasks with LLaMA (Touvron et al., 2023) models. We denote E/WIA the bit-width of word embeddings, model weight and activations, respectively.
dataset following the setting of GPTQ (Frantar et al., 2023). The data preprocessing and score calculation are based EleutherAl evaluation har- ness!. In Table 1, we compare FPQ the floating- point PTQ baselines, and state-of-the-aar PTQ and QAT methods, including SmoothQuant (Xiao et al., 2022) and GPTQ (Frantar et al.. 2023), and LLM- QAT (Liu et al., 2023).
tization precision to ultra-low 4/4/4 bit setting, with 28.9% and 23.8% accuracy degradation on LLaMA-7B, respectively. In this extreme case, the previous state-of-the-arr PTQ and QAT meth- ods, SmoothQuant (Xiao et al., 2022) and LLM- QAT (Liu et al., 2023) also suffer severe accu- racy downgrade. In comparison, FPQ demonstrates strong capability of handling extra-low bit set- tings and achieves only 8.2/5.8 3/% accuracy drop on LLaMA-7B/13B with 4/4/4 bit-width, outperform- ing SmoothQuant (Xiao et al., 2022) by large margin, yet with less bit-width and smaller calibra- tion size. Moreover, FPQ even achieves 5.3% accu- racy improvements compared to LLM-QAT (Liu et al., 2023) in the 4/4/4 setting and 1.5% over GPTQ (Frantar et al., 2023) in the 4/4/16 configu- ration on LLaMA-7B. For practitioners. a crucial consideration is determining the appropriate quantization meth- ods for various bit-widths. Therefore, based on our findings, we offer two recommendations that balance the trade-off between accuracy and
general, methods, except for the naïve Min- Max INT Quantization, produce comparable out- comes in the 8-bit setting on both LLaMA-7BB and LLaMA-13B. Additionally, we observe that the naïve MinMax FP Quantization achieves nearly lossless results and even surpasses the state-of- the-art integer post-training quantization method, SmoothQuant (Xiao et al., 2022), which indicates that floating-poin quantization naturally has strong capability in handling the distributions in transformers. However, both MinMax FP Quant and FPQ baseline ine fail when pushing the quan-

---

Quant Method
#Bits (E/W/A) # Calib MNLI...
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg.
(Full -reli-precision)
32-32-32
84.9
91.4
92.1
93.2
59.7
90.1
86.3
72.2
83.7
MinMaa INT Quant
8/8/8
128
77.0
89.9
88.9
92.9
51.8
88 2
83.8
71.5
80.5
MinMax HP Quant (E2255)
8/8/8
128
78.9
90.8
88.6
92.9
52.7
88
84.3
69.0
80.7
MinMax FP Quant (E3M4)
8/8/8
128
84.5
90.9
91.5
93.2
58.3
89.3
87.7
71.8
83,4
MinMax HP Quant (E4M3)
8/8/8
128
84.7
90.9
91.7
93.0
58.6
89.3
86.5
72.2
83.4
MinMax FP
Quant (ESM2)
8/8/8
128
84.1
91.4
58.1
87.5
71.8
83.3
FPQ baseline
8/8/8
128
84.6
90.9
91.7
93.1
58.6
89.3
88.0
72.2
83.5
FPQ
8/8/8
128
84.6
91.0
91.6
93.3
58.8
89.3
88.0
72.2
83.6
MinMax INT Quant
6/6/6
128
31.9
62.0
52.8
58.8
0.0
12.7
32.1
52.7
37.9
MinMax PP Quant (E22M3)
6/6/6
128
43.5
85,4
79.4
90.5
45.2
86.0
66.9
$9.9
69.6
MinMax FP Quant (E3M2)
6/6/6
128
83.9
90.8
90.8
92.2
58.2
88.6
87.0
72.2
83.0
MinMax FP Quant (F4MI)
6/6/6
128
84.4
90.1
92.2
58.2
89.2
85.3
69.7
82.4
FPQ baseline
6/6/6
128
84.6
90.9
91.2
93.2
58.8
88.7
87.5
70.8
83,2
FPQ
6/6/6
128
84.5
90.8
91.6
93.1
57.3
89.3
88.7
71.8
83.2
MinMax INT Quant
4/4/8
128
33.1
63.8
60.1
49.3
0.0
44.0
50.2
49.1
43.7
MinMax JP Quant (E2M1)
4/4/8
128
60.6
70.9
77.4
79.9
5.5
78.6
46.8
56.6
59.5
MREM-S (Bai et al., 2022)
4/4/8
4096
83.5
90.2
91.2
91.4
55.1
89.
84.8
71.8
82.1
MREM-P (Bai al., 2022)
4/4/8
83.4
91.0
54.7
86.3
71.1
82.2
FPQ baseline
4/4/8
128
84.4
90.6
91.4
92.9
58.6
83.7
88.2
73.3
82.9
FPQ
4/4/8
128
84.5
90.6
91.1
92.7
58.8
89.3
88.7
73.3
83.6
MinMax INT Quant
4/4/4
128
31.8
39.7
50.5
49.1
0.0
6.7
31.6
$4.5
32.9
MinMax FP Quant (E2MI)
4/4/4
128
33.6
54.0
50.6
0.0
31.6
52.0
34.1
BrecQ (I. et al., 2021)
R4/4
4096
31.9
62.3
50.7
50.9
0.9
6.4
31.7
52.3
35.8
Drop (Wei al., 2022)
8/4/4
4096
71.4
79.0
76.8
40.9
81,9
79.2
60.7
72.3
FPQ baseline
4/4/4
128
38.9
68.3
55.3
83.6
10.6
0.0
43.8
55.2
44.5
FPQ
4/4/4
128
82.3
89.2
86.6
91.5
52.6
85.5
83.8
69.0
80.1
Table 2: Results on the GLUE development set with BERT (Bai et al., 2022) model. We denote E/W/A as the
bit-width of word embeddings, model weight and activations, respectively.
scarch/optimization efficiency. First of all, since the difference between MinMax FP Quant and the of the methods is marginal for the 8/8/8 set- ting, we recommend simply using the MinMax FP Quant method for the 8/8/8 setting as the MinMax method does not involve search process. However, for more demanding scenarios, especially with ac- tivation quantization to 4 bits, we recommend em- ploying FPQ for minimizing accuracy degradation with negligible inference overhead.
128 calibration data points.
5.2.3 Generalizability on Vision Transformer
Based on our findings that vision transformers also exhibit consistent activation distribution pattern as language transformers, characterized by high inter-channel variance and low intra-channel vari- ance, detailed in Fig. 2, we extended pro- posed methods to ViT and compared FPQ with floating-point PTQ baselines and state-of-the-art PTQ method for ViT on the ImageNet classifica- tion task. Table shows that findings on ViT are consistent with that on language models: previous state-of-the-art integer-bass based methods struggled to maintain reasonable accuracy when quantizing the transformer to lower bits. In comparison, the pro- posed FPQ outperformed both PTQ4ViT and APQ- ViT on bits, and also achieved 40.9% and 31.5% absolute accuracy improvement over PTQ4ViT and APQ- ViT on DeiT-S in the 4-bit configuration.
5.2.2 BERT Model
We evaluate the proposed quantization techniques for BERT model on GLUE tasks (Wang et al., 2019). Full-prceision BERT-basc models finc- tuned on GLUE datasets are obtained from Hug- gingface public r We randomly sample 128 data from the training set as the calibration set. In Table 2, FPQ demonstrates remarkable perfor- mance, achieving absolute average accuracy im- provements of 44.3% compared BrecQ (Li et al., 2021) and 7.9% over QDrop (Wei et al., 2022) with 4/4/4 bit setting. Further, with 4-bit weight and 8-bit activation, MREM-S/MREM-P (Bai et al., 2022) present a 1.6/1.5% accuracy gap to the full- precision model with 4096 calibration data, while FPQ achieves almost no accuracy loss with only
5.3 Ablation Study
In this section, we first compare the influence of dif- ferent calibration sizes FPQ. We vary the calibra- tion size in (32, 61,, 128, 256} and test on MNLI, QQP. and CoLA. Table 4 shows that the evalua- tion on MNLI and QQP is more robust different settings, and the variance is more significant on CoLA. We observe that FPQ performs well with a
http:://hngamnaaaaccooeeermaeeee [TASK [TASK_NAME] NAME

---

W/A
Quant Method
Deit-S
Deit-B
VII-S
E/WA
71+
MNLI-M
QQP
CoLA
Full-precc
79.9
81.8
81.4
4/4/4
0.011,
82.3
89.2
52.6
6/6
PrQaviT(Yuan al., 2022)
76.3
4/4/4
0.2
82.2
89.1
53.6
6/6
APQ-ViT((ing) Vii(Ding et al., 2022)
77.8
80.4
79.2
4/4/4
0.5, 1.5
82.3
88.4
52.8
6/6
MinMax FP Quant (E3M2)
79.3
6/6/6
0.011 1,2
84.5
90.8
57.3
6/6
FPQ baseline
79.43
81.7
80.9
0.1.1.2
84.7
90.8
57.5
6/6
FPQ
79.5
81.8
6/6/6
0.5.1.5
84.7
90.8
57.8
4/4
PTQ4ViI(YYan et al., 2012)
34.1
64.4
42.6
Table Ablation studies different search range.
4/4
APQ-VT VIT (Ding et al., 2022)
43.6
67.5
48.0
4/4
MinMax FP Quant (E2M1)
0.4
4/4
FPQ baseline FPQ
6.57
0.71
4/4
75.0
79.4
73.2
Format
Adder(unn)) Multiptier(zmma) MAC(am")
Table 3:
Comparison on the ImageNet dataset with
INT4
93
182
410
INT6
132
340
529
vision transformer structures.
E2M1
111
92
443
E3M2
223
138
498
E/WA
#Calib
MNLIIM
QQP
CoLA
E2MI EIM2
105
107
432
4/4/4
81.5
89.4
44.4
Table 6: Area differences of INT, FP and mixed Format
4/4/4
64
81.8
89.4
47.9
FP operators across different bit-widths.
4/4/4
128
82.3
89.2
52.6
4/4/4
256
81.9
89.0
52,9
6/6/6
32
84.8
90.8
55.0
90,9
mary cost for INT and the adder for FP. Notably, the disparity between FP4 and INT4 adders is small, while INT has twice the hardware cost for the mul- tiplier. Moreover, the mixed-format FP4 operator has comparable hardware area as the standard FP4 operator. These findings indicate that the proposed FPQ approach imposes negligible overhead in terms of hardware implementation when compared to the standard FP operators and the hardware cost for FP is comparable with INT.
6/6/6
128
84.5
90.8
57.3
6/6/6
256
84.6
90.8
Table 4: Ablation studies of different calibration sizes.
calibration size of 128 data points. However, we also find that remains robust and maintains competitive accuracy even with limited access to calibration data, such as when using as few as 32 data points.
We investigate the robustness of FPQ to dif- ferent search ranges (2). Table 5 presents the results of FPQ using three sets of (y1, ?2): (0.01, 1.2), (0.1, 1.2), (0.5, 1.5), on MNLI, QQP. and CoLA. It is observed that no single search range outperforms the others consistently across all tasks. For instance, the search range (0.01, 1.2) performs better than (0.5, 1.5) on MNLI and QQP. but slightly worse on CoLA in the 4-bit configu- ration. Overall, FPQ exhibits robustness various 7/1 and 722 long as the search range is not overly aggressive.
Conclusion
This paper presents the first successful demonstra- tion of 4-bit floating-point post-training quantiza- tion for weights, activations, and embeddings natural language transformer architectures, includ- ing both large language models and BERT model. We also extend our method to vision transform- ers and observe its robust generalization ability, Our approach involves practical search-based technique which establishes strong baseline and achieves state-of-the-at art results for 6-bit and 8-bit quantization. Furthermore, we address the chal- lenge of high inter-channel variance in transform- ers by proposing pre-shifted exponent bias, which proves highly effective in achieving accurate 4-bit quantization.
5.4 Hardware Cost
We further examine the hardware utilization of low- FP multiplication
bit INT, FP, and operators, including adder, multiplier, and multiply- accumulate (MAC) units, in terms of hardware area. Mixed-format FP refers to the multiplication of floating- numbers with different formats, e.g., E2M1 multiplies with E1M2. We implemented the MAC operator by Verilog HDL and utilized Cadence Genus obtain the synthesized area un- der TSMC 40mm technology and 0.5GHz clock frequency.
Acknowledgement
This research is supported by National Natural Science Foundation of China/ HKSAR Research Grants Council Joint Research Scheme under Grant N H KUSSTT ST627 /20, and Foshan HKUST Projects under Grant FSSST21
Table illustrates the hardware cost of the INT and FP operators, with the multiplier being the pri-
KUSSTOO

---

Limitations
International Conference on Multimedia, MM '22. page 5380- 5380-5388, New York, NY, USA. Association for Computing Machinery.
Our experiments were conducted on publicly avail- able datasets with finite sentence lengths, and the generalizability of our method to extremely long sequences or streaming data has not been verified and may require further investigation. In addition, it remains to be seen how our proposed method can generalize to other domains beyond language and vision, such as audio. It would also be interesting to see the applicability of our method to generative tasks and other applications.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate post-training compression for generative pretrained transformers. In International Conference on Learning Representa- lions.
References
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.055 000361.
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. Vatt: Transformers for multimodal self- supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206-24221.
Jacob Devlin Ming-Wes Weig-Weiii Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Haoli Bai, Lu Hou, Lifeng Shang. Xin Jiang, Irwin King, and Michael Lyu. 2022. Towards efficient post- training quantization of pre-trained language models. In Advances Neural Information Processing Sys- tems.
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort, 2022. Fp8 quantization: The power of the exponent. Ad- vances Neural Information Processing Systems. 11144651-16666
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. Understanding and overcoming the challenges of efficient transformer quantization.
Jemin Lee, Yongin Kwon, Jeman Park, Misun Yu, and Hwanjun Song. 2023. Q-hyvit: Post-training quan- tization for hybrid vision transformer with bridge block reconstruction.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to- woanncce-t---uueneeeeeee pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871 1871-7880.
Yaohui Cai. Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Zeroq: A novel zero shot quantization framework. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisiley. 2019. Low-bit quantization of neural net- works for efficient inference.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu. Wei Wang, and Shi Gu. 2021. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zeulemoyer. 2022. Lnm.in 8-bit matrix multipli- cation for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-30332
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chan- dra. 2023. LIm-qat: Data-free guantization aware training for large language models. arXiv preprint arXiv:2305. 17888.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai,
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021. Post-training quantization for vision transformer. Advances in Neural Informa- tion Processing Systems, 34:28092-22 28092228103 103.
Junjie Liu, Xiaolin Wei, and Xianglong Liu. 2022. Towards accurate post-training quantization for vi- sion transformer. In Proceedings of the 30th ACM

---

Paulius Micikevicius, Dusan Stosic, Neil Burgess, Mar- ius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu. 2022. Fp8 formats for deep learning.
Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu. and Guangyu Sun. 2022. Ptq4vit: Post- Post-training quantization for vision transformers with twin uni- form quantization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pages 191-207. Springer.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quan- tization. In International Conference on Machine Learming, pages 7197-7206. PMLR.
Yijia Zhang. Lingran Zhao. Shijie Cao. Wenqiang Wang. Ting Cao, Fan Yang. Mao Yang, Shanghang Zhang. and Ningyi Xu. 2023. Integer floating point? new outlooks for low-bit quantization large language models.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-free quantization through weight equalization and bias correction.
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research,
21 21()):54855555
Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Hervé Jé- gou. 2021. Training data-efficient image transform- & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. 2022. QDrop: Randomly dropping quantization for extremely low-bit post-training quan- tization. International Conference on Learning Representations.
Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang. 2020. Easyquant: Post-training quantization via scale optimization.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate efficient post-training quantization for large guage models. arXiv preprini 22Xi:22111.04333

---

A Hessian-Based Loss Metric
B Quantization Error of Different Floating-Poo Point Formats
The objective of post-training quantization to minimize the perturbation (6X = XFP - XR)
Figure 4 compares the quantization error of differ- ent formats in 8-bit quantization, including INT8, E2M5, E3M4, E4M3, and E5M2. We apply these formats different BERT modules in the first, fifth, and last layers. The figures demonstrate that the op- timal FP formats differs depending on the specific module that we are quantizing.
introduced by quantization to the pre-trained real-
valued network:
min B[L(XR + 6X) [(RR)
(20)
Following the Taylor series expansion, we have ( + 6x) ((X)))
C Inter-Channe Variance Visualization
~ 6x x
H
(21)
2
Figure 5 and 6 depict the output of different fully- connected layers in BERT for the MNLI task, DeiT- for the ImageNet- task, and LLaMA-7B for the zero-shot reasoning task. The visualizations reveal a noticeable inter-channel variance presented in both language and vision transformers.
~ xx 2
(X)
Here, g (X) is the gradients and x is the Hes- sian matrix. Since the pre-trained model is well- converged, we can assume that g(x) has near zero value in every element, and thus term (X))(() can be neglected.
D Efficient Matrix Multiplication
The Hessian matrix H is computed as:
Figure 7 displays a comprehensive list of all the granularity options that allow for efficient ma- trix multiplication. While per-token quantization theoretically provides greater precision in terms of quantization granularity, the accuracy gains achieved through this method are minimal and do not justify the additional computational overhead required. As result, we have opted to use per- tensor quantization when quantizing activations.
(22)
where (X) denotes the Jacobian matrix of the layer output O w.r.f X, and (O) is the Hessian matrix WAY O. We then substitute the above equa- tion back equation 21
8X H 5X
(Jo(X) O
H 0) (Jo()
(23)
Ö HH () 0)
E Learning Format and Maximum Value
Here Ô is the intermediate output of the quantized layer and O is the original layer output. Note that under the assumption that 6X is relatively small (Li et al., 2021), we can approximate (Ó - O) as Jo(X) (X)oX using first-order Taylor expansion. Nevertheless, the calculation of H is still bur- densome, therefore, we use the diagonal entries of the Fisher Information Matrix of O to substitute O)) following (Li et al., 2021; Yuan al., 2022), and the new
We compare the previous gradient-based method (Kuzmin et al., 2022) with the proposed search-based method for finding the optimal format and maximum value. On DeiT-S, the learnable method only achieves 74.38% accuracy for an 8-bit quantized model on ImageNet, contrast, FPQ can attain an almost loss-less result of 79.88%. We analyze the gradients for the number of exponent bits derived in (Kuzmin et al.. 2022) and observe that each time the exponent bits change, the gradients experience exponential variations, leading high instability. Based on this observation, we assert that employing a search-based method to determine the optimal formats is crucial in post-training quantization (PTQ).
metric becomes:
OL )
ØL 00
B O''ddia((
) O)
00,
(24)
Here, each entry of O is assumed to be independent and n denoted the total number of elements in O. In this study, this hessian-based metric used as the reconstruction metric search for the optimal FP quantization function for both the weight and ac- tivation when performing layer-wir wise reconstruction in BERT and Vision Transformer models.
F Reconstruction Choices
The previous works on integer post-training quanti- zation involves breaking down the target model into

---

sub-modules and reconstructing them separately (Nagel et al., 2020; Li et al., 2021; Bai al., 2022; Yuan et al., 2022). This addresses the problem of over-fitting, given that only a limited amount of unlabeled calibration data available. In this study we find the layer-wise reconstruction and parallel quantization works best for floating-poin PTQ: Layer Reconstruction: Recent research (Li et al., 2021; ; Bai et al., 2022) suggests increasing the reconstruction granularity from layer reconstruc- tion (Nagel et al., 2020) to block reconstruction (Li et al., 2021) even larger granularity (Lee et al.. 2023). This is achieved by jointly optimizing all the linear layers matrix multiplication components within each module prevent the propagation of reconstruction errors among the layers. Despite this, we have observed that increasing the recon-
struction granularity does improve the accuracy of FPQ baseline or sometimes even lead to worse results. Therefore, we choose layer reconstruction. Parallel Quantization: Sequential quantization is the most commonly used approach (Wu et al., 2020; Nagel et al., 2020; Li et al., 2021) where modules are quantized consecutively based on their sequential order, and the input for the current cali- brating module is generated using all the previously quantized modules. However, some recent works (Yuan et al., 2022; Bai al., 2022) proposed a new parallel quantization framework. This framework uses the raw output of the full full-precisio modules as input and makes the calibration of each module independent from one another. In this work, we use parallel quantization, as it yields better results than its sequential counterparts.
Layer Quantization Error
Layer 5 Quantization Error
Layer 12 Quantiratian Error
230
250
INT3
INTS
INT3
230
E2M5
202
B1M3
E4MS
150
150
E5K2
100-
100
Layer Laper Intermedate
Layar oulppt
Layer Laper Intermeciate
Lager
Layer albertion Lager Intermadiata Layer oulput
Figure 4: Quantization error of different formats for BERT layers.
Bert
DeñT-S
layer' attertion dense
attentionn Eddense
Lettention dense
lypen pra
layer 5 atthuro)
sytr:1..ttm.prr
129
200
Chansy
100
2E7 Cramos
M0 all
Chanse
220
(b)
Syero.intemed
bbyer
boyer.
layer p.lc1
byer. mip.fc.
bysr11.rttttt
3307
1000
po0t go0 Daanigg
Chinver
Chooee
Chosser 1553
(c)
byeer outgut
lajen outpurt deree
byyer output
bayerSunip.cc PC2
bye:: mlc.fc2
100
J00
Cheenal
2001
(e)
(
Figure 5: Magnitude of the output activations of different modules in BERT (left column), and DeiT-S (right column).

---

LLaMa-77 7B
byeedd w.coffffeaeooooo patttnooorr
layenS.sel, sttnn
2000
2000 Dosina
1000 Oranse
Chainea
#000
(a)
b.yor..m d.wwo
byen. mlp down_prd
loyer mlp
2000
4002
4000
(b)
cellaattn
self
self
25
300
500
200
Chancy
1000
Chanseal
4002
Channul
1000
Figure 6: Magnitude of the output activations of differ- ent modules LLaMA-7B.
0x
aw
8
W 1,1
W
2,1
X
2,
2.11
2.2
Per-tensor
Per-tensor
c
K
a
X
w
W
1
ax
X
2,1
X
2,2
W21
Per-token
Per-channel
Figure 7: Quantization granularity options that support efficient matrix multiplication. The dimensions that share the same scaling factor indicated with red dotted frames


Reviews:
[{"review_id": "ext6zix8ru", "reviewer": "Reviewer_v99B", "summary": "", "questions": "* I am not quite clear how is floating number being compressed to ultra-low bit such as 4bit, how does it maintain sufficient information? For example, with only 1 bit for mantissa, is it fair to use such low bit? However, as long as the result in paper is credible, it is really impressive then.\n\n\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1: The hessian-base objective subsection 4.1 is not new]**\n\nThanks for the comments. Prior studies that employed hessian-based metrics focused solely on INT PTQ. Different from INT quantization, FP quantization involves searching the format choices as well as the associated maximum quantization value, and the hessian-based search method taking these factors into consideration is absent in the literature. As a result, the proposed hessian-base method for FP PTQ establishes a strong baseline that can already achieve lossless results in both 8-bit and 6-bit configurations, making it a robust cornerstone for future endeavors.\n\n**[Weakness 2: Concerns regarding the practicality of the proposed methods on the current mainstream DNN accelerator]**\n\nThe primary focus of this paper is to investigate the effectiveness of low-bit FP formats from a generally applicable perspective, without delving into the intricate details of hardware implementation for dedicated DNN accelerators. We believe that such an assessment would be highly sensitive to specific hardware design choices, considering the multitude of factors involved, such as overflow handling, NaN management, multiplier and accumulator design, and support for sparsity, among others [1,2,3]. Nevertheless, we did analyze the hardware utilization of low-bit INT, FP, and mixed-format FP multiplication operators in Section 5.4, which offers a more hardware-agnostic perspective. Our findings indicate that FP formats generally incur higher costs for addition compared to the INT format but require less energy for multiplication. Moreover, the discrepancy between the INT and FP formats gradually diminishes as the bit-width decreases, suggesting a reduction in latency difference between INT and FP as well. \n\nFurthermore, since FP8 is already widely supported by various hardware platforms, we believe that adopting similar hardware implementations to accommodate FP formats with bit-widths lower than 8 bits should be relatively straightforward. Our proposed FPT framework can serve as a valuable reference for those seeking to develop their own low-bit FP transformer accelerators based on their own hardware use case.\n\n**[Question 1:  how is floating number being compressed to ultra-low bit such as 4bit]**\n\nThanks for this insightful question. We also note that the difference between ultra-low FP numbers and INT numbers is small. For example, compared to INT4, FP4 has only one more representation which is E2M1, since E1M2 is identical to INT4. As a result, we can observe that in Table 1, the naive MinMax FP did not achieve better results compared to integer quantization. Nevertheless, the proposed pre-shifted exponent bias utilizes the characteristics of the FP format to handle the high channel-wise variance problem and improve the FP quantization accuracy significantly especially for FP4 setting. The code will be released for future works to easily follow up and reproduce our results.\n\n[1]: FP8 formats for deep learning, Paulius Micikevicius, et al., 2022\n\n[2]: FP8 Quantization: The Power of the Exponent, Andrey Kuzmin, et al., 2022\n\n[3]: FP8 versus INT8 for efficient deep learning inference, Mart van Baalen, et al., 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Floating point quantization by replacing the scaling factor or bias from real value to exponent may help handle the outlier problem during quantization.\n\n* Good performance showed with sufficient experiments.\n\n* Detailed and clear explanation on the workflow with good expression.", "reasons_to_reject": "* The Hessian-based objective subsection 4.1 is not new. \n\n* As far as I know the mainstream DNN accelerator do not hold specific for such kind of quantization scheme. Just a bit skeptical on the practicality. You see the bit width itself is not the only concern as speed/computation cost matters more in most cases (although I would agree that pre-shifted bit bias can help with computation cost).\n", "questions_to_authors": "* I am not quite clear how is floating number being compressed to ultra-low bit such as 4bit, how does it maintain sufficient information? For example, with only 1 bit for mantissa, is it fair to use such low bit? However, as long as the result in paper is credible, it is really impressive then.\n\n\n", "paper_topic_and_main_contributions": "* This paper has proposed a channel-wise floating quantization method for transformer-based models which has shown prominent performance in NLP and CV tasks. \n* This work brought hessian-based loss into the FP quantization and analyze the inter-intra channel variance to show the motivation.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hOnZfE6NRa", "reviewer": "Reviewer_SAuf", "summary": "", "questions": "1. how many bits for representing tilde b in Eq. 7, and for scaling factors like tilde alpha? \n2. what is r_x in Algorithm 1, line 10? is it a set containing tilde b in Eq. 7, tilde alpha and v in Eq. 8? if so, what is the searching complexity? 100^3? \n3. where does Eq.9 come from? is there a reference to understand it?\n4. why does Eq. 10 only contain tilde alpha? what about v as in Eq. 8?\n5. line 356 to line 358, why scaling factor cannot be a shared constant along the multiplication direction? it looks like in Eq. 10, each column of W has a shared scaling factor. why can't activations X have per channel scaling factor? it looks the section 4.3 is proposing a way of computing a shared scaling factor for each row of X.\n6. line 399-401, it looks like you compute and store activation bias from calibration data, but how does the bias become zero in Eq. 20? if it is absorbed like in line 209, then it should not be tilde rho in Eq. 20? \n7. line 422, is the searching of rho also for minimizing Eq. 15? \n8. can you give an example of quantization result? for example in your 6 bit quantization result, how many of exponential bits, how many of mantissa bits? \n9. according to Table 4, it looks like the algorithm variance can be huge, thus, it is reasonable to believe your Table 1 results may not be significantly better than others. \n10. In Table 2, smoothquant seems reporting only quantization in 8 bits, did you run their algorithm on 4 bit quantization? how is your 8 bit quantization result compared with their model? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive feedback and the detailed comments. To clarify, our main contribution lies in being the first to successfully quantize a Transformer model into 4 bits with floating-point post-training quantization. In contrast,  previous studies primarily focused on 6-bit or 8-bit quantization methods.\n\n**[Q1.  how many bits for representing tilde b in Eq. 7, and for scaling factors like tilde alpha?]**\n\n$\\tilde{b}$ and $\\tilde{\\alpha}$ are both FP numbers using 32-bit. As stated in lines 210 and 211 in the paper, $\\tilde{b}$ denotes a relaxed tensor-wise real-valued exponent. $\\tilde{\\alpha}$ is also a real-valued number as indicated in line 209. “Real-valued” number is a term that has been widely used in the quantization literature [1], referring to the original numeric representation before quantization, which is FP32 in our context.\n\n**[Q2. what is r_x in Algorithm 1, line 10? is it a set containing tilde b in Eq. 7, tilde alpha and v in Eq. 8? if so, what is the searching complexity? 100^3?]**\n\n$r_x^i$ denotes one of the FP formats, for example, E1M2/E2M1 for FP4, as mentioned in line 1 of Algorithm 1. $r_x^i$ does not contain  $\\tilde{b}$ / $\\tilde{\\alpha}$ / $v$, but each format corresponds to the associated $\\tilde{b}$ / $\\tilde{\\alpha}$ / $v$. The search complexity varies with the bit-width. For example, for FP4, the search space should be 2 * 100, because we first choose a format from two choices (E1M2, E2M1) and then choose the $\\tilde{b}$ / $\\tilde{\\alpha}$ / $v$ from 100 choices.\n\n**[Q3. where does Eq.9 come from? is there a reference to understand it?]**\n\nEq.9 is a standard FP quantization formula, derived from Eq. 1. You may refer to Section 4.1 of [6] for more detail.  Sec.3.1 is a bit dense in formulas; we added a figure to make it easier to follow.\n\n**[Q4. why does Eq. 10 only contain tilde alpha? what about v as in Eq. 8?]**\n\nEq. 10 only contains $\\tilde{\\alpha}$, as $v_x$ and $v_w$ are integrated into the $\\tilde{X}$ and $\\tilde{W}$, respectively. We will add $\\tilde{X} = v \\cdot \\left\\lfloor \\frac{X_{\\rm R}''}{\\tilde{\\alpha} \\cdot v} \\right\\rceil$ after Eq. 10 to enhance the readability. \n\n**[Q5. line 356 to line 358, why scaling factor cannot be a shared constant along the multiplication direction? it looks like in Eq. 10, each column of W has a shared scaling factor. why can't activations X have per channel scaling factor? it looks the section 4.3 is proposing a way of computing a shared scaling factor for each row of X.]**\n\nThe reason why activations X cannot have per-channel scaling factors is that the multiplication direction for the activation tensor operates on a per-row basis rather than per-channel, as visualized in Figure 5 in the appendix. Lines 356 to 358 indicate that to achieve efficient matrix multiplication, the scaling factor must be a shared constant along the multiplication direction. However, applying per-channel scaling factors for activations introduces challenges to efficient matrix multiplication, since the scaling factor is **NOT** a shared constant along the multiplication direction and cannot be extracted as shown in Equation 10. This limitation of per-channel scaling factors for activation X is also discussed in the SmoothQuant paper [2]. Therefore, in Sec. 4.3, we propose pre-shifted exponent bias which utilizes the characteristic of FP format to compute a shared scaling factor for each **channel**(column) of X while being able to maintain efficient matrix multiplication\n\n**[Q6. line 399-401, it looks like you compute and store activation bias from calibration data, but how does the bias become zero in Eq. 20? if it is absorbed like in line 209, then it should not be tilde rho in Eq. 20?]**\n\nThank you for your question. We acknowledge that this particular part is complicated, and as a result, we have reorganized the explanations and equations as follows:\n\nBefore the pre-shifted exponent bias,\n\n\\begin{aligned}\nX_{\\rm FP} &=  2^{-\\tilde{\\mathbf{b}_j}} \\cdot(-1)^s2^{p-0}(1+\\frac{d_1}{2}+...+\\frac{d_m}{2^m}) \\\\\n    = 2^{-\\tilde{\\rho}}(-1)^s2^{p-\\mathbf{b}^{ori}_j}(1+\\frac{d_1}{2}+...+\\frac{d_m}{2^m})\n\\end{aligned}\n\n\n\\begin{aligned}\nW_{\\rm FP}=2^{-\\tilde{\\mathbf{b}}^{W}}(-1)^s2^{p-0}(1+\\frac{d_1}{2}+...+\\frac{d_m}{2^m})\n\\end{aligned}\n\nAfter pre-shifted exponent bias, \n\n\\begin{aligned}\nX_{\\rm FP}&=2^{-\\tilde{\\rho}}(-1)^s2^{p-0}(1+\\frac{d_1}{2}+...+\\frac{d_m}{2^m})\n\\end{aligned}\n\n\\begin{aligned}\nW_{\\rm FP}=2^{-\\tilde{\\mathbf{b}}^{W}}(-1)^s2^{p-\\mathbf{b}^{ori}_j}(1+\\frac{d_1}{2}+...+\\frac{d_m}{2^m})\n\\end{aligned}\n\nLooking at these formulas, we can see the proposed pre-shifted exponent bias transfers the activation bias to the weight tensor, as visualized in Figure. 2. \n\n**[Q7. line 422, is the searching of rho also for minimizing Eq. 15?]**\n\nYes, we will add it in line 422, that the algorithm searches for the optimal $\\tilde{\\rho}_{\\mathbf{X}}$ by minimizing Eq.15.\n\n**[Q8. Can you give an example of quantization result? for example in your 6 bit quantization result, how many of exponential bits, how many of mantissa bits?]**\n\nAs mentioned in lines 304 and 305 of Section 4.2, the search for the optimal format is performed on a layer-by-layer basis. This implies that each weight and activation tensor in different layers may have different search results. We listed the search results of each linear layer in the 6-bit quantized Bert for your reference and will add these results in the Appendix of the paper. From the result, we can observe that the majority of modules reach E2M3 once the search is completed. However, certain modules, such as the activation of output.dense and intermediate.dense, require E3M2 to effectively handle distributions with longer tail.\n\n\n| Layer # | attn.query (W/A) | attn.key (W/A) | attn.value (W/A) | attn.output.dense (W/A) | intermediate.dense (W/A) | output.dense (W/A) |\n| ----- | ----- |----- | ----- | ----- | ----- | ----- |\n| 0 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | \n| 1 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | \n| 2 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E3M2/E3M2 | \n| 3 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E3M2/E3M2 | \n| 4 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | E2M3/E3M2 | E3M2/E3M2 | \n| 5 |  E2M3/E2M3 | E2M3/E3M2 | E2M3/E3M2 | E2M3/E2M3 | E2M3/E2M3 | E3M2/E3M2 | \n| 6 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | \n| 7 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | E2M3/E3M2 | \n| 8 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | E2M3/E3M2 | \n| 9 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | \n| 10 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 | \n| 11 |  E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E2M3 | E2M3/E3M2 |\n\n**[Q9. according to Table 4, it looks like the algorithm variance can be huge, thus, it is reasonable to believe your Table 1 results may not be significantly better than others.]**\n\nTable 4 is an ablation for different numbers of calibration sentences, not the variance of experiments. Too few calibration data will lead to accuracy degradation, for example, 32/64 sentences. Thus, we recommend using 128 data points, which delivers stable and satisfactory accuracy on most datasets. Note that other baseline methods [3, 4, 5], which employ a much larger calibration set with 4096 sentences. For the experiment variance, We also did multi-trial experiments on the Bert model, the variance is only 0.002 point.  \n\n**[Q10. In Table 2, smoothquant seems reporting only quantization in 8 bits, did you run their algorithm on 4 bit quantization? how is your 8 bit quantization result compared with their model?]**\n\nYes, we reimplemented Smoothquant with a 4-bit setting and reported the results.\n\nAccording to your suggestion, we compare the 8-bit quantization results with Smoothquant.  It is worth noting that naive MinMax FP8 quantization can already achieve nearly lossless accuracy, as also mentioned in the paper. Consequently, the difference in performance between MinMax, SmoothQuant, and our proposed FPT methods is insignificant in the 8-bit setting, as demonstrated below:\n\n| Quant Methods          | #Bits (E/W/A) |  #Calib | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | Avg. | \n| --------------          | -------------- | -------------- | -------------- | -------------- | -------------- | --------------| -------------- | -------------- | -------------- | \n| LLaMa-7B Full-precision       | 16/16/16 |  -  | 75.1 | 78.7 | 56.9 | 69.9 | 75.3 | 41.9 | 66.3 | \n| MinMax (E4M3)       | 8/8/8 | 32  | 74.86  | 78.62 | 56.8 | 69.46 | 75.46 | 41.55 | 66.12 |  \n| SmoothQuant         | 16/8/8 |  512  | 73.95 | 77.46 | 55.02 | 69.61 | 74.34 | 37.37 | 64.62 |  \n| FPT         | 8/8/8 |  32  | 75.6 | 78.18 | 56.57 | 70.24 | 74.62 | 40.7 | 65.98 | \n\n**[Typo: In Eq. 14, it should be J(X) not J(W)]**\nThanks for pointing this out, we will correct the typo in our final version of the paper.\n\n[1]: A Survey of Quantization Methods for Efficient Neural Network Inference, Amir Gholami, et al., 2021\n\n[2]: SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, Guangxuan Xiao, et al., 2023\n\n[3]: Towards Efficient Post-training Quantization of Pre-trained Language Models, Haoli Bai, et al., 2022\n\n[4]: Brecq: Pushing the limit of post-training quantization by block reconstruction, Yuhang Li, et al., 2021\n\n[5]: QDrop: Randomly dropping quantization for extremely low-bit post-training quantization, Xiuying Wei, et al., 2022\n\n[6]: FP8 Quantization: The Power of the Exponent, Andrey Kuzmin, et al., 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "reasonable algorithm, good results", "reasons_to_reject": "not good writing, hard to understand", "questions_to_authors": "1. how many bits for representing tilde b in Eq. 7, and for scaling factors like tilde alpha? \n2. what is r_x in Algorithm 1, line 10? is it a set containing tilde b in Eq. 7, tilde alpha and v in Eq. 8? if so, what is the searching complexity? 100^3? \n3. where does Eq.9 come from? is there a reference to understand it?\n4. why does Eq. 10 only contain tilde alpha? what about v as in Eq. 8?\n5. line 356 to line 358, why scaling factor cannot be a shared constant along the multiplication direction? it looks like in Eq. 10, each column of W has a shared scaling factor. why can't activations X have per channel scaling factor? it looks the section 4.3 is proposing a way of computing a shared scaling factor for each row of X.\n6. line 399-401, it looks like you compute and store activation bias from calibration data, but how does the bias become zero in Eq. 20? if it is absorbed like in line 209, then it should not be tilde rho in Eq. 20? \n7. line 422, is the searching of rho also for minimizing Eq. 15? \n8. can you give an example of quantization result? for example in your 6 bit quantization result, how many of exponential bits, how many of mantissa bits? \n9. according to Table 4, it looks like the algorithm variance can be huge, thus, it is reasonable to believe your Table 1 results may not be significantly better than others. \n10. In Table 2, smoothquant seems reporting only quantization in 8 bits, did you run their algorithm on 4 bit quantization? how is your 8 bit quantization result compared with their model? ", "paper_topic_and_main_contributions": "This paper proposes to find optimal exponent bits and scaling factor in floating point quantization by using Hessian based loss function. It uses brute force search for find the optimal format, and it utilizes the absorbed bias for generating per channel variant used in multiplication. Experimental resutls are reported on 8bits and 6 bits Bert model and recent LLaMA model. Generally, the moder performance is well maintained after quantization. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Eq. 14, it should be J(X) not J(W)"}, "extra_text": {}}, {"review_id": "4Z5kiYjNH9", "reviewer": "Reviewer_KRMt", "summary": "", "questions": "(1) Can you provide a comparison of inference latency between the proposed method and other methods (latency from hardware simulation is also acceptable)?\n(2) How does the proposed method deal with the layernorm layer and residual addition in the transformer? Should fp32 or fp16 be used in these two cases?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1 and Question 1: Lack of inference latency comparison between the proposed method and other methods]**\n\nComparing the inference latency between the proposed FP format and the INT format is highly challenging due to the sensitivity of the results to various hardware design choices. In cases where the bottleneck is the data transfer bandwidth, the latency difference resulting from INT8 and FP8 formats may be considered negligible or similar [1,2]. To avoid the complexities associated with hardware choices, in Sec. 5.4 of the paper, we conducted hardware-agnostic experiments focusing on the utilization of low-bit INT, FP, and mixed-format FP multiplication operators, such as adders, multipliers, and multiply-accumulate (MAC) units, in terms of hardware area.\n\nOur simulations reveal that FP formats generally incur higher costs for addition compared to the INT format, but they require less energy for multiplication. Furthermore, as the bit-width decreases, the disparity between INT and FP formats gradually diminishes, suggesting a reduction in latency difference as well. Additionally, FP8 has gained wide adoption across various hardware platforms and has shown no significant latency drawbacks compared to its INT counterpart. Consequently, we believe that the FPT approach not only serves as a robust baseline for future FP quantization research but also provides valuable insights for hardware researchers aiming to design ultra-low-bit transformer accelerators.\n\n\n**[Weakness 2: Experiments on the LLM generative tasks; More experiments on models larger than LLaMa-7B ]**\n\nWe conducted experiments for generative tasks on LLaMa-7B, using perplexity evaluation on Wiki2 and C4, as shown in the following table. The results indicate that our proposed FPT framework consistently achieves better results than all the baselines, including MinMax and SmoothQuant. \n\n| Quant Methods          | #Bits (E/W/A) |  #Calib | Wiki2 | C4 |\n| ------          | ------  | ------  |------  | ------ |\n| LLaMa-7B Full-precision | 16/16/16 | - | 5.67 | 7.57 |\n| FP MinMax (E2M1)          | 4/4/4 |  32  |  100176  |  109849 |\n| SmoothQuant          | 16/4/4 |  512  | 174.87  | 107.73 |\n| FPT          | 4/4/4 |   32  |  $\\textbf{8.66}$  | $\\textbf{11.28}$ |\n\n\nFollowing your suggestion, we performed additional experiments on LLaMa-13B. The results indicate that both MinMax and SmoothQuant are unable to maintain accuracy in ultra-low bit settings. In contrast, the FPT method achieves an average accuracy of 63.1 with a mere 5.8 point loss in accuracy.\n\n| Quant Methods          | #Bits (E/W/A) |  #Calib | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | Avg. | \n| --------------          | -------------- | -------------- | -------------- | -------------- | -------------- | --------------| -------------- | -------------- | -------------- | \n| LLaMa-13B Full-precision       | 16/16/16 |  -  | 77.9 | 79.2 | 59.9 | 72.6 | 77.4 | 46.4 | 68.9 | \n| FP MinMax (E2M1)       | 4/4/4 | 32  | 45.8 | 51.7 | 25.5 | 49.5 | 25.0 | 22.8 | 36.7 |  \n| SmoothQuant         | 16/4/4 |  512  | 57.6 | 61.3 | 56.0 | 52.6 | 49.9 | 25.1 | 50.4 |  \n| FPT         | 4/4/4 |  32 | 71.9 | 74.8 | 53.3 | 66.7 | 71.7 | 39.9 | $\\textbf{63.1}$ | \n\nWe will add these results in the experiment section of the paper to make it more comprehensive.\n\n**[Question 2: How does the proposed method deal with the layernorm layer and residual addition in the transformer? Should fp32 or fp16 be used in these two cases? ]**\n\nWe follow previous works [3,4,5,6] which keep the residual addition and layernorm in fp32.\n\n**[Typos: In contribution, \"per-shifted\" should be \"pre-shifted\". This caused quite some confusion at first.\n]**\n\nThanks for the comments; we have thoroughly checked the paper and fixed typos in the manuscript.\n\n\n[1]: FP8 Quantization: The Power of the Exponent, Andrey Kuzmin, et al., 2022\n\n[2]: FP8 versus INT8 for efficient deep learning inference, Mart van Baalen, et al., 2023\n\n[3]: SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, Guangxuan Xiao, et al., 2023\n\n[4]: Towards Efficient Post-training Quantization of Pre-trained Language Models, Haoli Bai, et al., 2022\n\n[5]: Brecq: Pushing the limit of post-training quantization by block reconstruction, Yuhang Li, et al., 2021\n\n[6]: QDrop: Randomly dropping quantization for extremely low-bit post-training quantization, Xiuying Wei, et al., 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The proposed Hessian-base joint search method for determining the optimal exponent bias and maximal quantization value outperforms existing techniques in terms of stability and performance, establishing a strong baseline for floating point post-training quantization.\n(2) The proposed pre-shifted exponent bias method can effectively addresses the challenge of high inter-channel variance in the transformer while incurring negligible computation cost.\n(3) By combining the above two methods, the paper quantize both weights and activations in the Bert model to only 4-bit and achieves a significantly increased average GLUE score. The advantage is most obvious when activaiton bitwidth go below 8-bit.", "reasons_to_reject": "(1) The inference latency comparison between the proposed method and other methods is lacking in the paper, making it difficult to evaluate the feasibility and comprehensive cost performance of the proposed method.\n(2) It is necessary to supplement some experiments on the LLM generative tasks to verify the effectiveness of the proposed method. Since the method is PTQ, so dealing with models larger than LLaMa-7B should be possible and not that effort-taking?", "questions_to_authors": "(1) Can you provide a comparison of inference latency between the proposed method and other methods (latency from hardware simulation is also acceptable)?\n(2) How does the proposed method deal with the layernorm layer and residual addition in the transformer? Should fp32 or fp16 be used in these two cases?", "paper_topic_and_main_contributions": "This paper presents a 4-bit floating-point post-training quantization method for weights, activations, and embeddings in transformer models, including both Bert model and GPT models. The paper first proposes a Hessian-based joint search for determining the optimal exponent bias which consistently delivers desirable quantization results; then absorb the bias into the exponent bias of the floating point values. This technique is used to deal with high inter-channel variance in the transformer. Experimental results demonstrate that the proposed method yields the first usable FP4 quantized Bert model with competitive performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In contribution, \"per-shifted\" should be \"pre-shifted\". This caused quite some confusion at first."}, "extra_text": {}}, {"review_id": "niq7jd5ZXP", "reviewer": "Reviewer_EuFY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1: More experiments on different configurations]**\n\nBased on your suggestion, we conducted additional experiments on both Bert and LLaMa-7B using the FPT baseline for the 4/4/4 setting. The results indicate that in 4-bit quantization, the FPT baseline is unable to maintain accuracy for such a low-bit setting due to significant variance across channels. In comparison, after utilizing the proposed pre-shifted exponent bias, FPT is capable of dealing with channel-wise variance and significantly improves accuracy. For 4/4/16 settings in Table 2, FPT is the same as FPT baseline since the activations are not quantized. We will add these results to the manuscript.\n\n|Methods | #Bits (E/W/A) | #Calib | $\\textbf{MNLI}_{\\rm -m}$ | $\\textbf{QQP}$ |  $\\textbf{QNLI}$ | $\\textbf{SST-2}$ | $\\textbf{CoLA}$ | $\\textbf{STS-B}$ | $\\textbf{MRPC}$ | $\\textbf{RTE}$ | $\\textbf{Avg.}$ |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Bert (Full-precision) | 32-32-32 | - | 84.9 | 91.4 | 92.1 | 93.2 | 59.7 | 90.1 | 86.3 | 72.2 | 83.7 |\n|MinMax(E2M1)| 4/4/4 | 128 | 33.6 | 54.0 | 50.6 | 50.8 | 0.0 | 0.0 | 31.6 | 52.0 | 34.1 |\n| BrecQ | 8/4/4 | 4096 | 31.9 | 62.3 | 50.7 | 50.9 | 0.9 | 6.4 | 31.7 | 52.3 | 35.8 |  \n| QDrop | 8/4/4 | 4096 | 71.4 | 79.0 | 76.8 | 88.1 | 40.9 | 81.9 | 79.2 | 60.7 | 72.3 |\n| FPT Baseline | 4/4/4 | 128 | 38.9 | 68.3 | 55.3 | 83.6 | 10.6 | 0.0 | 43.8 | 55.2 | 44.5 |\n| FPT | 4/4/4 | 128 | 82.3 | 89.2 | 86.6 | 91.5 | 52.6 | 85.5 | 83.8 | 69.0 | $\\textbf{80.1}$ | \n\n| Quant Methods          | #Bits (E/W/A) |  #Calib | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | Avg. | \n| --------------          | -------------- | -------------- | -------------- | -------------- | -------------- | --------------| -------------- | -------------- | -------------- | \n| LLaMa-7B Full-precision       | 16/16/16 |  -  | 75.1 | 78.7 | 56.9 | 69.9 | 75.3 | 41.9 | 66.3 | \n| MinMax (E2M1)       | 4/4/4 | 32  | 47.3 | 53.1 | 25.7 | 50.7 | 25.1 | 22.4 | 37.4 |  \n| SmoothQuant         | 16/4/4 |  512  | 54.1 | 62.8 | 41.5 | 52.6 | 50.6 | 32.9 | 49.1 |  \n| FPT Baseline      | 4/4/4 |  32  | 57.3 | 56.6 | 30.2 | 51.1 | 37.71 | 23.2 | 42.7 | \n| FPT         | 4/4/4 |  32  | 64.2 | 73.5 | 47.8 | 63.7 | 65.9 | 33.6 | $\\textbf{58.1}$ | \n\n\n**[Weakness 2:  More figures for better understanding]**\n\nThanks for the suggestion. We added a visual representation to illustrate the distinction between FP and INT quantization in the preliminary section. Additionally, we moved Figure 2 one page ahead and provided more explanations in the figure caption to explain the process and make it easier to follow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is excellently written and highly accessible, even for readers outside the domain, thanks to the well-presented background on linear and floating-point quantization. The addressed problem carries significant importance within the NLP community. The proposed method is technically sound and demonstrates promising empirical performance across both language and vision benchmarks. Additionally, the authors' provision of ample implementation details facilitates easy reproduction by other researchers and practitioners in the community.", "reasons_to_reject": "The paper would significantly benefit from an improved presentation of results. It is essential to specify whether the baselines in both Table 1 and Table 2 are using linear or floating-point quantization techniques. While the authors have included the results of the FPT baseline for 8/8/8 and 6/6/6 settings in Table 1, they should also include the results for the other settings in Table 1 and in the other tables. These additional results are pivotal in allowing readers to gain a comprehensive understanding of the effectiveness of the proposed pre-shifted exponent bias and its performance under various configurations.\n\nWhile the paper is well-written overall, some readers may find it a bit dense in terms of the mathematical formulations. To address this concern, the inclusion of more accompanying figures and visual aids could significantly improve the paper's accessibility. Illustrating the key concepts and processes graphically would help bridge the gap between complex mathematical expressions and readers' comprehension, resulting in a more inclusive and engaging reading experience.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel floating-point (FP) post-training quantization (PTQ) technique for transformer-based models. The authors integrate hessian-based loss into quantization parameter search to develop a strong FP-PTQ baseline. They further propose per-channel activation quantization and demonstrate that reparameterizing additional scaling factors as exponential biases of weights incurs minimal computational cost. The method achieves impressive compression, quantizing both weights and activations in BERT to 4 bits, with just a 3.6-point drop in average GLUE score compared to the full-precision model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a floating-point-based post-training quantization regime for the transformer models. Reviewers initially had some problems with the clarity of presentation and the novelty of the methods, the authors have addressed the concerns. Reviewers unanimously agreed to accept this paper, thus, the AC deemed this paper should be accepted to the main conference.
Decision: Accept-Main