Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
Authors: Cheng Zhang
Jianyi Cheng
Ilia Shumailov
George Anthony Constantinides
Yiren Zhao
Venue: EMNLP 2023 Main (2023)
Keywords: large language model
LLM
quantization
NLP
TL;DR: We share the insights into nearly-lossless 6-bit & 4-bit LLM quantisation

Abstract: The inference of Large language models (LLMs) requires immense computation and memory resources. To curtail these costs, quantisation has emerged as a promising solution, but existing LLM quantisation mainly focuses on 8-bit. In this work, we explore the statistical and learning properties of the LLM layer and attribute the bottleneck of LLM quantisation to numerical scaling offsets. To address this, we adapt block quantisations for LLMs, a family of methods that share scaling factors across packed numbers. Block quantisations efficiently reduce the numerical scaling offsets solely from an arithmetic perspective, without additional treatments in the computational path. Our nearly-lossless quantised 6-bit LLMs achieve a $19\times$ higher arithmetic density and $5\times$ memory density than the float32 baseline, surpassing the prior art 8-bit quantisation by $2.5\times$ in arithmetic density and $1.2\times$ in memory density, without requiring any data calibration or re-training. We also share our insights into sub-8-bit LLM quantisation, including the mismatch between activation and weight distributions, optimal fine-tuning strategies, and a lower quantisation granularity inherent in the statistical properties of LLMs. The latter two tricks enable nearly-lossless 4-bit LLMs on downstream tasks. Our code is open-sourced.

Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
Cheng Zhang', Jianyi Cheng', Ilia Shumailov George A. Constantinides', Yiren Zhao
'mperial College London, PUniversity of Oxford {cheng.zhane 22, jianyi. cheng17, g. constantinides, a. 2.2hha))@impriial..c uk ilia. shumailov@chch. ox. ac. uk
Abstract
these parameters in floatl6 format (Brown et al., 2020). Quantisation serves as natural solution for reducing the cost of running inference on these LLMs (Yao et al., 2022; Xiao et al., 2022; Dettmers et al., 2022), as low-precision format enables cost savings across all relevant efficiency metrics: reduced on-chip memory, increased arithmetic intensity for matrix multiplies, and decreased DRAM bandwidth requirement. On the other hand, the growing popularity of running services such as ChatGPT (OpenAI, 2022) provides an impetus for exploring the use custom silicon to support LLM inference. This raises the question: What would low-precision number system look like in these near-future LLM hardware accelerators (ASICs)?
The inference of Large language models (LLMs) requires immense computation and memory resources. To curtail these costs, quan- tisation has emerged as a promising solution, but existing L.M quantisation mainly focuses on 8-bit. In this work, we explore the statisti- cal and learning properties of the LLM layer and attribute the bottleneck of LLM quantisa- tion to numerical scaling offsets. address this, adapt block quantisations for LLMs, a family of methods that share scaling factors across packed numbers. Block quantisations efficiently reduce the numerical scaling offsets solely from an arithmetic perspectíve, with- out additional treatments in the computational path. Our nearly-lossless quantised 6-bit LLMs achieve a 19x higher arithmetic density and 5x memory density than the float32 baseline, surpassing the prior art 8- bit quantisation by 2.5x arithmetic density and 1.2x mem- ory density, without requiring any data calibra- tion or re-training. We also share our insights into sub-8-bit LLM quantisation, including the mismatch between activation and weight dis- tributions, optimal lneettunig strategies, and a lower quantisation granularity inherent the statistical properties of LLMs. The latter two tricks enable nearly-lossless 4-bit LLMs on downstream tasks. Our code is open-sourced! !
LLM quantisation is challenging because of the activations with large absolute magnitudes, also known as activation outliers (Bondarenko et al., 2021; Xiao et al., 2022). Previous ap- proaches have proposed various techniques to ad- dress such outliers. However, these either require additional treatments in the integer quantisation domain (LLM. int8() and SmoothQuant) or yield unsatisfactory performance (ZeroQuant); and prior work has primarily focused arithmetics that can be ported GPUs. We observe that the presence of outliers necessitates different scaling factors a finer granularity than per-tensor or per-token level (Yao et al., 2022; Xiao et al., 2022). This insight naturally leads us to revisit arithmetic systems with small exponents, such as MinilFloat (Sun et al.. 2019), Block Minifloat (Fox et al., 2021), Block Logarithm (Miyashita et al., 2016), and Block Floating Point (Kalliojarvi and Astola, 1996), they can effectively represent outliers in Trans- former models. To the best of our knowledge, our work the first to systemically investigate short- exponent arithmetics for LLM quantisation.
1 Introduction
Pre-trained Large Language Models (LLMs) (Brown et al., 2020; Black et al., 2021: Zhang et al., 2022) have demonstrated impressive performance on range of Natural Language Processing (NLP) tasks. However, their underlying computational and memory costs are critical bottleneck to their usability. For instance, the larger variants in the GPT family scale up hundreds of billions of parameters, requiring least 300GB of memory to store
Figure illustrates the variance of the tensors joining the GEMMs in an OPT-6.7B (Zhang et al.,
'https: httt::///gittubb.mo////enggggggu 98/11m-mixed----

---

METHOD
(QW, QACT)
BITWIDTH
PTQ OR TAQ
# QUANTISED GEMMS
ZEROQUANT (YAO ET AL., 2022)
V
W4A8
TAQ
8/8
LLM.INT80 (DETTMERS ET Al.,, 2022)
(
W8A8
PTQ
6/8
GPTQ (FRANTAR ET AL., 2022)
(V.
W4
PTQ DC
6/8
SMOOTHQUANT (XIAO ET Al.,, 2022)
W8A8
PTQ DC
6/8
OURS
(V.V)
W6A6/W4A4
PTQ/TAQ
8/8
Table I: A comparison of different LLM quantisation methods. (QW, QAct) shows whether quantisations are applied to weights or activations, WzAy means z-bit quantisation for weights and y-bit quantisation for activation. PTQ and TAQ represents Post Training Quantisation and Training After Quantisation respectively. DC means data calibration. There are eight general matrix multiplications (GEMMs) per transformer layer (0-& in Algorithm 2). Only ZeroQuant and ours quantise all of them. Other approaches leave ! and ( in float 32/ft 1o format, which take up 20.6% floating-poin operations in OPT-6.7B' B self-attention. means outliers in LLM. INT8() is computed in float 16; this improves arithmetic density but memory density kept identical canonical float16.
2
Algorithm Transformer layer
Q
K
Require: X
> Iaput features
Require: H
& Number of heads
2
1: Xn. <- LayerNNorm(X)
Be
2: for i E [0, H) do
B1
3:
Q: X..Wa.
e
Wa
4:
K; + Xx,WK.
Variance
2-3
5:
Vi - X,Wv,,
WV
A, 9.K?
Wo
6:
Vau
2-7
Wa
7:
Å. softmaz(Ai, aris -1)
W1
8:
B. - A.V.
5
9: end for
2-11
10: B. & concul( .o..... BH-1)
11: Bp B.Wo + bn
12: В. + Layer Norm Ho
13: B, + (B, bi)
15
10
15
20
30
14: B2 B,W + by
15: O B2 Bo X
Layer ID
16: return O
Figure 1: The algorithm on the left is the forward pass computation of single Transformer layer (Vaswani et al., 2017) in mainstream LLMs, wherein values in blue (e.g. X.,) represent tensors with predetermined min-mai max values, such the outputs of normalisation layer or softmax. Values red have unbounded min-man max, and are plotted on the upper right for different layers of OPT-6.7B (Zhang et al., 2022). We show that for almost all activation tensors, their variances increase at deeper layers, resulting in scaling offsets in their quantisation, while weight tensors on the lower right have smaller variances. This statistical trend enlightens our LLM quantisation study.
2022). After feeding 128 samples from Wikitext2 to the pretrained float32 model, we make three in- teresting observations. 1) The variance of most ac- tivations in Figure increases with the depth of the
tire network. From an arithmetic perspective, we refer to this phenomenon as numerical scaling off- sets, as it requires different numerical ranges and granularities for quantisation. To ensure optimal performance, these layers should be subjected to fine tne-grained non-linear quantisation strategies.
layer; 2) Certain tensors (e.g. K) consistently have greater variance compared to others; 3) All the weight variance smaller than activations. Similar trends can be observed in other LLMs. We provide a variance plot of Vicuna-7B (Zheng et al., 2023) in Appendix (Figure 4).
Table provides comparison between our work and existing LLM quantisation methods. Our quan- tisation considers all GEMMs (8/8) in transformer layers and both Post-Training Baaiiing-QQantisatiion (PTQ) and Training-Afern-uuaattttion (TAQ) scenarios. In this work, we also explore suitable places to perform TAQ and quantisation search within the
The presence of varying numerical ranges across layers and tensors poses a challenge to the efficacy a single quantisation configuration for the en-

---

entire NLP pipeline. We make the following con- tributions:
is worth noting that sharing scaling factor is similar to, but not necessarily the same as, sharing the exponent (Darvish Rouhani et al., 2020). This distinction arises because scaling factors can be arbitrary float32 values, whereas exponent val- ues must be integers represented by the assigned number of bits. Our work focuses on sharing the exponent exponent bias. When the block size of the shared exponent is 1, fall back to the minifloat representation such as FP8 (Sun et al., 2019). These approaches showed promising re- sults primarily for vision models or relatively small Transformer-base -bssed models, while we shift the fo- cus to quantising LLMs with significantly larger parameter count.
We address the LLM quantisation problem with activation outliers and examine it as a scaling offsets problem from an arithmetic de- sign perspective. We demonstrate the efficacy of family of arithmetic systems with short exponents shared across block of numbers.
We propose a novel quantisation framework based on block arithmetic, and demonstrate its effectiveness in performing W6A6 inference for various tasks. Our nearly-lossless W6A6 outperforms prior work in terms of arithmetic density and memory density, without requir- ing data calibration fine-tuning.
2.2 LLM Quantisation
We present two methods to achieve 4-bit quan- tisation on downstream tasks: one is fine- tuning-based, and the other is miixed-precisoo search. The latter further demonstrates the po- tential advantage of shifting LLM inference to cost-effective ASICs.
Efficient quantisation techniques for language models have been explored in previous works. Zafrir et al. proposed an approach for quantising BERT (Shen et al., 2019) into 8-bit integers (Zafrir et al., 2019), while Shen et al. (Shen et al., 2019) proposed
ultra-low precision quan- tisation for the same model. Zhang et al. (Zhang et al., 2020) quantised BERT to ternary values lever- aging layer-wise knowledge distillation, and Bai et al. (Bai et al., 2021) further pushed the quantisation of BERT weights binary values.
2 Related Work
While quantisation of earlier Machine learning (ML) models has been extensively studied, effec- tive quantisation of LLMs still remains an open problem. In this section, we review the previous works on block-based quantisation and compare to the existing LLM quantisation techniques.
The recent surge of interest in quantising LLMs has presented a unique challenge distinct from the prior art summarised above. This challenge stems from the increased model sizes of L.Mss. Yao et al. proposed ZeroQuant, which quantises both weights and activations of large transformers into small integers with shared scaling factors (Yao et al., 2022). However, as mentioned by Xiao et al. (2022), ZeroQuant suffers from severe accuracy loss. Dettmers al. introduced LLM. inti (), method that computes outlier GEMMs in float16 and the rest in 8-bit integer (Detmmers et al., 2022). Xiao el al. extended 8-bit LLM quantisation with their PTQ technique named SmoothQuant, Xiao el al. proposed SmoothQuant which scales down activations by row and scales up weights by col- umn proportionally before 8-bit fixed-point quanti- sation (Xiao et al., 2022). Frantar et al. proposed GPTQ, which quantises the weights of LLMs or 4-bit integers while keeping the activations in float32. Most LLM quantisation methods, di- rectly or indirectly, reserve LLM activation outliers.
2.1 Block-based Quantisation
Block-based quantisation a technique that quan- tises a block of values into a compact format, where the elements within each block share common dig- its. This technique offers significant memory foot- print reduction while maintaining minor round- off error. A number of previous works rely on this method to quantise Convolutional Neural Net- works (CNNs). Lin el al. utilised a linear com- bination of multiple binary bases, equivalent to each binary matrix having a scaling factor (Lin et al., 2017). Subsequently, Zhang et al. intro- duced LQ-Nets that rely on form of block quan- tisation with shared scaling factor at the vec- tor level (Zhang al., 2018). Further investiga- tions explored grouping numbers at various gran- ularities, including layer-wise (Wu et al., 201 1b),, channel-we wise (Krishnamoorthi, 2018), and vector- wise quantisation (Dai et al., 2021).

---

(FP32)
sign, exponent, mantissa
Exponent
IEEE Floatté (FP16)
MiniFloat: Demormed Minifloat (DMP)
Mantissa
1-bit sign, 5-bit exponent, 10-bit mantiisa
1-bit sign, 4-bit exponent, 3-bit mantiisa
Exp
Block Minifleat (BM)
Block Flonting Point (BFP)
Block Logorithmn (BL) 1-bit sign. E bit expoment
1-bit sign, E: bib exponent, M bit mantissu B-hit shared exponent bias
1-bit sign. M bit maniisua E-hit shared exponent
B-bit shared exp bias
exp
shured exp
shared exp
270
#51
2x
Figure An illustration of different quantisation methods considered in this work: MiniFloat (Sun et al.. 2019) and Denormed MiniFloat (DMF), Block MiniFloat (BM) (Fox al., 2021), Block Floating-Pone (BFP) (Darvish Rouhani et al., 2020) and Block Logarithm (BL).
3 Method
MiniFloat and Denormalised MiniFloat Mini- Float is an efficient floating-point representation that requires fewer bits than traditional floating- point numbers. Traditionally, an 8-bit MiniFloat inherits the definition of FP32 by assigning E = 4 and M = 3. We saturate MiniFloat when e 2 1 and thus + inf is included.
this section, we outline our quantisation strategy for LLMss. We first define block-based quantisation and then describe the metrics we use for evaluating quantisation methods. Finally, we detail preci- sion search that lowers the quantisation granularity down the tensor level, effectively accommodat- ing the statistical distribution inherent in LLMs.
In this paper. we also introduce a Denormalised MiniFloat (DMF) with zero as the implicit lead- ing bit in the mantissa. Similar to MiniFloat, we saturate the infinity to a maximum finite value. DMF provides a higher precision than MiniF-ooat for small values the expense of narrowing down the value range. We investigate this trade-off in the context of quantising LLMs.
3.1 Block-based Arithmetic
Figure illustrates the data representation we ex- plore address LLM quantisation as well as the standard float32/float16. We outline the speci- fications for traditional floating-poing point numbers and extend them to block-based quantisation. Detailed definitions can be found in Appendix C.
Block MiniFloat, Block Floating-Point and Block Logarithm As shown in Figure 2, Block quantisation packs values in block in which common scaling factor shared across N values where N is the block size, reducing the computa- tion in vector inner products. This work mainly explores three block quantisation arithmetics on LLMs: BM, BFP and BL.
Standard floating-point A standard IEEE Hoating-poin number defined as a 4-tuple. (s, e, m. b) (Kahan, 1996). & E {0, 1} is the sign bit, E N is the exponent field; E N is the ex- ponent bias; and m E N is the mantissa. Let the bit widths of the exponent and the mantissa be E and M, respectively. The IEEE standard float32 (FP32) number has E 8 and M 23, where the other bit used as the sign bit. Note that the exponent bias depends on F: 2 1, sepa- rating the exponent field symmetrically. Similarly, 16 (FP16) has E and M 10.
Block Minifloat (BM) shares B-bit exponent bias (Fox al., 2021). This representation achieves high precision and high range at the same time, the cost of a larger quantisation error medium value than standard floating point. This is poten- tially amenable to values in multimodal distribu-

---

tion, where values close a peak can be efficiently represented in a block. Block Floating- Point (BFP) shares an E-bit exponent. This shared exponent bounds the range the block and is amenable to values with small block variances. Block Loga- rithm (BL) sets the mantissa in BM to I and shares a B-bit exponent bias, resulting in values that are powers-of-twos. This contrasts with BFP and is amenable to values with large dynamic ranges.
Method
Config
E
M
B
Fixed-poine
W8A8
MiniFloat
W8A8
4
3
DMF
W8A8
4
BFP
W8A8
8
BFP
W6A6
BFP
W4A4
BM
W8A8
8
BL
W8A8
7
8
All these quantisation methods are non-linear and thus can be useful tools to address the sealing offsets phenomenon depicted in Figure More- over, the hyper-ppau parameter block size allows for flex- ible quantisation granularity, ranging from layer- wise, tensor-wise, and channel-wise, to slice-wise (a slice along the token/channel vector).
Table The quantisation configuration used the following sections, where E, M, and B are the bit- width of exponent (shared exponent), mantissa, and bias (shared bias) respectively.
precision. The search space increase exponentially as the layer count increases. We leverage accu- racy and memory density to design the objective function: O acc + a mem. Here O,, acc, mem represent the objective function, accuracy, and memory density of the searched quantised mod- cls, respectively. The constant a is used to balance acc and mem. To determine the a for a specific search, we initially set a to 1.0 and perform the search while recording the values of (acc, mem) until convergence. The final value of a is deter- mined as mem.' where (acce, mem.) represents the converged values. Detailed search parameters in Appendix B.
3.2 Arithmetic and Memory Densities
Reducing model size is not the only advantage of quantisation; it also simplifies the computation, thereby accelerating inference. We evaluate quanti- sation arithmetics using adopted memory and arith- metic densities (Darvish Rouhani et al., 2020). We define memory density as the reciprocal of the size of the activation and weight data in a model, and the arithmetic density as the reciprocal of the area/the number of Look- Look-Up-Tie Up-TTales (LUTTs) to synthesise muutiply-accumullan (MAC) unit, which serves the basic cell for matrix multiplication in cus- tom inference circuits. An efficient quantisation method should make good trade-off among task accuracy, memory density, and arithmetic density. We implemented MAC units with different above- mentioned arithmetics in FPGAs to obtain the num- ber of LUTSs. A detailed description of this proce- dure can be found in Appendix D.
acc.
Evaluation
We conducted a comprehensive set of experiments to identify the key factors influencing the perfor- mance of sub-8-bit LLMs. We begin with lan- guage modelling task to eliminate less promising quantisation methods (Section 4.2), and then the promising ones on downstream tasks. For the tasks that proved challenging even for FP32 mod- els, we resort to fine-tuning. Additionally, we conducted a mixed-precision search on two tasks where the quantised 4-bit model struggle. The re- sults of this search provide insights into how to further refine quantisation at the tensor level.
3.3 Quantisation Search
Previous works (Dong et al., 2019; Habi et al., 2020) observed that the layers in CNNs exhibit varying tolerance, or "sensitivity", to quantisation we also notice this phenomenon in LLMs. The crucial aspect is identifying the layers that are sen- sitive and determining tailored quantisation config- urations. To achieve this, we apply Tree-structured Parzen Estimator (TPE) (Bergstra et al., 201 1) to conduct fine-grained search for quantisation pre- cision multiple times and analyse the statistics in- herent in the quantised models that recover more accuracy. Our search space constructed on a per- tensor basis, allowing each input tensor or weight tensor in 1- (See Algorithm 2) to have its own
4.1 Experiment setup
Baselines We compare approach with four baselines: 8-bit plain fixed-point quantisation, LLM. int8() (Dettmers al., 2022), GPTQ (Fran- tar al., 2022), and SmoothQuant (Xiao et al., 2022). We amend SmoothQuant's source code to ensure its consistency with their paper (See Ap-

---

Perplexity (L)
Hardware metrics
Method
Config
125M
350M
1.3B
2.7B
6.7B
Mem 1
Arith 1
FP32
27.65
22.00
14.62
12.47
10.86
1x
1x
LLM. (
W8A8t
27.72
22.03
14.64
12.49
10.86
2x
7.7x
GPTQ
W4
31.12 -
24.24
15.47
12.87
11.39
1.6x
SmoothQuant
W8A8
14.62
12.50
10.85
< 4x
< 7.7x
SmoothQuant-c
W8A8
_1
_7
17.97
26.88
42.90
4x
7.7x
Fixed-point
W8A8
275
117
1.78E4
7.81E3
3.77E3
4x
7.7x
MiniFloat
W8A8
28.16
22.24
15.03
12.73
10.99
4x
17.4x
DMF
W8A8
30.41
23.89
18.08
14.55
11.95
4x
17.4x
BFP
W6A6
28.27
22.22
15.08
12.54
10.90
4.9%
19.2x
BFP
W4A4
41.94
33.98
24.70
19.34
13.59
7.1x
37.3x
BM
W8A8
5.6E3
2.7E4
1.17E4
1.33E4
8.61E3
3.8x
14.4x
BL
W8A8
780
1.26E3
323
950
289
3.8x
16.1x
Table 3: Perplexity (4) values with zero-shot Post- Dost-Training-o QuaiiiisttQnniiiioinnnn (PTQ) WikiText2, this means directly quantise the pre-trained model and apply on WikiTex12. Mem and Airth represent Memory and Arithmetic density accordingly. DMF, BM, BFP and represent Denormalised MiniFloat, Block Minifloat, Block Floating Point and Block Logarithm respectively. SmoothQuant-c multiplications are now also quantised. means the inliner matrix multiplications are calculated in 8-bit fixed-point, and outliers are calculated in FP16. means the weights of GPTQ are kept in FP32. means Smooth Quant repository does not include the weight scaling matrices for 125M and 350M. We highlight the best block- based quantisation arithmetic, bit BFP, considering perplexity, memory density, and arithmetic density together.
improved implementation where the two activation matrix
Model
FP32
LLM. int8()
W6A6 BFP
Models and datasets We choose the represen- tative OPT (Zhang et al., 2022) family, and evaluate Wikitext2 (Merity al., 2016), ARC(easy) (Clark al., 2018), LAMBADA (Pa- perno et al., 2016), PIQA (Bisk et al., 2020), COPA (Roemmele et al., 2011), QNLI (Wang et al., 2018), SST2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), and COLA (Warstadt et al., 2019). To demonstrate the generalizability of our method, we also report the Wikitex12 perplexity of quantized LLaMA models (Touvron et al., 2023; Chiang al., 2023; Taori al., 2023). Following prior work (Zhang al., 2022; Xiao al., 2022), we use Im-eval-harness (Gao et al., 2021) to evaluate models on downstream tasks in the con- text of zero-shot prompting.
LLaMA-7B
5.79
5.83 (+0.04)
5.83 (+0.04)
Vicuna-78 7B
7.06
7.07 (+0.01)
7.08 (+0.02)
Alpaca-7B
7.01
7.02 (40.01)
7.02 (+0.01)
LLaMA-13B
5.17
5.22 (+0.05)
5.20 (+0.03)
Vicuna-v1.5-13B
6.13
6.16 (+0.03)
6.16 (+0.03)
Table Perplexity ()) values LLM family quantized by W6A6 BFP. We compare method with FP32 and LLM. int8() () and find that our method achieves nearly lossless perplexity on Wikitext2. We exclude GPTQ and SmoothQuant-c in this table because they have obvious perplexity increase larger than 0.2 and 5.0 respectively.
pendix B) and add this amended version (referred to as "SmoothQuant-e") to the result table.
4.2 Zero-shot PTQ on Wikitext2 and downstream tasks
Quantisation configuration Table 2 clarifies the quantisation configuration used in the following sections, where E, M, and B are the bit-width of exponent (shared exponent), mantissa, and bias (shared bias) respectively. All these representations include 1-bit sign bit. The block size of block- based methods is set to [1, 16] for both the weight and activation matrix (a slice along matrix row in Algorithm 2) unless otherwise specified.
In this section we present our results in setup we call zero-shot Post- SoadTTraiing-- Quaiiiisainniiiitioonn (PTQ), which also adopted by prior work on LLM quantisation (Dettmers al., 2022: Frantar et al., 2022: Xiao et al., 2022). In this approach, we take a pre-trained OPT model from Huggingface, quantise it, and apply it on Wikitext2 to calculate

---

Mean accuracy (t.%)
Method
Config
125M
350M
1.3B
2.7B
6.7B
Float32
52.7
57.5
69.6
63.4
73.4
LLM.int8()
W8A8
52.5 (-0.2)
58.3 (+0.8)
69.2 (-0.4)
65.3 (-0.1)
73.5 (+0.1)
inM.4
W4A4
50.8 (-1.9)
55.8 (-1.7)
67.0 (-2.6)
64.5 (-0.9)
72.5 (-0.9)
SmoothQuant-c
W8A8
67..
(-0.2)
72.2 (-1.2)
MiniFloat
W8A8
52.1(-0.6)
(5.1(-2.4)
64.7(-4.9)
65.7(+0.3)
70.5(-2.9)
BFP
W4A4
47.8 (-4.9)
51.7 (-5.8)
57.2 (-12.4)
(-9.7)
67.2 (-6.2)
BFP
W5A5
51.1 (-1.6)
56.8 (-0.7)
65.5 (-4.1)
(-.6)))
72.0 (-1.4)
BFP BFP
W6A6
52.6 (-0.1)
57.6 (+0.1)
67.8 (-1.8)
65.5 (+0.1)
72.9 (-0.5)
W8A8
52.8 (+0.1)
57.6 (+0.2)
69.1 (-0.5)
65.2 (-0.2)
73.1 (-0.3)
Table 5: Mean accuracy (f, %) values with zero-shot prompting PTQ on ARC (casy), COPA, LAMBADA, PIQA, and SST2, this means we directly quantise the pre-trained trained model and benchmark on these downstream tasks using zero-shot prompting, We highlight 6-bit BFP which also achieves accuracy close to FP32 on these tasks.
perplexity, and the eight downstream tasks short- listed in Scction 4.1 to calculate accuracy. The zero-shot PTQ setup is particularly advantageous in scenarios where LLMs lack prior knowledge, it eliminates the need for downstream task fine- tuning and Training Training-After--e -ie--Afftetetisaatttiii Quantisation (TAQ).
LLM. int8() as baselines. We observe that 6-bit BFP still achieves nearly lossless perplexity on these models, verifying the efficacy of our method across model architectures.
Accuracy downstream tasks We exclude fixed-point, DMF, BM, and BL from downstream task evaluation due their poor language mod- elling performance. Table represents the mean ac- curacy on ARC (easy), COPA, LAMBADA, PIQA, and SST2. The results of QNLI, MRPC, and COLA are not included in this table as even FP32 LLMs exhibited poor accuracy close to random guess. A plot depicting how these methods match FP32 curacy the model scales up and complete result table are in Appendix E.
Perplexity Wikitext2 Table compares our results with the baselines in terms of perplexity, memory density, and arithmetic density. Similar to prior work (Dettmers et al., 2022; Xiao et al., 2022), plain fixed-point quantisation performs poorly. In contrast, non-linear arithmetic, such as MiniFloat, yields significantly better perplexity at similar memory density. MiniFFoot yields slightly better results than DMF, indicating the 2x higher range is more important than precision in this context.
Besides LLM. int8() and SmoothQuant-c, we also report a 4-bit version LLM. int8() (referred to as LLM. in)4()) reported by Dettmers (2023) on downstream tasks. We observe that 6-bit BFP achieve nearly lossless accuracy, below FP32 and LLM. inn8()() and above SmoothQuant-c and LLM. int4(). Note that 6-bit BFP has the high- est memory density and arithmetic density among these methods. The b-bit BFP suffers severe accu- racy degradation because its shared exponent and 3-bit mantissa cause large quantisation errors. Overall, we make the following observations:
Block-baser quantisation exhibits inconsistent performance on Wikitext2. A noteworthy result is that our 6-bit BFP achieves higher memory den- sity, higher arithmetic density, and lower perplex- ity than the prior art GPTQ and SmoothQuant-c without requiring data calibration. BM and BL per- form poorly compared BFP. BM was originally proposed in the context of Quantisation-Aware Training (QAT), whereas our evaluation based on PTQ. Without retraining, the 3-bit mantissa of BM and the 1-bit mantissa of BL. may be the reason for the poor perplexity.
Fixed-point representation performs inade- quately due to unability linear quantisation to address the scaling offset issue caused by varying variances.
Table shows the perplexity of W6A6 BFP on LLaMA family, including LLaMA-7B/-13B (Tou- vron et al., 2023), Vicuna-7B (Zheng et al., 2023), Alpaca-7B (Chiang et al., 2023), and Vicuna- v1.5-13B (Chiang et al., 2023), with FP32 and
LLMs have different tolerance to block-based

---

4-bit
5-bit
6-bit
7-bit
Percentage occurrence
0.5
6
12
18
20
24
26
30
32
Layer ID
Figure 3: The bit width distribution of Q in Line 6, Algorithm 2 from 2688 searches. We identify the layers less often need more bits than
tolerant to aggressive quantisation in OPT-2.7B. For example, layers 18, other layers. Keeping these layers in relatively high precision recovers the accuracy from 36.2% to 61.3% without decreasing the memory density, equivalent to a 4.3-bit OPT-2.7B on average.
quantisations. BM and BL exhibit subpar per- formance compared BFP, indicating that non-linear quantisation still needs sufficient mantissa length capture the learned weight distribution, or retraining may be required.
retrain this quantised model downstream tasks, which refer as TAQ on downstream tasks.
We compare these two cases on four downstream tasks (SST2, QNLI, MRPC, and COLA) that zero- shot prompting struggles to handle. The result table is in Appendix F. We observe that:
BFP strikes a good balance in the trade- off between range and resolution. Our nearly-lossless 6-bit LLMs, without data calibration/rr-ttraiiigg outperform prior art methods in terms of perplexity (accuracy), memory density, and arithmetic density.
Both options effectively improve accuracy, abling nearly lossless downstream accuracy even if 4-bit BFP is applied.
TAQ on downstream tasks reaches a slightly better accuracy (a gain of 0.2% on average) than PTQ fine-tuned FP32 given the same bit-width. However, the former is harder optimize through backpropagation because of the forward quantisation error and the Straight- Through Estimator (STE) (Bengio et al., 2013) used backpropagation.
We also observe that sub-6-bit BFP has severe accuracy drop. To address this problem, we fur- ther investigate two approaches for improving the accuracy of 4-bit LLMs.
4.3 4-bit LLMS via fine-tuning
Previous study (Brown et al., 2020; Zhang et al., 2022) reported FP32 LLMs' low accuracy on sev- eral downstream tasks in the context of zero-shot prompting. In our experiments, OPTs also exhibit poor accuracy on QNLI, MRPC, and COLA. Fine- tuning language models downstream tasks has proven be helpful for improving accuracy (De- vlin al., 2019). We explore the fine-tuning and quantisation LLMs on downstream tasks.
4.4 4-bit LLMss via mixed precision
Currently, our block-based quantisation uses uni- form configuration, where the block size and bit- width remain constant across the entire model. What if we push the barrier further? Existing works on CNN compression have explored mixed- precision quantisation (Wu al., 2018a; Wang et al., 2019), thereby increasing memory density. This subsection lowers the block size granularity and the bit-width granularity to the tensor level demonstrate uncharted possibilities of aggressive LLM quantisation.
There are two stages where quantisation can be applied. LLMs are typically pre-trained in FP32. The first option is continue fine-tuning the FP32 model downstream tasks and subsequently quan- tise this fine-tuned FP32 model. We refer to this setup as PTQ on fine- tine-teeed FP32. The second op- tion is to quantise the rreitrained FP32 model and
Variation-aware block size By comparing the activation variance and weight variance Figure I, we observe that the weight variance remains stable

---

and much smaller, suggesting that we can increase the weight block size while decreasing the activa- tion block size. This approach enhances accuracy while maintaining memory density.
Limitations
Different from many prior arts LLM quantisation that focus on integers, our work puts particular em- phasis on minifloat variants. However, the potential gains of our work have not manifested in GPU sys- tems due to a lack of CUDA kernel implementation. The implementation of some proposed quantisation methods in this paper requires specialised kernels and hardware, however, major focus of our work to explore potential designs for next- next-generation hardware to run LLM inference. Another limita- tion is that our search algorithm does not include arithmetic density due to a lack of hardware models for LLMs. We ran a mixed-precision search with hardware models on a small transformer. The result included in Appendix G is promising. We leave suf- ficient study on hardware-aware LLM quantization as a future work.
We repeat the quantisation search described in Section 3.3 on downstream tasks and filter out less promising quantisation con- figurations using an accuracy threshold and a mem- ory density threshold. Each time we start TPE search with a different random seed, so the distribu- tion of filtered quantisation configurations exposed the sensitivity of the searched tensors in LLMs. An example of mixed -eexedprecision search result is pre- sented in Figure 3. We find certain layers were con- sistently assigned with higher precision, while oth- ers tended to have lower bit widths. By preserving high precision for these sensitive layers, we recov- ered the 4-bit LLM accuracy from 36.2% to 61.3% on LAMBADA without compromising memory density. The memory density of the searched OPT- 2.7B is 7.42x, which is slightly better than the uni- form 4-bit BFP's 7.1 1 Figure 7 in Appendix G compares uniform 4-bit BFP and mixed-precision 4-bit BFP on LAMBADA and ARC (easy), high- lighting the effectiivness of our mixed-precision quantisation. We include more tasks and model sizes in Appendix G. In conclusion, variance-awaa aware block size and mixed precision allow aggressive quantisation beyond 6-bit without fine-tuning.
References
AMD Xilinx Vivado. 2023. [link].
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang. Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2021. Binarybert: Pushing the limit of bert quantization.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprial 1388:3333.3433
5 Conclusion
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper- parameter optimization. Advances in neural information pro- cessing systems, 24.
This study focuses on addressing the scaling off- set issue in LLMs and provides valuable insights into the quantisation of LLMs. Through exten- sive experimentation, we identify key factors that significantly impact LLM quantisation. When aim- ing for quantisation above or equal to 6-bit, BFP surpasses previous methods in terms of accuracy, memory density. and arithmetic density, without requiring for data calibration or training. Moreover, we demonstrate that fine-tuning or mixed precision techniques enable 4-bit LLMs on downstream tasks. Fine-tuning is suitable for GPUs, and mixed preci- sion has the potential to shift the inference platform from GPUs to cost-effective ASICs. Our findings contribute advancing the field of LLM quantisa- tion and provide practical guidance for achieving good quantisation performance. Our work has been open-sourced and will also contribute to an MI. hardware framework named MASE for hardware deployment (Cheng et al., 2023).
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence.
Sid Black, Leo Gao, Phil Wang, Connor Leahy. and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive I Language Modeling with Mesh- Tensorflow. If you this software, please cite it using these metadata.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109. 12948.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 197-1

---

Jianyi Cheng, Cheng Zhang, Zhewen Yu, Alex Montgomerie-Conn Corcoran. Can Xiao, Christos-Savvas Bouganis, and Yiren Zhao. 2023. Fast prototyping next-generation accelerators for new ml models us- ing mase: MI accelerator system exploration. arXiv preprint arXiv: 155177
International Conference on Learning Representa- tions.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-traie training quantization for generative pre-trained transformers. arXiv preprint arXi::2200.170. 17323.
Wei-Lin Chiang, Zhuohan Li, 7i Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang. Laria Reynolds, Eric Tang. Anish Thite. Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for fow-shot language model evaluation.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arviv: :rxXvv180:: 1803.05457.
Hai Victor Habi, Roy H Jennings, and Arnon Netzer. 2020. Hmq: Hardware friendly mixed precision quantization block for cnns. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pages 448 463. Springer.
Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zim- mer, William Dally, and Brucek Khailany. 2021. Vs- quant: Per-vector scaled quantization for accurate low-precision neural network inference. Proceedings of Machine Learning and Systems, 3:873-884,
William Kahan. 1996. Ieee standard 754 for binary floating-point arithmetic. Lecture Notes on the Status of IEEE, 754(94720-11766:11
Bita Darvish Rouhani, Daniel Lo. Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vino- gradsky, Sarah Massengill, Lita Yang, Ray Bittner, al. 2020. Pushing the limits of narrow precision in- ferencing at cloud scale with microsoft floating point. Advances in neural information processing systems, 33:10271-1000 10281.
Kari Kalliojarvi and Jaakko Astola, 1996. Roundoff errors in block-ftoating-pun point systems. IEEE transac- tions on signal processing. 44(4):783-790.
Raghuraman Krishnamoorthi. 2018. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.0088 080.00322.
Tim Dettmers. 2023. Bitsandbytes.
Xiaofan Lin, Cong Zhao, and Wei Pan. 2017. Towards accurate binary convolutional neural network.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 O: 8-bit matrix mul- tiplication for transformers at scale. arXiv preprint arXiv:2208.00 073399
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint 1X00:1810004805
Daisuke Miyashita, Edward Lee, and Boris Mur- mann. 2016. Convolutional neural networks us- ing logarithmic data representation. arXiv preprint arXiv:1603.0100
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
OpenAL. 2022. Introducing chatgpt. Accessed on June, 23th, 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring broad discourse context. Proceedings of the 54th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing SIaiiinggWIWWW (IWP2005).
Zhen Dong. Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed- precision. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 293- 302.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon, 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Sean Fox, Seyedramin Rasoulinezhad, Julian Faraone, Philip Leong, et al. 2021. block minifloat rep- resentation for training deep neural networks. In

---

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2019. Q-bert: Hessian based ultra low pre- cision quantization of bert.
Bichen Wu, Yanghan Wang. Peizhao Zhang, Yuan- dong Tian, Peter Vajda, and Kurt Keutzer. 2018a. Mixed precision quantization of convnets via differ- entiable neural architecture search. arXiv preprint arXiv:
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. 2018b. Training and inference with integers in deep neural networks. arXiv preprint orXiv:8002 1802.046800
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. arXiv preprint arXiv:2211. 104338
Jianlin Su, Yu Lu, Shengfeng Pan. Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint 2xxx:2104.09880
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang. Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168- 27183.
Xiao Sun. Jungwook Choi, Chia-Yu Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishman. 2019. Hybrid 8-bit floating point (hfp8) training and inference for deep neural net- works. Advances in neural information processing systems, 32.
Otir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8hert: Quantized 8bit BERT. In 2019 Fifth Workshop Energy Efficient Machine Learning and Cognitive Computing NeurlPS Edi- rion (EMC2-NIPS). IEEE
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An catructioo-followiinn Ilama model. https:// github. wtw./otttm tatsu-ttss-aba labuuullbsssaa /t/s/standddaaaa
Dongging Zhang, Jiaolong Yang. Donggiangzi Ye, and Gang Hua. 2018. Lq-nets: Learned quantization for highly accurate and compact deep neural networks.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen. Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint ar XrXiv:2205.0006
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13397.
Wei Zhang. Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Ternryyber: Distillation aware ultra-low bit bert.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Mlaa Polosukhin. 2017. Attention is all you need. Advances in neural information proceessinnpooosssingg systems, 30.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chathot arena. arXiv preprint wxXiv23366.05685
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
A Tensor variance in LLMs
We investigate the variance trend of Vicuna- 7B (Chiang al., 2023) and observe the same trend of increasing activation variances as OPT-6.7B. Fig- ure 4 depicts the forward pass of Vicuna-7B and the variance trend of its tensors. Interestingly, the vari- ances of self-attention imput activations (Q₂ and KJ in line 6 of the algorithm above) are consistently high from the first Transformer layer to the last in Vicuna-7B. We assume this is because of the Ro- tary Positional Encoding (RoPE) (Su et al., 2021) layers.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. 2019. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612-8620.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
We additionally analyse the trend of inereasing activation variance when the model size increases,

---

Algorithm 2 Transformer layer (Vicuna)
X;
Require: X
Input features
2
Q
Require: H
P Number of heads
K
Noje))Norm(X)
V.
2: for [0, H) do
B
3: Q: Wn
2-3
4:
K; f WK, X
5:
<- Wv
6:
Q 4 RoPE(Q;), K RoPE(Ki)
Variance
D
A, + giKe"
2-7
Wa
7:
Vdl
WK.
8:
Å. + softmaal A., aais 4 -1)
Wv.
9: B. + V.A.
Wo
10: end for
2-11
We
11: B. + concul( Bo, BH-1)
Wo
12: B. <- WoB.
WD
13: B. <- RMSSNom B. Xn))
2 15
14: G +- WaBn
15: U + Wu
10
15
20
25
30
16: D + (D(SiLU(G X U)
Layer
17: 0 + + Ho +
18: return O
Figure 4: The algorithm on the left the forward pass of Vicuna-7B. The graph on the right depicts how the tensor variances change with layer number. We observe the same trend of increasing activation variances in Vicuna-7B, which is similar to OPT-6.7B. interestingly. the variances of self-attention input activations (Q₃ and K2 in line 6 of the algorithm above) are consistently high from first Transformer layer to last in Vicuna-7B. assume this is because of the Rotary Positional Encoding (RoPE) (Su et al., 2021) layers.
Figure 1, for OPT-6.7B, we plotted the variances of all tensors that have unbounded input ranges and that are taken as input operands to matrix multipli- cations in the Transformer layer. Figure further illustrates the results for OPT-350M and OPT-2.7B. We observe that:
B Experiment details
B.1 Setup and Implementation
Hardware resources We run the experiments us- ing four NVIDIA RTX3090s, three A100s, and eight V100s with 64GB, 192GB, and 128GB RAM respectively. The evaluation of PTQ perplexity on Wikitext2 takes around 64 GPU hours in to- tal; the zero-shot prompting evaluation on down- stream tasks takes around 160 GPU hours in total; the fine-tuning of FP32 models on SST2, QNLI, MRPC and COLA takes around 30 GPU hours in total; the fine-tuning of quantised BFP models takes around 70 GPU hours in total; the evaluation of fine-tuned models takes around 6 GPU hours in total; the mixed-precision search takes around 120 GPU hours in total.
If we consider V, Be and B₁ as the main information path 2, these components have much smaller variances than K and Q.
Bigger models tend to have small variances at shallow layers and larger variances at deep layers.
These observations explain why linear quanti- sation, such integer quantisation, is effective for smaller models but struggles with larger ones, This increasing activation variance trend can be considered into variance-aware block size. Since a higher variance implies a higher possibility of extreme outliers, we can apply larger block sizes to those tensors with smaller variance and smaller block sizes to those with higher variance. Limited by time, leave this exploration well as the combination of fine-tuning, variance-aware block size, and mixed precision in future work.
Implementation We download the model codes and pre-trained weights from HuggingFace Trans- formers 3 and implement the quantisation arith- metics using PyTorch 4 We use Vivado to re- port arithmetic density and Optuna to perform the mixed-precision search.
'https://githhnn http.///wgggnaaee/errssfoooooomm http:://ggtthb...cm/ptttrrhh//// 'https://optuna" headdteeeeoccc.ioo////tttaaaa index. html
"The computation of Q and K yields attention factors (post-softmax) that are applied to V.

---

B.
B1
B
23
25
21
Variance
2-3
Varionce
2
2-7
2-7
2-11
2 2-11
2-15 15
2 2-15
10
15
10
15
20
25
Layer ID
Layer ID
(a) OPT-350M
(b) OPT-2.7B
Figure 5: We demonstrate a similar analysis to Figure I, where on the left we have OPT-350M variance vs layer ID and OPT-2.7B variance layer ID on the right, The trend of increasing activation variance is obvious larger models.
Evaluation We follow the code base of GPTQ (Frantar ul., 2022)6 estimate LLM's per- plexity on Wikitext2. We chop Wikitext2's test set into sequences of 2000 tokens, feed the sequences to LLMs, and normalise the cross entropy loss by the sequence length and batch size. To evaluate LLM accuracy downstream tasks, we follow OPT (Zhang et al., 2022) and SmoothQuant (Xiao al., 2022) use 1m-eval -hannss in the zero- shot prompting setup.
plain 8-bit fixed-point quantization. In LLM. int8() all the tensors are stored as FP16 numbers, which
is the reason why LLM. int8() has 2x memory density while plain 8-bit fixed-point has 4x mem- ory density in Table 3. LLM. int8 targets GPUs while ours targets ASICs. LLM. int8() is not as friendly as uniform BFP to ASICs, because LLM. int8 separates two (one for inliers the other for outliers), casts inliers 8-bit, performs 8-bit matrix multiplica- tion for inliers and an FP16 matrix multiplication for outliers respectively. This separation is per- formed on the fly. In comparison, 6-bit uniform BFP does not require runtime separation or an FP16 matrix multiply engine. All tensors are stored and calculated in 6-bit BFP.
matrix multiply into
B.2 Comparison with SmoothQQant
The SmoothQuant paper (Xiao al., 2022) de- clares all the eight GEMMs (0-3 2) are quan- tised. However, their codes? do not support quantis- ing ( and 6, which takes up 19.6% floating-poine operations (FLOPs) in OPT-6.7B's self-attention. We amend their code and refer to the amended version as "SmoothQuant-c", which should be the SmoothQuant-02 in the paper. We observe that SmoothQuant-c has much higher perplexity and slightly lower accuracy on downstream tasks than SmoothQuant. Besides, the SmoothQuant repository does not include the scaling factor files of OPT-125m and OPT-350m, so the perplex- ity/accuracy for these two models is missing in our result table.
B.4 Quantisaion search
The specific search configuration depends on model size, task, and FP32 performance. We use the ac- curacy threshold and memory density threshold to sort out promising mixed-precision configs. Given model and task, the accuracy threshold is 2% below the FP32 values. The memory density is set 7. in most search configs.
Note that to estimate the memory density of quantisation config candidates, we need the model architecture information including input sizes and weight sizes for all the GEMMs in Algorithm 2 across all layers. We implement a FLOP profiler to collect this information and feed as input the search algorithm. The numeric values of these parameters can be found in the bash scripts of our
B.3 Comparison with LLM. int8()
We give short comparison between LLM. int8() and method. LLM. int8() is different from
hhttps://githhu wtt.:/ooommmi/sseeee htttps: http//gtttttttttt/ cww/mitccmmm.mmm.eeee.... hah- lab.smothhuuunn

---

source code.
Similar to BM, the shared exponent bias is also determined by the maximum FP32 values when casted from FP32.
C Definition of quantisation arithmetics
FP32, FP16 and MiniFloat A traditional floating-poin representation follows IEEE floating- point standard (Kahan, 1996), which can define seating--pont number as 4-tuple, (s, e, m,b)) where
Logarithm quantisation was proposed by Miyashita et al. (2016) to perform QAT on CNNs. Block Logarithm (BL) was used as a baseline compare with BM in (Fox et al., 2021), BL shares the exponent bias and does not have mantissa bits (mantissa is always 1).
E {0, 1} is the sign bit of the number;
Basically, block-based quantisation facilitates the vector's inner product by simplifying the ac- cumulation after multiplication. For example, the inner product between two BFP vectors x and y is,
E is the exponent field;
E N the exponent bias; and
m N is the mantissa.
x =( =- [z], (I,....IT--1) (-1)"ye" (1....;18-11
Given the bit widths of the exponent and the man- tissa be E and M, the value z of floating point number can be obtained via:
(4)
= < (ar +.... 18-
where is the block size. Since exponents are shared across vectors, the element products can be accumulated without shifting. The block sizes of the two vectors are not necessarily the same.
1) x x 2 x 2
0
1) 2 x (1+
h 0 0<e<2" -
x
== 2 1,m =
NaN
others
(1) where e is the unsigned integer value repre- sented by the exponent bits, and m is the un-
D Estimate arithmetic density via logic synthesis
signed integer value represented by mantissa bits. The exponent bias (b) is a constant depending on E: b = 2 1. FP32, FP16, and MiniFloat have E = M = 23, E = M = 10 and = M, = 3. respectively. Note that the "1" in the fraction term of Line 2, Equation (1) comes from the implicit leading bit in the mantissa.
We implemented the hardware designs of the cor- responding modules and measured their arithmetic density using hardware synthesis tools. Each de- sign contains versions for Int8, Float32, Mini- Float and all the block arithmetic types above. All these designs are functionally verified in AMD Xilinx simulator on of test vectors. The hardware arithmetic density is obtained us- ing the same formulation by Darvish Rouhani er al. (Darvish Rouhani ct al., 2020) with area FPGAs. The area results were obtained from the Plat- & Synthesis report in AMD Xilinx Vi- vado (AMD Xilinx Vivado, 2023). We estimate the total circuit area in LUTs, and DSP is considered to be equivalent to 100 LUTs. We used the Ultra- Scalc+ family of FPGA devices for experiments, and the version of AMD Xilinx software is 2020.2. The arithmetic densities of various quantisation methods are present in Table 6.
We additionally saturate MiniFloat when
2 1, thus the value of a MiniFloat
221 x
(-1) x x (1+
0 2
(2)
NaN
others
DMF
The definition of DMF is the same as Mini-
Float except that there no implicit leading bit in the mantissa:
(-1)"x2-
25
(3)
NaN
others
BM (Fox et al., 2021) shares the exponent bias and was proposed in the context of Quantisaion-AAare--eaee
BM, BI., and BFP
E PTQ on downstream tasks
(QAT). When an FP32 value is cast BM, the exponent bias is determined by the maximum value the block.
We quantised the pre-trained model and apply it the downstream tasks in the zero-shot prompting setup. Figure depicts how the performance of quantised models scale with model sizes. Our 6- bit BFP align with FP32 at various model sizes.
BFP (Darvish Rouhani et al., 2020) shares the exponent and was proposed in the context of PTQ.

---

Method
Config
Block size
#DSPs
#LUTs
Area factor
Arithmetic density
FP32
5
335
835
x
Integer
W8A8
1
9
109
7.7 x
MiniFloat
W8A8
0
48
48
17.4x
BM
W8A8
0
27
51
16.4x
BFP
W8A8
16
0
544
58
14.4x
BL
W8A8
28
52
16.1x
BFP
W6A6
16
313
43.6
19.2x
BFP
W4A4
16
0
358
22.4
37.3x
Table The arithmetic density of various quantisation configurations explored this paper. calculate the area factor, convert the Digital Signal Processing units (DSPs) to equivalent LUTs to get the area factor, and then divide the quantisation arithmetic's s area factor density by FP32.
FP32
LLM. A.M.(n LLM. int4(
G Searched mixed-precision LLMs
SmoothQuant-c
6-bit BFP
BFP
Mixed-precision quantisation is also helpful for re- covering downstream task accuracy. Figure 7a and Figure 7b depict the performance of 4-bit LLMs on LAMBADA and ARC (casy) as the model scales up. The searched mixed-precision configuration effectively recovers the accuracy. Figure 8 and 9 is the activation distribution after searching LAM- BADA 2688 times. Keeping these layers in high precision effectively recovers the accuracy from 36.2% to 61.3% without decreasing the memory density, equivalent to a 4.3-bit OPT-2.7B.
75
70
Mean accuracy (%)
65
60
55
50
125M
350M
1.3B 2.7B Model size
6.7B
H Mixed-precision with hardware model
We additionally performed a hardware-aware quan- tization search on BERT-Base (Devlin al., 2018). We implemented the actual BFP hardware on FPGAs via high-level synthesis and model the hardware cost using Token per second (TPS) for speedup and TPS per LUT (TPS/LUT) for cir- cuit area efficiency. Our hardware-aware search takes accuracy, memory bitwidth, and hardware cost as feedback. We compare the search traces of hardware-aware search and software-only search in Figure 10, and observe that the hardware-awan search curve higher than the original one, prov- ing our new objective function guides the search for better hardware efficiency. We will scale up experiments in future work.
Figure Mean accuracy (t. %) of various quanti- sation methods on downstream tasks. We observe that 6-bit BFP align with FP32 as model size scales, above SmoothQuant-c and LLM. int4(), below FP32 and LLM. int8(). Note that 6-bit BFP (-+-) achieves the highest memory density and arithmetic density among these methods. 4-bit BFP (-+-) has severe accuracy drop.
Table 7 presents the detailed accuracy of each task. Note that QNLI, MRPC, and COLA results are not included in this table because even FP32 LLMs yield an accuracy close to random prediction.
F PTQ on fine-tuned FP32 vs TAQ on downstream tasks
Table 8 compares the two options on four down- stream tasks (QNLI, SST2, COLA. MRPC), that FP32 LLM cannot handle. We observe that both align 4-bit BFP LLMs' performance with FP32 on downstream tasks.

---

Method
Contig
Model size
ARC (easy)
COPA
LAMBADA
PIQA
QNLI
SST2
Average
125M
43.5%
66.0%
37.9%
63.0%
49.4%
53.3%
52.7%
350M
44.0%
72.0%
45.2%
64.4%
49.5%
61.8%
57.5%
FP32
1.3B
57.0%
79.0%
57.9%
71.7%
51.3%
82.2%
69.6%
2.7B
60.8%
77.0%
63.6%
73.9%
51.1%
51.7%
65,4%
6.7B
65.6%
81.0%
67.7%
76.3%
50.9%
76.5%
73.4%
125M
43.6%
66.0%
37.7%
63.0%
49.5%
52. 1%
52.5%
350M
43.8%
72.0%
45.3%
64.2%
49.5%
66. 1%
58.3%
LLM. AMM.n
W8A8
1.3B
57.5%
79.0%
57.7%
71.6%
51.1%
80.1% 1%
69.2%
2.7B
60.6%
78.0%
62.9%
72.9%
51.2%
52. 1%
65.3%
6.7B
65.5%
83.0%
66.6%
76.1%
50.7%
76.4%
73.5%
125M
41.5%
65.0%
34.3%
62.1%
49.5%
51.1%
50.8%
350M
41.6%
68 0%
44.6%
64.0%
49.5%
60.8%
55.8%
int4)
W8A8
1.3B
55.9%
78.0%
54.5%
70.0%
51.7%
76.6%
67.0%
2.7B
58.7%
77.0%
62.2%
73.7%
51.3%
50.8%
64.5%
6.7B
64.5%
81.0%
66.2%
74.7%
51.6%
76.4%
72.5%
125M
350M
SmoothQuant-c
W8A8
1.3B
55.8%
78.0%
55.3%
71.2%
51.1%
75.9%
67.2%
2.7B
60.4%
77.0%
64.0%
72.6%
51.3%
51.7%
65.2%
6.7B
65.3%
81.0%
68.5%
74.7%
51.1%
71.6%
72.2%
125M
42.6%
69.0%
37.1% %
62.6%
49.4%
51.6%
52.6%
350M
43.7%
72.0%
42.8%
65.1%
49.6%
64.6%
57.6%
BFP
W6A6
1.3B
57.0%
79.0%
51.8%
71.6%
51.5%
79.7%
67.8%
2.7B
60.9%
76.0%
64.1%
73.3%
49.9%
53.1%
65.5%
6.7B
65.2%
80.0%
67.2%
75.8%
50.9%
76.
72.9%
125M
37.0%
65.0%
28.7%
58.9%
49.1%
49.5%
47.8%
350M
39.9%
65.0%
38.7%
58.9%
49.3%
55.9%
51.7%
BFP
W4A4
1.3B
50.0%
71.0%
41.4%
65.7%
50.2%
58.1%
57.2%
2.7B
52.4%
70.0%
36.2%
68.0%
51.4%
51.7%
55.7%
6.7B
61.5%
84.0%
56.0%
73.1%
52.3%
61.1%
67.2%
125M
42.9%
66.0%
38.3%
62.7%
49.6%
50.8%
52.1%
350M
44.0%
69.0%.
44.9%
64.2%
49.8%
53.3%
55.1%
MiniFloat
W4A4
1.3B
57.2%
80.0%
54.6%
71.4%
51.5%
60.2%
64.7%
2.7B
59,9%
75.0%
63.4%
73.7%
49.8%
56.7%
65.7%
6.7B
64.9%
82.0%
67.3%
76.0%
51.8%
62.2%
70.5%
Table 7: A complete comparison LLM quantisation methods on downstream tasks in the zero-shot prompting setup. QNLI, MRPC, and COLA results not included because FP32 L.LMs yield accuracy close random prediction.

---

Task
Fine-tuning style
Config
Model size
zero-shot prompting
epoch
cpoch
epoch 2
W32A32
125m
53.3%
91.2%
92.9%
92.6%
W32A32
350m
61.8%
92.3%
93.1%
93.4%
FP32
W32A32
1.3b
82.2%
93.9%
93.2%
94.0%
W32A32
2.7b
51.7%
94.6%
94.5%
94.7%
WSAS
125m
49.5% (-3.8%)
91.3% (0.1
91.7% (-1.2%)
92.0%
(-0.6%)
W5AS
350m
55,9% (-5.9%)
92.7% (0.4%)
92.5% (-0.6%)
92.2% (-1.2%)
SST2
PTQ downstream
WSAS
1.3b
58.1% (-24, 1%))
93.7% (-0.2%)
93.2% (0.0%)
93.6% (-0.4%)
W5A5
2.7b
51.7% (0.0%)
94.0% (-0.6%)
95.3% (0.8%)
94.5% (-0.2%)
WSAS
125m
49.5% (-3.8%)
92.0% (0.8%)
92.0%% (-0.9%)
91.6% (-0.9%)
WSAS
350m
55.9% (-6. (-6.0%)
91.1% (-1.3%)
91 6% (-1.5%)
91.6% (-1.7%)
TAQ on downstream
WSAS
1.3h
58.1% (-24.1% 11%)
94.3% (0.3%)
94.2% (0.9%)
94.2% (0.1%)
WSAS
2.7b
51.7% (0.0%)
(0.0%)
95. (0.4%)
94.7% (0.0%)
W32A32
125m
49 4%
87.8%
88.3%
88.7%
W32A32
350m
45.9%
86.5%
88.5%
89.1%
FP32
W32A32
1.3b
51.3%
89.0%
90.6%
91.7%
W32A32
2.7b
51.1%
61.2%
73.8%
85.3%
WSAS
125m
10.1 (-0.3%
82.0%
(-5.8%)
80.8% (-7.5%
85.7%
(-2.0%)
WSAS
350m
49.3% (-0.2%)
85.5% (-0.9%)
87.3% (-1.2%)
88.2% (-0.9%)
QNLI
PTQ downstream
W5AS
1.3h
50.2% (-1.1%%
88.6% (-0.5%)
89. 1% (-1.5%%
90.3% (1.4%)
WSAS
2.7b
51.4% (0.2%)
60.9% (-0.3%%
71.6% (2.2%)
84.2% (-1.1%)
WSAS
125m
49.1% (-0.3%%
86. 1% (-1.7%)
87.4% (-0.9%
88.2% (-0.5%)
WSAS
350m
49.3% (-0.2%)
85.5% (-1.0%)
88.3% (-0.1 %%
88.6% (-0,5%)
TAQ on downstream
W5AS
1.3b
50.2% (-1.1%)
86. 6% (-2.5%)
89.5% (-1.1%)
91.1% (0.7%)
WSAS
2.7b
51.4% (0.2%)
86.2% (25.0%)
88.1% (14.3%)
92.5% (4.3%)
W32A32
125m
0.0%
41.2%
47.3%
49.8%
W32A32
350m
0.0%
13.0%
39.8%
47.2%
FP32
W32A32
1.3b
-6.9%
29.1%
56.3%
56.9%
W32A32
2.7b
-3.5%
0.0%
8.0%
25.9%
W5AS
125m
-1.1% (-1.1%)
37.9%
(-3.3%)
40.4%
(-6.9%)
49.3% (-0.5%)
WSAS
350m
0.0% (0.0%%)
7.5% (-5.5%)
32.8% (-7.0%)
46 (- 11.2%)
COLA*
PTQ on downstream
W5AS
1.3b
-1.6% (5.3%)
21.0% (-8.1 1%))
51.9% (-4.4%)
55. (-1.1%)
W5AS
2.7h
-3.1% (0.0%%
0.0% (0.0%)
-2.1% (-10.1%)
26.0% (0.15))
W5AS
125m
(-1.1%)
43.1% (1.9%)
41.1% (0.7%)
43.7% (-5.6%)
WSAS
350m
0.0% (0.0%
12.7% (-0.3%
31.5% (-1.3%)
42. (-3.9%)
TAQ on downstream
W5AS
1.3b
-1.6% (5.3%)
33.9% (4. 8%)
59.0% (-2,9%%)
54. 6% (-1.2% (-1.2%)
WSAS
2.7h
-3.1% (0.4%)
0.7% (0.7%)
0.0% (2.1%)
18.6% (-7.4%)
W32A32
125m
68.4%
70.8%
76.8%
79.7%
W32A32
350m
68 68%
68.4%
69.6%%
70.6%
FP32
W32A32
1.3b
66.4%
69.6%
70.8%
66.7%
W32A32
2,7h
67.9%
70.1%
81.6%
82.6%
W5AS
125m
68.4% (0.0%)
71.3% (0.5%)
74.0% (-2.8%)
78.8% (-0.95))
WSAS
350m
68.4% (0.0%
68.4% (0.0%)
69.6 (0.0%)
69.4% (- (1.2%))
MRPC
PTQ on downstream
WSAS
1.3b
57.8% (-8.6%)
69.1 69.1% (-0.5%)
70. 1% (-0.7%)
67.6% (0.9%)
WSAS
2.7h
60.3% (-7.6%%
68.6%% (-1.5%
79.9% (-1.7%)
78.9% (-3.7%)
WSAS
125m
68.4% (0.0%)
70.3% (-0.5%)
71.6% (-5.2%%
77.8% (-1.99))
W5AS
350m
68.4% (0. (0.0%)
69 4% (1.0%)
69.7% (0.1%)
70.6% (0.0%)
TAQ on downstream
WSAS
1.3b
57.8% (-8.6%
68. 68.9% (-0.7%)
71.1% (0.3%)
71.1% (4.4%)
W5A5
2.7b
60.3% (-7.6%)
68.4% (-1.7%)
68.45 (-13.2%%
81 44 (-1.2%)
Table 8: The comparison between PTQ fine-tuned FP32 and TAQ on SST2, QNLI, COLA, and MRPC. Both cases align 4-bit BFP LLMs with FP32 after fine-tuning. The latter may achieve slightly better accuracy. 1 means COLA is evaluated using the Matthews Correlation Coefficient (MCC), while the other tasks are evaluated using accuracy.

---

Percentage of occurrence
4-bit
5-bit
6-bit
7-bit
5
10
15
20
25
30
Layer ID
(a) X in Line Algorithm 2.
FP32 *- 4-bit uniform
4-bit mixed
Percentage occurrence
S 4-bit 5-bit
6-bit
7-bit
Mean accuracy (%
65
0.5
55
45
10
15
20
25
30
35
Layer ID
(h) X, in Line Algorithm
25
125M
350M
1.3B
2.7B
6.7B
Percentage of oecurrence
< 4-bit
5-bit
6-bit
7-bit
Model size
(a) LAMBADA
FP32
4-bit uniform
4-bit mixed
0.5
Mean accuracy (%))
60
15
Layer ID
50
(c) X in Line 5, Algorithm 2.
Percentage of occurrence
S 4-bit
5-bit 6-bit
7-bit
40
125M
350M
1.3B
2.7B
6.7B
0.5
Model size
(b) ARC (easy)
10
15
20
30
Figure 7: The accuracy of FP32 model, 4-bit uni- form BFP, and 4-hit mixed-preeision on LAMBADA and ARCH (easy) in the zero-shot prompting context. Searched mixed-precisionn quantisation configuration captures the distribution inherent LLMs, effectively recovering the accuracy.
Layer ID
(d) K; in Line 6, Algorithm 2.
Percentage of occurrence
4-bit
5-bit
6-bit
7-bit
5
10
15
20
25
30
Layer ID
(e) A Line 8, Algorithm 2.
Figure The searched bit-width distribution of OPT- 2.7B. Notably, some layers are frequently assigned rel- atively high precision, indicating these layers are less tolerant to quantisation.

---

Pareentage of necurrence
4-bit 5-bit
6-bit
7-bit
0.5
0
5
10
15
20
25
30
Layer ID
(a) in Line Algorithm
Percentage of necurrence
I 4-bit
5-bit
6-bit
7-bit
5.49
0.5
25
5.36
hardware-aware
Layer ID
software-only
(b) Be Line 11, Algorithm 2.
Percentage of necurrence
4-bit
5-bit
6-bit
7-bit
20
100
500 1,000
3,600
Time (s)
Figure comparison between the new hardware- aware search algorithm and the previous one that only considers accuracy and memory density. We introduce additional hardware metries, Token Per Second (TPS) and TPS Per LUT (TPS/LUT), such that the search algorithm can be aware of the hardware effi- ciency. The new objective function O, mem - 02 tps + 0g fpl, where tps and tpl denote TPS and TPS/LUT respectively. We plot the traces of both searches versus search time for BERT-base. We observe that the hardwane-aware curve is higher than the original one, proving our new objective function guides the search for better hardware efficiency in terms of speedup and circuit area.
0.5
0
5
10
15
20
25
30
Layer ID
dq
(c) Ba in Line 13, Algorithm 2.
Percentage necurrence
4-bit
5-bit
6-bit
7-bit
0.5
0
5
10
15
20
25
30
Layer
(d) B1 in Line 14, Algorithm 2.
Figure 9: The searched bit-width distribution of OPT- 2.7B. Notably, some layers frequently assigned rel- atively high precision, indicating these layers are less tolerant quantisation.


Reviews:
[{"review_id": "s1li0VqEZ2", "reviewer": "Reviewer_JYyW", "summary": "", "questions": "* How should we think about the results applying to models other than OPT? Would they likely hold? What would be some of the factors that could enable us to see the same/different results?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for the valuable feedback.\n\n> Q1: The authors select OPT to have a variety of model sizes across their experiments. If I have understood correctly the OPT models are the only models used in experiments. I wonder how much we would expect the results to hold across other pretrained models?\n\nFollowing your suggestion, we extended our methods to other LLMs. The new results on other LLMs further support the claims in our paper. \n\nThe table below shows the comparison of our 6-bit uniformly quantized BFP LLMs with the baselines. We now include LLAMA, Vicuna, and Alpaca. We observe that 6-bit BFP archives better hardware efficiency under the same perplexity constraint. We will add these new results to Table 2 in the revised version.\n\n| Model           | FP32 |  LLM.int8()  | Ours 6-bit BFP |\n|-----------------|:----:|:------------:|---------------:|\n| LLAMA-7B        | 5.79 | 5.83 (+0.04) |   5.83 (+0.04) |\n| Vicuna-7B       | 7.06 | **7.07 (+0.01)** |   7.08 (+0.02) |\n| Alpaca-7B       | 7.01 | 7.02 (+0.01) |   7.02 (+0.01) |\n| LLAMA-13B       | 5.17 | 5.22 (+0.05) |   **5.20 (+0.03)** |\n| Vicuna-v1.5-13B | 6.13 | 6.16 (+0.03) |   6.16 (+0.03) |\n\nRebuttal Tab. 1: The perplexity ($\\downarrow$) of a range of LLMs other than OPT quantized by our 6-bit BFP method. We compared our method with FP32 and LLM.int8() and found that our method achieves nearly lossless perplexity on Wikitext2.\n\n---\n\n> Q2: How should we think about the results applying to models other than OPT? Would they likely hold? What would be some of the factors that could enable us to see the same/different results?\n\nOur answer is that our conclusion still holds for other LLMs. The intuitive explanation is that all these models share similar network architectures (Transformer) and are trained on language modeling tasks. All of these LLMs would have the numerical scaling offsets issue described in the introduction section. \n\nTo verify our hypothesis, we performed two groups of experiments. \n- First, we plotted the tensor variances of LLMs other than OPT, and observed the same increasing trends as Figure 1 in the paper. Similar to OPT-6.7B, Vicuna-7B’s activation variance also increases with the layer number and is consistently larger than the weight variance.\n\n  [Rebuttal Fig.1: variance curves of Vicuna-7B](https://imgur.com/LOo5m6X)\n\n- Second, we quantized newer and larger LLMs out of OPT family and compared their perplexity with baselines. Here is the result table (the same one in the answer of Q1).\n\n  | Model           | FP32 |  LLM.int8()  | Ours 6-bit BFP |\n  |-----------------|:----:|:------------:|---------------:|\n  | LLAMA-7B        | 5.79 | 5.83 (+0.04) |   5.83 (+0.04) |\n  | Vicuna-7B       | 7.06 | **7.07 (+0.01)** |   7.08 (+0.02) |\n  | Alpaca-7B       | 7.01 | 7.02 (+0.01) |   7.02 (+0.01) |\n  | LLAMA-13B       | 5.17 | 5.22 (+0.05) |   **5.20 (+0.03)** |\n  | Vicuna-v1.5-13B | 6.13 | 6.16 (+0.03) |   6.16 (+0.03) |\n\n  Rebuttal Tab. 1: The perplexity ($\\downarrow$) of a range of LLMs other than OPT quantized by our 6-bit BFP method. We compared our method with FP32 and LLM.int8() and found that our method achieves nearly lossless perplexity on Wikitext2.\n\nWe will add the results of other LLMs to the appendix of the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a seemingly comprehensive study of quantization methods for LLMs with a focus on 8-bits or less. These results include a comparison to both:\n* baseline methods such as LLM.int8(), SmoothQuant, GPTQ\n* block based methods: BM, BFP, BL, etc.\n\nThe results compare performance on a variety of task settings: WikiText2, LAMBADA, QNLI, lm-eval-harness, and many others. The proposed approaches in this paper achieve some of the best results, e.g., Table 3. This includes the fine-tuning setting (both PTQ/TAQ). \n\nThe paper presents the approaches in a thoughtful way with experiments that provide a detailed landscape of quantization in LLMs. This could be a benefit to readers understanding the design space and research landscape. ", "reasons_to_reject": "Concerns about the paper include:\n* The authors select OPT to have a variety of model sizes across their experiments. If I have understood correctly the OPT models are the only models used in experiments. I wonder how much we would expect the results to hold across other pretrained models? ", "questions_to_authors": "* How should we think about the results applying to models other than OPT? Would they likely hold? What would be some of the factors that could enable us to see the same/different results?\n", "paper_topic_and_main_contributions": "This paper investigates quantization methods for large language models. Through analysis of activations, the authors indicate that variances tend to increase in deeper layers of the network, yielding scaling offsets in their quantization. The work studies a variety of block arithmetic based methods. The paper contributes a wealth of empirical analysis encompassing multiple tasks / settings as well as model sizes. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The highlighting in Table 4 might be more effective by defining what close means and then highlighting all methods that meet that definition."}, "extra_text": {}}, {"review_id": "0ZkunWezbO", "reviewer": "Reviewer_Qm4i", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: What are the tradeoffs of using FPGAs and GPUs?\n\nThank you for your questions. In order to provide a comprehensive comparison of the tradeoffs, it is better if we look at between GPU, ASIC (Application-Specific Integrated Circuits), and FPGAs (Field Programmable Gate Arrays) for accelerating custom non-linear arithmetics.\n\nIt is worth noting that there is a growing trend towards utilizing domain-specific hardware for accelerating ML workloads, exemplified by Google's TPU (v1-v5) for AI acceleration (ASIC accelerator), Microsoft BrainWave project (FPGAs) as compared to NVIDIA's GPGPU. In this regard, we hold the belief that custom hardware optimized for efficiency and cost will be the future generation hardware for LLM inference. FPGAs, however, serve as an intermediary between GPUs and ASICs. Due to their arithmetic density and memory density, we consider FPGAs to be a viable approximation of ASICs in our analysis.\n\nNVIDIA’s GPU is the most common choice for training and testing neural networks in the deep learning community because deep learning frameworks like PyTorch have high-performance tensor computation libraries built on CUDA e.g., Aten for PyTorch. However, when it comes to deployment/inference for production (no training is performed), ASICs for neural networks, also known as AI accelerators, are a better choice than GPUs in terms of efficiency (speed, energy, etc.). Another limitation of GPUs is the lack of native support for custom quantization arithmetics like the int4, BFP, BL, and BM mentioned in this paper. For example, NVIDIA RTX 4090 only supports FP64, FP32, Bfloat16, FP16, and Int8, and PyTorch supports training & testing with FP64, FP32, Bfloat16, and FP16. Though sometimes it is possible to support custom quantization arithmetics via manually-designed CUDA kernel, the efficiency is still much lower than ASICs.\n\nFPGAs are reconfigurable hardware that can be programmed to meet the developer’s requirements. The programmability of FPGAs makes it a good choice for implementing computation units for custom quantization arithmetics. There have been considerable publications on accelerating neural networks on FPGAs. Another advantage of FPGAs is that the development life circle is much shorter than ASICs, thus FPGAs are suitable for research. Additionally, FPGA implementations are typically perceived as a valuable testbed for ASIC designs. The higher performance numbers achieved through FPGAs directly correlate with the potential performance of corresponding ASIC designs, which can take advantage of the faster clock speed.\n\nIn conclusion, GPUs are a good choice for training but FPGAs have higher efficiency in terms of accelerating the inference of custom non-linear quantization arithmetics like BFP.  Moreover, the programmability of FPGAs enables us to report the arithmetic density of the custom BFP computation units via logic synthesis. It will be hard to model the memory density and arithmetic density on GPUs even if we have CUDA kernels implemented.\n\n---\n\n> Q2: Ablations with and without quantization search. How far can you go with the non-linear quantization methods BM, BFP, BL without quantization search? How important is it? Can it be used with other quantization methods?\n\nWe have included a series of ablation studies in the paper but maybe didn’t clearly point out where they are. We will clarify our observation (with/without search) in the revised version. Here we split your question into two parts and address them one by one.\n\n> > Q2.1: Ablations with and without quantization search. How far can you go with the non-linear quantization methods BM, BFP, BL without quantization search? How important is it?\n\nSection 4.2 compares uniform quantization arithmetics (without a quantization search), and Section 4.3 is the only section that performs a quantization search. Here are the ablation studies we conducted.\n\n- For BFP, 6-bit is the minimum bitwidth we could achieve while maintaining comparable accuracy/perplexity to baselines.\n  - Line 510 is our observation that “By preserving high precision for these sensitive layers, we recovered the 4-bit LLM accuracy from 36.2% to 61.3% on LAMBADA without compromising memory density”. Here the zero-shot prompting accuracy of 36.2% is the one without quantization search and 61.3% is the one using quantization search. Figure 8 in the appendix is a more comprehensive ablation study of mixed-precision search including a range of model sizes on the two datasets. In Figure 8 the curve “4-bit uniform” is 4-bit BFP quantization without search and the curve “4-bit mixed” means that the searched mixed-precision LLM has an average bitwidth of 4. This figure implies that quantization search is an effective way to recover the accuracy for 4-bit LLMs.\n\n    [shortcut to Figure 8 in the paper](https://imgur.com/ygKtncK)\n\n  - Table 3 shows the uniform 6-bit BFP and 4-bit BFP without quantization search on wikitext2. We also have results of 5-bit BFP but we did not include it in Table 3 due to the page limit. We instead drew a conclusion that “We also observe that sub-6-bit BFP has a severe accuracy drop” in line 442.\n\n    [Shortcut to Table 3 in the paper](https://i.imgur.com/ERrgibE.png)\n  - Here is a table of the perplexity comparison of BFP bitwidth. We can see that 5-bit and 4-bit BFP have increased perplexity with an average of around 1 and 10 respectively compared to FP32, which is worse than existing state-of-the-art quantization schemes like LLM.int8() and SmoothQuant. We will add the “BFP W5A5” row if the paper is accepted since the camera-ready version allows one additional page.\n\n    |          |      OPT-125M      |      OPT-350M      |      OPT-1.3B      |      OPT-2.7B     |   OPT-6.7B |\n    |----------|:--------------:|:--------------:|:--------------:|:-------------:|--------------:|\n    | FP32     |      27.65     |      22.00     |      14.62     |     12.47     |         10.86 |\n    | BFP W6A6 |  28.27 (+0.62) |  22.22 (+0.22) |  15.08 (+0.46) | 12.54 (+0.07) | 10.90 (+0.04) |\n    | BFP W5A5 |  29.91 (+2.26) |  23.77 (+1.77) |  15.96 (+1.34) | 13.01 (+0.54) | 11.23 (+0.37) |\n    | BFP W4A4 | 41.94 (+14.29) | 33.98 (+11.98) | 24.70 (+10.07) | 19.34 (+6.87) | 13.59 (+2.73) |\n\n    Rebuttal Tab. 2: A perplexity ($\\downarrow$) comparison of 4-bit, 5-bit, and 6-bit BFP on Wikitext2. We observe that 6-bit BFP is the lowest bitwidth we can achieve without mixed-precision quantization search or fine-tuning.\n  - Table 4 shows the uniform 6-bit, 5-bit, and 4-bit BFP without quantization search on five downstream tasks (Table 4 reports mean zeros-shot prompting accuracy, and Table 6 in the appendix reports complete results). In Table 4, we also observe that 6-bit BFP is the best we can do without compromising accuracy. 5-bit BFP OPT-1.3B has a mean accuracy drop of 4.1%.\n    \n    [shortcut to Table 4 in the paper](https://i.imgur.com/AUt5DJc.png)\n\n- For BM and BL, as we observed in Table 3, even 8-bit BL or 8-bit BM has a perplexity that is much higher than existing baselines such as LLM.int8() and SmoothQuant (8-bit BL and BM’s perplexity is increased by ~20x and ~800x respectively), mixed-precision search for 8-bit BL or BM will not find promising results. We drew this conclusion in line 430 of the paper and excluded BL and BM from the mixed-precision search and ablation study.\n\n>> Q2.2: Can it be used with other quantization methods?\n\nYes, basically the search objective in line 312 can be extended to support other quantization methods if software modeling or hardware modeling is provided. Here we provide an example of extending our current search method.\n\nExample: mixed-precision fixed-point quantization search. \n\nWe add the memory density model of fixed-point quantization to our search algorithm, and search for mixed-precision fixed-point quantized LLMs (LLAMA-7B, Vicuan-7B, Alpaca-7B, and OPT-6.7B) on SST2. Here are our results,\n\n| Models    | Fixed-point W8A8 | *Mixed-fixed-point* |   BFP W8A8  |   Mixed-BFP   |         FP32 |\n|-----------|:----------------:|:-----------------:|:-----------:|:-------------:|-------------:|\n| LLAMA-7B  |    (53.33, 8)   |  *(62.50, 4.7)* | (83.14, 8) | (84.38, 4.5) | (83.26, 32) |\n| Vicuna-7B |    (53.33, 8)   | *(58.59, 5.1)* | (8028, 8) | (82.91, 4.4) | (83.94, 32) |\n| Alpaca-7B |    (55.85, 8)   | *(58.59, 5.1)* | (81.65, 8) | (89.06, 4.5) | (84.98, 32) |\n| OPT-6.7B  |    (49.08, 8)   | *(53.91, 4.7)* | (76.15, 8) | (72.94, 4.4) | (76.49, 32) |\n\n Rebuttal Tab. 3: The (zero-shot prompting accuracy (%), average bitwidth) of mixed-precision fixed-point quantization search. We extended our search method and searched for mixed-precision strategies of fixed-point quantized LLMs on SST2. We observed that the searched fixed-point LLMs achieved higher accuracy than uniform 8-bit fixed-point, but had lower average bitwidth.\n\n---\n\n> Q3: The finetuning discussion appears to be out of place in the paper and it does not add to the general argument of the paper.\n\nSorry for any confusion. Respectfully, we believe that finetuning plays a significant role in our contribution. In this paper, we present the following arguments: \n- BFP enables uniform 6-bit post-training quantization on LLMs, and significantly outperform existing methods in terms of accuracy and hardware efficiency.\n- If we target sub-6-bit LLMs, there are two possible ways,\n  - one is fine-tuning a uniform quantized LLM (no mixed-precision search), which is also commonly known as Quantization Aware Training (QAT).\n  - the other is searching for a mixed-precision LLM in the context of post-training quantization (no fine-tuning).\n\n  We demonstrated both approaches allow 4-bit LLMs.\n\nWe know LLMs pre-trained on language modeling tasks may fail on some downstream tasks such as QNLI and COLA, thus fine-tuning is necessary. This then stems the question: if we apply quantization, should we (1) quantize the pre-trained LLM first, then do quantization-aware training on downstream tasks, or (2)  fine-tune the pretrained FP32 LLM first, then do post-training-quantization? We compare both options on a range of downstream tasks that zero-shot prompting cannot handle (fine-tuning is necessary even for FP32 pre-trained LLMs) in the fine-tuning subsection (Section 4.3) and drew the following conclusion in line 469,\n\n- Both options can achieve nearly lossless 4-bit LLMs in terms of accuracy on challenging downstream tasks.\nThe fine-tuning of (1) is easier to converge due to the Straight-Through Estimator (STE) in (2)’s backward pass.\n\n- Moreover, prior work like LLM.int8(), SmoothQuant, and GPTQ only consider PTQ of LLM, but our work **considers both PTQ and QAT (PTQ for uniform 6-bit and 4-bit mixed-precision search, QAT for fine-tuning 4-bit)**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Insights in quantization methods which lead to improved arithmetic and memory density with little perplexity degradation.\n- Interesting demonstration of use of FPGAs to improve the efficiency of LLM inference though quantization.", "reasons_to_reject": "Because of the quantity of information presented, the paper comes across as difficult to follow and scattered and the value of the insights is greatly diminished. Some of the elements that would benefit more clarification and further ablations to justify the design choices:\n- What are the tradeoffs of using FPGAs and GPUs?\n- Ablations with and without quantization search. How far can you go with the non-linear quantization methods BM, BFP, BL without quantization search? How important is it? Can it be used with other quantization methods?\n- The finetuning discussion appears to be out of place in the paper and it does not add to the general argument of the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores non-linear quantization methods applied to LLM inference (up to 6.7B). It provides insights in LLM quantization and analyzes arithmetic and memory density of various quantization methods. \n\nInstead of scaling blocks by normalizing to a predefined interval which introduces quantization constants, this paper suggests a combination of approaches:\n1. non-linear quantization using Block Mini-Float, Block Floating Point, and Block Logarithm methods. These methods share an exponent or exponent bias for elements of a block. \n2. implement these methods on FPGAs.\n3. use quantization search to optimize the quantization configuration for each layer of the transformer.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1HRlyhRH2x", "reviewer": "Reviewer_ra67", "summary": "", "questions": "The authors should address the weaknesses mentioned in the above section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the reviewer's helpful suggestions and questions. Here we answer the questions one by one.\n\n> Q1: The authors highlight that they have not implemented the quantisation methods on GPU systems to demonstrate real speedups due to a lack of CUDA kernel implementation.\n\nIn this paper, we try to highlight that block-based quantization arithmetic has the potential to be the next-generation AI accelerator for LLM inference (domain-specific hardware) rather than GPGPUs. After we submitted the draft, we finished a series of experiments of mixed-precision BFP LLMs on FPGA-based AI accelerators where BFP computation units were implemented. Our FPGA-based accelerator outperforms GPUs in terms of hardware efficiency without sacrificing accuracy. Here are our results,\n\n|                       |   Models  | 0-shot Accuracy (%) | Mean bitwidth | SPS ($\\uparrow$) | SPS/W ($\\uparrow$) |\n|-----------------------|:---------:|:-------------------:|:-------------:|:----------------:|-------------------:|\n| GPU (RTX 3090)        |  OPT-2.7B |        51.27        |       32      |        69        |          **0.051** |\n|                       |  OPT-6.7B |        76.49        |       32      |         6        |              0.008 |\n|                       | Alpaca-7B |        84.98        |       32      |         7        |              0.008 |\n| Our mixed-BFP on FPGA |  OPT-2.7B |        62.61        |      4.0      |      **85**      |              0.038 |\n|                       |  OPT-6.7B |        72.94        |      4.1      |      **33**      |          **0.012** |\n|                       | Alpaca-7B |        89.06        |      4.5      |      **25**      |          **0.011** |\n\nRebuttal Tab. 4: a comparison between FP32 LLMs on GPUs and mixed-precision BFP LLMs on FPGAs. We report zero-shot prompting accuracy on SST2. The last two columns are hardware metrics. We use Sequence Per Second (SPS) and SPS per Watt (SPS/W) to evaluate hardware efficiency. Sequence per second is the number of text sequences that our FPGA-based accelerator processes per second, which estimates the inference throughput. SPS per Watt measures the SPS our accelerator reaches per unit of power, which estimates the energy efficiency. Our FPGA-based accelerator reaches SPS and SPS/W higher than GPUs.\n\nWe use Sequence Per Second (SPS) and SPS per Watt (SPS/W) to evaluate hardware efficiency and report zero-shot prompting accuracy. Sequence per second is the number of text sequences that our FPGA-based accelerator processes per second, which estimates the inference throughput. SPS per Watt measures the SPS that our accelerator reaches per unit of power, which estimates the energy efficiency. Our SPS and SPS/W are usually higher than GPUs, proving that the mixed-BFP actually speeds up the LLM inference and improves power efficiency compared to GPUs.\n\n---\n\n> Q2: The paper also mentions that the search algorithm does not include arithmetic density due to a lack of hardware models.\n\nWe acknowledge that the lack of arithmetic density is our limitation. To solve this problem, we plan to introduce more accurate hardware metrics (Sequence Per Second (SPS) and SPS per LUT (SPS/LUT)) into our search algorithm. Please refer to the answer to Q3 below for a detailed explanation.\n\n---\n\n> Q3: Although the authors have mentioned the limitations in the paper, they should provide a more detailed plan on how they plan to address these drawbacks in their future work.\n\nWe will provide a more detailed plan for future work in the revised version. We plan to introduce two more accurate hardware metrics than arithmetic density for modeling the hardware cost.\n\nBasically, we can implement the actual BFP hardware on FPGAs via high-level synthesis and model the hardware cost (SPS for speedup and SPS/LUT for circuit area efficiency) using polynomial regression. In this way, our search algorithm can be aware of the hardware cost of quantization configurations. The improved search objective will become\n$$\nQ_f = acc + \\alpha_1 \\cdot mem + \\alpha_2 \\cdot sps + \\alpha_3 spl\n$$\nwhere $sps$ and $spl$ denote SPS and SPS/LUT.\n\nWe have verified the feasibility of this method on BERT-base and compared the new hardware-aware search algorithm with the previous one.\n\n[Rebuttal Fig2: Comparison between the search with and without SPS and SPS/LUT](https://i.imgur.com/Tl2dtYX.png)\n\nBy comparing the traces of the two search methods (with/without SPS and SPS/LUT), we observe that introducing hardware metrics indeed guides the search to find more hardware-efficient mixed-precision models. We leave scaling up experiments on LLMs (such as LLAMA) as future work.\n\n---\n\n> Q4: The evaluation is limited to language modeling and downstream NLP tasks. Testing the proposed quantization methods on other modalities like computer vision could benefit the paper.\n\nWe mainly target LLM quantization when drafting this paper as the observed increasing variance in LLMs (Figure 1) motivates our research. In our humble opinion, the quantization of vision models is out of scope of this paper but maybe a promising future direction.\n\n---\n\n> Q5: The authors should provide detailed ablation studies to isolate the impact of block-based quantization.\n\nThanks for your suggestions. We did not explicitly include an ablation study section, but the impact of block-based quantization can be observed in Table 3 by comparing block-based quantization with fixed-point quantization and MiniFloat.\n\n8-bit fixed-point quantization can be considered as an extreme case of 8-bit BFP where the block size equals the tensor size. MiniFloat or standard floating-point is another extreme case of BFP where the block size is 1. We can observe that the accuracy of quantized LLMs increases as the block size decreases (fixed-point -> BFP -> MiniFloat) but the hardware efficiency decreases as well. \n\nWe will add this analysis to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a new perspective on the problem of LLM quantisation by focusing on the issue of numerical scaling offsets.\n\n2. The authors adapt block-based quantisation methods for LLMs like Block Floating Point (BFP), demonstrating that these methods can efficiently reduce numerical scaling offsets without requiring additional treatments in the computational path.\n\n3. The paper shows nearly lossless 6-bit quantized LLMs using BFP, which achieve 19x higher arithmetic density and 5x higher memory density compared to the float32 baseline. This 6-bit BFP outperforms prior work like LLM.int8() and SmoothQuant by 2.5x in arithmetic density and 1.2x in memory density without requiring data calibration or re-training.\n\n3. The paper offers valuable insights into sub-8-bit LLM quantisation, including the impact of mismatches between activation and weight distributions, optimal fine-tuning strategies, and the implications of the lower quantisation granularity inherent in the statistical properties of LLMs.\n\n4. The authors present two methods to achieve nearly-lossless 4-bit quantisation on downstream tasks, including fine-tuning-based and mixed-precision search.\n\n5. The paper also explores that mixed-precision search has the potential advantage of shifting LLM inference to cost-effective ASICs.\n\n6. The authors conduct extensive experiments to validate their proposed approach, comparing their block-based quantisation technique with traditional methods on several language processing tasks. Their method consistently outperforms traditional methods across perplexity, accuracy, memory density, and arithmetic density.\n\n7. The paper is well-written and well-structured.", "reasons_to_reject": "1. The authors highlight that they have not implemented the quantisation methods on GPU systems to demonstrate real speedups due to a lack of CUDA kernel implementation.\n\n2. The paper also mentions that the search algorithm does not include arithmetic density due to a lack of hardware models.\n\n3. Although the authors have mentioned the limitations in the paper, they should provide a more detailed plan on how they plan to address these drawbacks in their future work.\n\n4. The evaluation is limited to language modeling and downstream NLP tasks. Testing the proposed quantisation methods on other modalities like computer vision could benefit the paper.\n\n5. The authors should provide detailed ablation studies to isolate the impact of block-based quantisation.", "questions_to_authors": "The authors should address the weaknesses mentioned in the above section.", "paper_topic_and_main_contributions": "The paper explores the problem of quantisation in Large Language Models (LLMs), which is used to reduce the computational and memory resources required for running inference on these models by representing the model parameters with a lower precision format. However, current quantisation techniques like LLM.int8() and SmoothQuant focus on 8-bit precision, and a further reduction in bits causes large accuracy drops. The authors attribute numerical scaling offsets as the main bottleneck to LLM quantisation. To address this problem, they propose adapting block-based quantisation methods for LLMs like Block Floating Point (BFP). These methods share scaling factors across packed numbers, thereby efficiently reducing numerical scaling offsets without necessitating additional treatments in the computational path.\n\nThe authors demonstrate that their block-based quantisation framework can achieve nearly lossless quantised 6-bit LLMs. These surpass prior work in terms of arithmetic and memory density without requiring any data calibration or re-training. Additionally, the paper provides insights into sub-8-bit LLM quantisation, including the impact of mismatches between activation and weight distributions, optimal fine-tuning strategies, and lower quantisation granularity inherent in the statistical properties of LLMs. The authors show that the latter two methods enable nearly lossless 4-bit LLMs on downstream tasks. Mixed-precision search further demonstrates the potential advantage of shifting LLM inference to cost-effective ASICs. The authors conduct experiments to validate their approach, comparing their block-based quantisation technique with traditional methods on several language processing tasks. Their method outperforms conventional methods in terms of perplexity, accuracy, memory density, and arithmetic density.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 4 in the Abstract, it should be \"emerged\" instead of \"merged\".\n\n2. \"MiniFloat\" is not written in a consistent format in some instances in the paper."}, "extra_text": {}}, {"review_id": "5yankpXUI8", "reviewer": "Reviewer_F1oA", "summary": "", "questions": "What exactly is the search space of hyperparameter optimization? In line 271-273, it seems like the granularity is a hyperparameter, and can chosen between layer-wise, tensor-wise, channel-wise, etc. Is this searched by HPO, or are they manually configured by authors? Does HPO algorithm choose the number of bits (4, 5, 6, 7, 8) for each of the block? That's my interpretation of \"search for quantisation precision\" (line 303) but I wasn't entirely sure, because the choice of precision across layers is only discussed in Mixed-precision experiments (Figure 3, line 499). Then, for the rest of experiments, do authors fix the number of bits, or do they always do the HPO search? Section 3.3 is written like HPO is always done.\n\nAlso, how does HPO iterations impact the performance of the model? Is it in general difficult to find a good configuration- do we consistently find a good configuration, or is there a high variance across runs?\n\nWhen we use mixed precision, shouldn't authors be reporting how much memory density was sacrificed? In line 513 authors mention memory density was not compromised, but it'd help to provide a number (1%? 0.1%? 0.01%?)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: I am not sure the proposed method has a comparative advantage over LLM.intX(). Table 3 doesn't have W8A8 version of BFP or LLM.int4(), so we cannot really do a head-to-head comparison with BFP with same bits. (Though LLM.int8 seems to only have 2x memory density, whereas other 8-bit precision methods in Table 3 have 4x-ish density; maybe I am terribly misunderstanding something.) In Table 4, 4-bit version of BFP W4A4 performs worse than LLM.int4(), and BFP W8A8 performs worse than LLM.int8(). 6-bit version BFP W6A6 is in between 4-bit and 8-bit counterparts, but if we are considering custom hardware chips, we could also consider LLM.int6() as a baseline?\n\nThanks for your questions. 8-bit/6-bit BFP outperforms LLM.int8() in terms of memory density and arithmetic density while maintaining comparable accuracy/perplexity to LLM.int8(). Here we answer the reviewer’s questions about LLM.int8()/int4() and provide a head-to-head comparison.\n\n- **Perplexity & accuracy**. We acknowledge that on OPT family from 125M to 6.7B, our 6-bit BFP’s performs slightly worse than LLM.int8() in terms of perplexity or accuracy. However, our 6-bit BFP has much higher hardware efficiency than LLM.int8() or LLM.int6() if it exists (See the detailed explanation in the next two paragraphs). Moreover, we further applied our experiments to newer and larger LLMs, and found our 6-bit BFP does not clearly perform worse than LLM.int8() in terms of perplexity. We assume this is because the numerical scaling offset described in Figure 1 is more obvious on larger models. Here are the perplexity results.\n\n  | Model           | FP32 |  LLM.int8()  | Ours 6-bit BFP |\n  |-----------------|:----:|:------------:|---------------:|\n  | LLAMA-7B        | 5.79 | 5.83 (+0.04) |   5.83 (+0.04) |\n  | Vicuna-7B       | 7.06 | **7.07 (+0.01)** |   7.08 (+0.02) |\n  | Alpaca-7B       | 7.01 | 7.02 (+0.01) |   7.02 (+0.01) |\n  | LLAMA-13B       | 5.17 | 5.22 (+0.05) |   **5.20 (+0.03)** |\n  | Vicuna-v1.5-13B | 6.13 | 6.16 (+0.03) |   6.16 (+0.03) |\n\n  Rebuttal Tab. 1: The perplexity ($\\downarrow$) of a range of LLMs other than OPT quantized by our 6-bit BFP method. We compared our method with FP32 and LLM.int8() and found that our method achieves nearly lossless perplexity on Wikitext2.\n\n  In the table above, our 6-bit uniformed quantized BFP achieves almost the same perplexity as LLM.int8().\n\n- **Memory density**. LLM.int8()/LLM.int4() is different from plain 8-bit/4-bit fixed-point quantization. In LLM.int8()/LLM.int4() all the tensors are stored as 16-bit floating-points, which is the reason why both LLM.int8() and LLM.int4() have 2x memory density while plain 8-bit fixed-point has 4x memory density.\n\n- **Arithmetic density**. LLM.intX() targets GPUs while ours target AI accelerators. If LLM.int6() is mapped to a custom hardware chip, the chip’s arithmetic density **will be lower than 6-bit BFP**. LLM.intX() filters out outliers (the matrix elements with large magnitudes) using a manual threshold, casts inliers to X-bit, and performs an X-bit matrix multiplication for inliers and an FP16 matrix multiplication for outliers respectively. This thresholding is performed on the fly, which means the GPU or the custom hardware chip does not have prior knowledge about the sizes of the inlier matrix or the outlier matrix. The input-dependent FP16 matrix multiplication in LLM.intX() leads to a much lower arithmetic density than plain X-bit fixed-point or X-bit BFP on custom chips because the chip’s FP16 matrix multiply engine must fit the worst case to match the accuracy reported in LLM.intX() paper, i.e., the engine must accommodate the maximum possible number of FP16 outliers. The arithmetic density also depends on the accelerator architecture. For example, if the LLM.int6() matrix multiply engine is time-multiplexed, we need to consider the max possible number of outliers for all matrix multiplications this engine will process. In comparison, 6-bit uniform BFP does not require an FP16 matrix multiply engine. All tensors are stored and calculated in 6-bit. \n\nWe will add the discussion of LLM.int8() to the revised version.\n\n---\n\n> Q2: I also found the description of hyperparameter search unclear, which makes it difficult to understand implications of experiments. I will leave them in questions for the authors.\nWhat exactly is the search space of hyperparameter optimization? In line 271-273, it seems like the granularity is a hyperparameter, and can chosen between layer-wise, tensor-wise, channel-wise, etc. Is this searched by HPO, or are they manually configured by authors? Does HPO algorithm choose the number of bits (4, 5, 6, 7, 8) for each of the block? That's my interpretation of \"search for quantisation precision\" (line 303) but I wasn't entirely sure, because the choice of precision across layers is only discussed in Mixed-precision experiments (Figure 3, line 499). Then, for the rest of experiments, do authors fix the number of bits, or do they always do the HPO search? Section 3.3 is written like HPO is always done.\n\n**The granularity of quantization search is fixed to per-tensor in this work (the granularity is not a searchable hyper-parameter)**. For example, OPT-125m consists of 12 transformer blocks, each block has 8 matrix multiplications, and each matrix multiplication has two tensor operands (ignore the bias for now). If the bitwidth (i.e., precision) choices include (4,5,6,7,8), then the search space will be 5 ^ (2 \\times 8 \\times 12), i.e., each tensor can have a different precision. The linear layers with bias have one more hyper-parameter to search. \n\n**Only Section 4.4 presents the results of the mixed-precision search.** Sections 4.2 and 4.3 are both uniform quantizations without mixed precision. Here are the arguments we tried to claim in the paper,\n\n- Section 4.2: BFP enables uniform 6-bit post-training quantization on LLMs, outperforming existing methods in terms of accuracy and hardware efficiency (no mixed-precision search)\n- If we target sub-6-bit LLMs, we demonstrate two possible ways,\n  - Section 4.3: one is fine-tuning a uniform quantized LLM (no mixed-precision search)\n  - Section 4.4: the other is searching for a mixed-precision LLM in the context of post-training quantization.\n\n  We showed that both approaches allow 4-bit LLMs.\n\n---\n\n> Q4: Also, how does HPO iterations impact the performance of the model? Is it in general difficult to find a good configuration- do we consistently find a good configuration, or is there a high variance across runs?\n\nWe find more iterations usually give better mixed-precision results in terms of both accuracy and memory density. Besides the OPT family in the paper, we also applied the mixed-precision search on BERT-large, LLAMA-7B, Alpaca-7B, and Vicuna-7B. We find that 128 iterations (n_trials=128) are enough for TPE to find promising results on all the models above.\n\nIn terms of the variance across runs, we calculated the mean and variance of 42 runs (each run has 128 iterations) of OPT-2.7B on LAMBADA (we used these runs to visualize the bitwidth distribution in Figure 3). Here is the variance and mean of accuracy and memory density across runs,\n\n|          | 0-shot Accuracy$^*$ ($\\uparrow$) | Memory density ($\\uparrow$) | Average bitwidth ($\\downarrow$) |\n|----------|:------------:|:--------------:|-----------------:|\n| Mean     |     0.6252    |      7.41$\\times$     |             4.31 |\n| Variance |   2.81E-04   |    4.48E-05    |         1.52E-05 |\n\nRebuttal Tab. 5: The variance of our search runs of OPT-2.7b on LAMBADA. We calculated the variance and mean of accuracy and memory density of 42 runs where each run consisted of 128 trials and found that the run variance is small, proving that our search method consistently finds promising mixed-precision configurations. $^*$ means the accuracy is reported on the training set of SST2, which is reasonable since we are performing post-training quantization without fine-tuning the LLM on the training set.\n\nWe observe that the variance of the 42 runs is small, which means the searched mixed-precision networks are consistently promising.\n\n---\n\n> Q5: When we use mixed precision, shouldn't authors be reporting how much memory density was sacrificed? In line 513 authors mention memory density was not compromised, but it'd help to provide a number (1%? 0.1%? 0.01%?)\n\nThanks for your questions and apologize for our unclearness. The memory density of the search OPT-2.7B in line 513 is 7.42x, which is even better than the uniform 4-bit BFP’s 7.11x. We will clarify this value in the revised version.\n\nIn this paper, the memory density is equivalent to the average bitwidth. We follow [this work](https://proceedings.neurips.cc/paper/2020/hash/747e32ab0fea7fbd2ad9ec03daa3f840-Abstract.html) to calculate the memory density by normalizing the average bitwidth to FP32.\n$$\nmem = \\frac{32}{aw}\n$$\nwhere $mem$ and $aw$ denote memory density and average bitwidth respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors conduct a systematic study over block floating point representations. Observations of which representations work better than other representations provide good insight on the numerical nature of LLM inference. Therefore, future research on numerical aspects of LLMs will find these observations useful. Authors also estimate the impact of quantizations on ASIC accelerators, which might be useful for active research on building hardware accelerators. Caveat is that details of the estimation are deferred to Appendix.", "reasons_to_reject": "I am not sure the proposed method has a comparative advantage over LLM.intX(). Table 3 doesn't have W8A8 version of BFP or LLM.int4(), so we cannot really do a head-to-head comparison with BFP with same bits. (Though LLM.int8 seems to only have 2x memory density, whereas other 8-bit precision methods in Table 3 have 4x-ish density; maybe I am terribly misunderstanding something.) In Table 4, 4-bit version of BFP W4A4 performs worse than LLM.int4(), and BFP W8A8 performs worse than LLM.int8(). 6-bit version BFP W6A6 is in between 4-bit and 8-bit counterparts, but if we are considering custom hardware chips, we could also consider LLM.int6() as a baseline?\n\nI also found the description of hyperparameter search unclear, which makes it difficult to understand implications of experiments. I will leave them in questions for the authors.", "questions_to_authors": "What exactly is the search space of hyperparameter optimization? In line 271-273, it seems like the granularity is a hyperparameter, and can chosen between layer-wise, tensor-wise, channel-wise, etc. Is this searched by HPO, or are they manually configured by authors? Does HPO algorithm choose the number of bits (4, 5, 6, 7, 8) for each of the block? That's my interpretation of \"search for quantisation precision\" (line 303) but I wasn't entirely sure, because the choice of precision across layers is only discussed in Mixed-precision experiments (Figure 3, line 499). Then, for the rest of experiments, do authors fix the number of bits, or do they always do the HPO search? Section 3.3 is written like HPO is always done.\n\nAlso, how does HPO iterations impact the performance of the model? Is it in general difficult to find a good configuration- do we consistently find a good configuration, or is there a high variance across runs?\n\nWhen we use mixed precision, shouldn't authors be reporting how much memory density was sacrificed? In line 513 authors mention memory density was not compromised, but it'd help to provide a number (1%? 0.1%? 0.01%?)", "paper_topic_and_main_contributions": "Authors address the problem of LLM quantization by exploring the combination of different floating point representations and precision levels. They find BFP, which shares an exponent across a block, performs better than various other alternatives (Fixed-point, MiniFloat, DMF, BM, BL). The proposed method is on par with LLM.int8 with 8 bits, a bit underperforms LLM.int4 with 4 bits, and the 6-bit version performs in between. Mixed precision further improves performance with minimal impact on memory density.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Citations in Section 2.2 are broken."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a thorough investigation into quantization methods for Large Language Models (LLMs), focusing on 8-bits or less. Reviewers commended the comprehensive comparison against both baseline and block-based methods across multiple task settings, emphasizing the innovative introduction of non-linear quantization strategies. Particularly notable are the impressive results achieved with nearly lossless 6-bit and 4-bit quantized LLMs using Block Floating Point (BFP), surpassing existing methods in arithmetic and memory density. The study's detailed insights, combined with its well-structured presentation, make it a significant contribution to both numerical LLM research and potential applications in hardware accelerators.

Pros:

extensive experiments and well written

Cons:

Could use more explanation on some ablations
Decision: Accept-Main