LO-BCQ: Locally Optimal Block Clustered Quantization for 4-bit (W4A4) LLM Inference
Authors: Reena Elangovan
Charbel Sakr
Anand Raghunathan
Brucek Khailany
Venue: Accepted by TMLR ()

Abstract: Post-training quantization (PTQ) is a promising approach to reducing the storage and computational requirements of large language models (LLMs) without additional training cost. Recent PTQ studies have primarily focused on quantizing only weights to sub-$8$-bits while maintaining activations at $8$-bits or higher. Accurate sub-8-bit quantization for both weights and activations without relying on quantization-aware training remains a significant challenge. We propose a novel quantization method called block clustered quantization (BCQ) wherein each operand tensor is decomposed into blocks (a block is a group of contiguous scalars), blocks are clustered based on their statistics, and a dedicated optimal quantization codebook is designed for each cluster. As a specific embodiment of this approach, we propose a PTQ algorithm called Locally-Optimal BCQ (LO-BCQ) that iterates between the steps of block clustering and codebook design to greedily minimize the quantization mean squared error. When weight and activation scalars are encoded to W4A4 format (with $0.5$-bits of overhead for storing scaling factors and codebook selectors), we advance the current state-of-the-art by demonstrating $<1$\% loss in inference accuracy across several LLMs and downstream tasks.

Published in Transactions on Machine Learning Research (10/2025)
LO-BCQ: Locally Optimal Block Clustered Quantization for 4-bit (W4A4) LLM Inference
Reena Elangovan NVIDIA Corporation
relangonan@nvviiia.ccm
Charbel Sakr NVIDIA Corporation
csakr@nvidia. com
Anand Raghunathan Deportment of ECE Purdue University
Senghanaathnn@purrirrr edu
Brucek Khailany NVIDIA Corporalion
bkhuulaary@@vvvid..cccm Cnwiiia. com
Reviewed on OpenReview: https: openreview. net/ forun? id= LoWISTgGWM
Abstract
Post-training quantization (PTQ) is promising approach to reducing the storage and computational requirements of large language models (L.LMs) without additional training cost. Recent PTQ studies have primarily focused on quantizing only weights to sub-8-bits while maintaining activations at 8-bits or higher. Accurate sub-8-bit. quantization for both weights and activations without relying on quantization-awant aware training remains a significant challenge. We propose a novel quantization method called block clustered quantization (BCQ) wherein each operand tensor is decomposed into blocks (a block is a group of contiguous scalars), blocks are clustered based their statistics, and dedicated optimal quantization codebook is designed for each cluster. As specific embodiment of this approach, we propose a PTQ algorithm called Locally-Optinn block clustering and codebook design to greedily minimize the quantization mean squared error. When weight and activation scalars are encoded to W4A4 format (with 0.5-bits of overhead for storing scaling factors and codebook selectors), we advance the current state-of-theeare by demonstrating 1% loss in inference accuracy across several LLMs and downstream tasks.
BCQ (LO-BCQ) that iterates between the steps of
1 Introduction
Quantization is a highly effective and widely adopted technique for reducing the computational and storage demands of Large Language Model (LLM) inference. While recent efforts (Wang et al. 2023 Tseng et al.) 2024
Egiazarian al. 2024 Frantar et al. 2023 Lin et al., 2023 have largely focused on weight-only quantization
targeting single-batel inference, activation quantization becomes critical for improving throughput during multi-batch inference scenarios such as cloud-scale deployments serving multiple users. Previous works Yao et al. 2023 Dai et al.) 2021) sub-8-bit quantization of both weights and activations have relied quantiizationaawarr training (QAT) to recover accuracy loss during inference. However, the prohibitive cost of training and unavailability of training data in recent LLMs has made QAT increasingly difficult and motivated recent post-training quantization (PTQ) efforts [iioo et. al. 2023 Rouhani et al. 2023a
Wu et al.
2023
In this paper, we develop a post-training quantization (PTQ) algorithm aimed minimizing the mean squared error (MSF) of any operand tensor. Traditional MSF-optimal scalar quantization algorithms such

---

Published in Transactions on Machine Learning Research (10/2025)
struggle to achieve aggressive bitwidth reduction without significant accuracy loss. To address this limitation, vector (or block) quantization methods have been explored in (Tseng et al., 2024 Egiazarian et al. 2024), which identify MSE-optimal vector codebooks. Despite their promising results for I 4-bit weight-only quantization, these existing approaches face two key challenges. First, they require complex codebook schemes involving weight updales to minimize quantization error; making them challenging to deploy for dynamic compression of activations. Second, they require large codebook sizes (on the order of 2 codebooks with 8 entries each) per-model or per-layer to achieve aggressive model compression.
Lloyd-Max Lloyd 1982
To overcome these limitations, we propose novel clustering and quantization framework called block clustered quantization (BCQ). BCQ consists of two key steps:(1) clustering step applied to operand blocks, and (2) a scalar quantization step individually applied to operand scalars based on their cluster membership. To minimize MSE in this process, we introduce LO-BCQ (locally optimal block clustered quantization), an iterative algorithm that jointly optimizes block clustering and per-cluster codebooks. We prove that LO-BCQ greodily minimizes quantization MSE across iterations by performing locally optimal steps at cach iteration. We apply LO-BCQ on calibration data to identify optimal codebooks. We demonstrate that these codebooks can be frozen during inference across models and across linear layers within models. Using 16 optimal codebooks with 16 entries each derived through LO-BCQ, we achieve state-of-the-art tradc-offs between bitwidth and accuracy without requiring any weight updates during 4-bit quantization of both weights and activations across diverse models and downstream tasks.
1.1 Related work
Recent sub-4-bit quantization proposals such as (Wang et al. 2023 Tseng et al.) 2024 Egiazarian et al. 2024) explore extreme weight quantization while maintaining activations at 8-bit or higher precision. In particular, BitNet (Wang et al. 2023) proposed W1A8 quantization resulting in an aggregate (weights + activations) bitwidth comparable to LO-BCQ. However, BitNet demands train- ing from scratch and despite this large training cost suffers significant. loss in accuracy in downstream tasks. QuiP Tseng et al.) 2024) and AQLM (Egiazarian al. 2024
Llama2-708
OPTQ
Perplexity loss (ower is better)
1.2
1.0
QuipI
0.8
AQLM
0.6
MXFP
propose W2A8 quantization through codebooks. These methods explore vector and additive codebook quantiza- tion, respectively, and rely on large codebook sizes (1MB of memory footprint) and are not directly applicable to ac- tivation quantization. These methods also require weight updates to minimize accuracy loss. In contrast, optimal codebooks (< 0.19KB of memory footprint) identified by LO-BCQ are applicable to both weight and activation quantization and achieves < 1% accuracy loss in down- stream tasks without any weight updates. Supporting such small codebook sizes makes LO-BCQ more amenable to potential hardware acceleration of decompression, Min- imizing quantization MSE using the 1D (Lloyd-Max) and 2D Kmeans clustering has been explored in (Han et al.) 2016 Cho et al.) 2021 2023) and (van Baalen et al. 2024], respectively. In contrast, LO-BCQ iteratively optimizes block clustering alongside Lloyd-Max based optimal scalar quantization ization of block clusters. W4AS quantization has been proposed in (Frantar et al.
0.4
MX
AWQ
Atom
0.2
LO-BCQ (ours)
BF16
0.0
0.26
0.28
0.34 Compression Factor
0.30
0.32
0.36
(wrights and activations and)
Figure 1: Wikitext perplexity loss relative to un- quantized baseline vs compression factor of LO-BCQ compared to previous LLM quantization proposals. Here, compression factor is the cumulative number of bits in the weight and activation tensors that need processed in each layer relative to unquantized BF16 baseline.
2023
Bai et al. 2021 Yao et al. 2022) involving weight updates to
preserve accuracy and (Lin et al.
2023
van Baalen al.] 2024) without any weight updates (PTQ).
Further, (Guo al. 2023 Wei et suppressing outliers.
2023 Kim et al. 2023a perform sub-&-bit weight quantization by
2

---

Published in Transactions on Machine Learning Research (10/2025)
Block (group) quantization is explored for aggressive quantization of both weights and activations in VSQ
(Dai et al. 2021), FuneQuant {Kim et al.
2023b). ZeroQuant-V2 (Yao et al.
2023) Atom (Zhao et al. 2024
through integer number formats, and in [hhag et al. 2023), ZeroQuant- (Wu et al. 2023). MX (Rouhani
ct al. 2023a MXFP (Rouhani et al.)
2023b) and BSFP (Lo et al. 2023 through floating-poin formats.
Figure 1 compares the perplexity loss vs compression factor of LO-BCQ to other quantization proposals.
Here, the perplexity loss relative to an unquantized basoline on the Wikitext-103 dataset for LO-BCQ, MX and MXPP4, and on the Wikitext2 for others. The compression factor refers to the total number of bits in the weight and activation tensors (computed as [A|BA + [W|Bw Bw following (akkr et al. (2017) F that need to be processed in each layer relative to an unquantized BF16 baseline. Depending on the target application, weight or activation quantization may be more important. For the sake of generality, we consider them to be
equally important in our metric. As shown in Figure | LO-BCQ advances the current stale-of-the-arl by achieving the best trade-off between perplexity and compression.
1.2 Contributions
The main contributions of this work are as follows:
We propose BCQ, a block clustered quantization framework that performs per-block quantization by first clustering operand blocks and then quantizing each block cluster using a dedicated codebook.
We derive a locally optimal version of BCQ called LO-BCQ that iteratively optimizes block clustering and per-cluster quantization to provably minimize quantization MSE for any value distribution. We demonstrate that LO-BCQ is applicable to quantization of both weights and activations of LLMs.
We propose block formats for LO-BCQ where each operand block is associated with an index that maps it to one of a set of static codebooks, and a group of blocks (called a block array) share a quantization scale-factor. We vary the length of blocks, block arrays and the number of codebooks to study different configurations of LO-BCQ.
When each of the weight. and activation scalars are quantized to 4-bits (effective bitwidth including per-block scale-factor factors etc. is 4.5 to 4.625 bits), achieve < 0.1 loss in perplexity across GPT3 (1.3B, 8B and 22B) and Llama2 (7B and 70B) models and < 0.2 loss in Nemotron4 (15B and 340B) models, respectively, on the Wikitexl-103 dataset. Further, we achieve < 1% loss in average accuracy across downstream tasks such as MMLU and LM evaluation harness.
To the best of our knowledge, we the first to achieve < 1% loss in downstream task accuracy when both LLM activations and weights are quantized to 4-bits during PTQ (no fine-tuning weight updates).
2 Block Clustered Quantiaztion (BCQ)
In this section, we introduce the concept of block clustered quantization (BCQ) and present the locally optimal block clustered quantization (LO-BCQ) algorithm that minimizes quantization MSE for any operand. We also introduce block formats to support various LO-BCQ configurations.
2.1 Mathematical Definition
Given a tensor X composed of Lx scalar elements, we denote its blockwise decomposition as {b,), where by's are blocks of L.g consecutive clements in X, and the number of blocks given by N₂ = Lx/ Lag. Block clustcred quantization (sce Figure 2 uses a family of N₂ codebooks C = [ where N₂ << No, and clusters the blocks into N, clusters such that each is associated with one of the No codebooks. This procedure is equivalent to creating mapping function f from block b to cluster index in {l, {l,...,Nell .NJ. Quantization (or encoding) proceeds in a two-step process: (i) mapping to assign a cluster index to a given block, and (ii)
1. The size of activations is measured for the prefill phase with context length of 4096 and batch size of 1.
2th notation XX] refers to the total number of scalars in tensor X, and Bx is the bitwidth of X.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Unquantized Block Clusters
Block Clustering (Map each block to one of N: clusters)
Cluster
Cluster 2
Cluster Ne
Operand Blocks:
Unquantized scalar
Bit-width bits
b1
b3
bNb
ON.
Ln scalars each block
Quantized
Block Clusters
Quantized scallr Bit-width bits
Figure 2: Block clustered quantization: Each operand block first mapped to a cluster based on a mapping function and then each scalar of that block is encoded B-bit index to the closest entry in the 2"-en codebook associaled with that cluster.
quantization of its scalars using the codebook corresponding to that index. Formally, denoting b as the result of block clustered quantization a given block in X, this procedure described as:
((b)) (b)
(1)
where is a 2n 2e-cntry codebook that maps each scalar in b to a B-bit index to the closest representation. Each quantized scalar of block obtained as:
b[]] arg min [1
[
(2)
where the notation æly] used describe the yth element in an arbitrary block æ. That is, each scalar in b is an index the closest entry by value in Cí(b).
Once mapped by invoking f, we store the log2(NNc--bit codebook selector for each block. Therefore, the effective bit-width of each quantized scalar is given by:
Bitwidthneq B log2((N.)
In practice, groups blocks called a block-array share 8-bit scale factors each and the codebook entries are 6-bit integers. We discuss this further section 2.4
2.2 Locally optimal block clustered quantization
Our goal is to construct family of codebooks C resulting in minimal quantization MSE during block clustered quantization. Figure 3 presents an algorithm called Locally Optimal BCQ (LO-BCQ) to achieve this goal. LO-BCQ consists of two main steps: (i) updating block clusters with fixed per-cluster codebooks, and (ii) updating per-cluster codebooks with fixed block clusters. This algorithm begins at ileration (initial condition) with set of Ne initial codebooks (c("),..
c. (C"",....,..)) and unquantized operand blocks as inputs. During step 1 of itcration n, with the per-cluster codebooks from the previous iteration (c(=-1),.. ... C.. "), perform block clustering by mapping each block to the codebook that achieves minimum quantization MSE. That is, we use the following mapping function:
(())()) = arg min b c(- (b)l
(4)
Since each codebook C, is associated with a cluster i, mapping to C, is equivalent to mapping cluster Specifically, at iteration n, we construct N, block clusters (a) (B(()) N
NEi where each cluster is defined as:
B((T) = (()))())(b))) for j E {1... No}
(5)
4

---

Published in Transactions on Machine Learning Research (10/2025)
LO-BCQ (iteration n of M)
Initial Condition
Updating
Updating Per-Cluster Codebooks
Updated block clusters
Unquantized Operand Blocks:
MSE;
Suppoose mirimum, map by to cluttr
Block Cluster
Block Cluster 2
Black Cluster N4
b1
DNo
D₁
MSE
D
LM
LM
LM
Initial Per-Cluster Codebooks:
MSE
Updated per-cluster
(9)
2
(00
(0)
(0)
Dyo
each quantizer (dlsstr)) with minimum MIE
codebooks
Llyyd-Max
Figure 3: Overview of LO-BCQ algorithm: The algorithm starts with a set of initial per-cluster codebooks, and then iteratively performs two steps (i) fix per-cluster codebooks and update block clusters and (ii) fix block clusters and update per-cluster codebooks.
NMSE vs iterations during LO-BCQ
MXFP4
LO-BCQ: improved init
LO 1D-BCQ naive init
0.014
0.012
VSQ4
MMSE
0.010
MX4
0.008
0.006
0.004
Iterations
Figure NMSE of LO-BCQ with naive initialization compared to the proposed initialization. Here LOBCQ is configured with block array size of 64 and 16 codebooks.
In step 2, given the updated block clusters from step 1 and quantization bitwidth B, apply the Lloyd-Max algorithm on each block cluster derive optimal 28-entry per-cluster codebooks {c(") ....
(2) ):
c(n) + Ll
(6)
where the Lloyd-Max algorithm (see A.1 Lloyd-Max is equivalent to K-means clustering on 1-dimensional data) invoked on the data of the corresponding cluster B("))
We iterate steps 1 and 2 until convergence find that LO-BCQ converges at M <= 100. Since each of these steps are locally optimal, we find that the quantization MSE is non-increasing for each iteration. As a result, for any given value distribution, our LO-BCQ algorithm greedily minimizes quantization MSE. theoretical proof of this claim provided in section A.2
pre-determined number of iterations M. Empirically, we
2.3 Convergence and Initialization
Prior to clustering, we find that normalizing the operand blocks improves convergence. However, a block-wise normalization factor (or scaling factor) induces computational and memory overheads. Therefore, we perform a second-level quantization this scaling factor to B,-bits and share it across an array blocks of length LA- Furthermore, better convergence observed for larger number of codebooks (N.) and for a smaller block length (Lo). Such trends increase the bitwidth BCQ in equation 3. meaning that LO-BCQ has an inherent trade-off between accuracy and complexity.

---

Published in Transactions on Machine Learning Research (10/2025)
4-bit index to closest representation (code)
Codebooks (static)
Block Array (length 16 to 128)
C
b2
Dn
C
CN.
scale (8-bits)
Block
log N.bbit per-block selector selects codebook
<= 16 codebooks
16 entries per codebook (each entry is INT6)
Figure 5: Block format for LO-BCQ. Each operand block is associated with a log2 selector that selects the best codebook and each scalar is 4-bit index that represents the closest value in the selected codebook. Each block array is associated with a 8-bit scale factor.
We initialize the per-cluster codebooks (C....... based on K-means+ initialization algorithm which maximizes pairwise euclidean distances. In our experiments, found such initialization to lead to significantly better convergence than random one. Further, in step of each iteration, when Lloyd-Max algorithm is invoked in equation G we set the initial centroids corresponding to the codcbooks identified in the previous iteration. Figure compares the MSE achieved by LO-BCQ with naive initialization to random codebooks lo the proposed improved initialization. As shown, LO-BCQ with the proposed initialization converges to a lower MSE than the naive initialization and other block quantization baselines.
2.4 Block formats for LO-BCQ
Figure 5 illustrates the LO-BCQ block format where each operand block of length La is associated with a ( index (result of the mapping function f in equation that selects the best codebook for that block. Each codebook is composed of 2 entries and each scalar in the operand block is a B-bit index that represents the closest value in the selected codebook. Each entry in the codebook is B.-bit integer. Finally, each block array A is associated with scale-factor 8A. This scale-factor and its quantization & to B,-bits are computed as:
$ (2 1) m//mx((bb((A))
& = QF (SA/Sx B.}
(8)
where 3x is per-tensor scale-factor that is shared by the entire operand tensor X and QF denotes quantizer that quantizes a given operand to format F (see section A.4 for more details number formats and quantization method).
The bitwidth of LO-BCQ is computed as:
Bitwidthio-Bco B - 33-oog((N)))4 + +B.//o LA
N * 2 Ba/Txx
where the term No * 2 B./Lx is usually negligible since the memory footprint codebooks (numerator) is negligible compared to the size the operand tensor (denominator). Indeed, we emphasize that LO-BCQ shares set of No 16 codebooks of size <= 0.19K among the scalars of the entire tensor, resulting in negligible memory overhead for storing the codebooks.
In this paper, assume B, = and the data format F is floating point E4M3. Further, each codebook entry is a 6-bit integer (i.e, B. 6) and we vary No between 2 and 16, Lg between 2 and 8, and LA between 16 and 128 to obtain varions LO-BCQ configurations. We list the configurations and their corresponding bitwidths in Table I We perform a detailed ablation of these configurations and present our insights in section 4.3.
3 Applying LO-BCQ for LLM Inference
In this section, we discuss specifics of applying LO-BCQ for LLM inference. Specifically, we first describe the codebook design process, followed by method for activation quantization on-the-fly.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Various LO-BCQ configurations and their bitwidths. Each block array of size LA share an 8-bit scale factor. Each operand block of size L, is associated with log2((N.)-bi) index that maps it one out of Ne codebooks that best represents it.
Lb
Lb
4
Lb =
2
8
16
2
4
2
LA
128
4.1875
4.3125
4.4375
4.5625
4.3125
4.5625
4.5625
64
4.25
4.375
4.5
4.625
4.375
4.625
4.625
32
4.375
4.5
4.625
4.75
4.5
4.75
4.75
16
4.625
4.75
4.875
5
4.75
5
5
Codebooks Number Formats
Layerwise NMSE during LO-BCQ
(MK4)
EZM1 (MXFP4)
ESMO
BCQ:
AANAA AA
LO-BOQ: C1
MX4 (E1M2)
LO-BCQ:
LD-BCQ:
AMSE
MXFM4 (E2M])
LO-BCQ: C5
LO 20-BOQ:C0
10-2
LO-BCO: 2 codebosks
LD-BCQ: 4 codebaoks
LO-BOQ:
BCQ: codebooks
LO-BCQ:
-1.0
-0.5
0.0
0.5
1.0
8
12
16
20
Codeword Values
GEMM Layer Number
Figure 6: LO-BCQ codebooks compared to 4-bit floating point formats and layerwise normalized MSE (NMSE). We compute NMSE for the weights first 20 GEMM layers (QKV, projection and fully-connected) of Llama2-7B model. Note that use the NMSE for better visualization across varying layer data.
We pre-calibrate the LO-BCQ codebooks for both weights and activations offline (prior to inference). Since weights are known, their own data can be used calibration set. On the other hand, activations are dynamic and vary for every input; thus, per common quantization strategics (Wu al. 2020a Sakr et al. 2022). employ randomly sampled calibration set from training data in order to build activation codebooks. Once codebooks are calibrated, we also quantize the codewords to 6-bit integers. The choice of 6- bib was based on empirical observations of accuracy being maintained with LA <= 128.
Figure (left) compares the codebooks identified by the LO-BCQ algorithm in a GEMM layer of a GPT3-1261 model to 4-bit floating point formats such as E1M2, F2M1 and E3M0. The TO-BCQ codebooks outperform other block formats shown in Figure 0 (right) by capturing the arbitrary and non-uniform patterns in the value distributions of LLM operands and allowing each block to map to the codebook that best represents it.
The mapping of operand blocks to the best of available codebooks can be conceptually compared to prior
works that have explored mixed-format quantization such as Tambe et al.
2020 Zadeh et al. 2022
LO-BCQ provides the quantization operation the fiexibility to assign data to any of the sign posts (codewords) in Figure 6 left.). The union of these sign posts covers the real line with resolution that clearly superior that of 4-bit quantizer. Thercfore, we hypothesized that these codebooks nced not be calibrated on a per-tensor (layerwise) basis, but rather, is likely that they would be universally appropriate quantize any tensor fweights and activations), at any layer, for any model. To verify this hypothesis, set of codebooks on data sampled from GPT3 models on Wikitext-103 dataset and froze it. We find that these codebooks achieve comparable quantization MSE compared to those calibrated individually on each operand as shown in Figure 7 which verifies our hypothesis. In our subsequent results, we always employ universally calibrated codebooks.
calibrated
Finally, the small size of LO-BCQ codebooks enables efficient activation quantization on the fly. Indeed, L.-BCQ involves computing the following values per-block array scale-factor 8A, per-block codebook
7

---

Published in Transactions on Machine Learning Research (10/2025)
Universally vs layerwise calibrated codebooks
0.009
0.008
0.007
NMSE
0.006
0.005
LO BCQ: 2 universal cadebooks
LO-BCQ: universal codebooks
8CO; layerwise codebooks
0.004
LO-8CQ: layerwise codebooks
10
15
20
25
30
GEMM Layer Number
Figure 7: Quantization NMSE acheived by universally calibrated codebooks compared to that calibrated layerwise Llama2-7B 7B inputs of first 30 GEMM (QKV, projection and fully-connected) layers.
selector 8, which is the rosult of the mapping function f (Eq. 40, and the index to closest representation in the selected codebook (Eq. 2). Note that the computation of 8,, similar to other block quantization methods Rouhaai et al. 2023b; Dai et al., 2021), simply corresponds to max-reduction (followed by quantization) over the block array whose size is small (<= 128). Further, with LO-BCQ, the codebooks are constant (frozen) with small size (<= 0.19KB). This is an important distinction with other works on codebook
quantization (Tseng et al. 2024 Egiazarian et al. 2024). As such, 56 and b can be concurrently computed across operand blocks.
4 Experimental Evaluation of LO-BCQ
In this section, we present our accuracy studies on downstream tasks comparing LO-BCQ to various other block quantization proposals. Next, we present ablation studies on varying LQ-BCQ configurations and our calibration methodology, namely universal vs local.
4.1
Experimental Setup
We perform accuracy studies on GPT3 (Shoyybi et al. 2020
(1.3B, 8B and 22B), Llama2 Touvron et al. 2024] models. We evaluate PTQ inference al. 2016), MMLU (Hendrycks) 2024). In LM evaluation harness, we infer Race (RA), Boolq (BQ), Hellaswag (HS), Piqa (PQ) and W Winogrande (WG) tasks and in the MMLU dataset evaluate all tasks. In all these models, we quantize GEMM layers including Query, Key and Value computations, Projection layer after self attention and the fully-connected layers.
2023 (7B and 70B) and Nemotron4 (15B and 340B) (Parmar et al.
accuracy on several downstream tasks including Wikitext- 103 (Merity
et al. 2021) and Eleuther Al's LM evaluation harness (Gao et al.
We apply the LO-BCQ algorithm to the operands before inference and pre-calibrate the optimal codebooks. our experiments, we perform this calibration on one batch activations from the training data of the GPT3-126M model and the Wikitext-103 dataset. We freeze these optimal codebooks across operands and models during all of accuracy evaluations. Further, represent each entry of the codebooks 6-bit. integer. That is, once decoded, the inner product computations with a block array during inference can be performed at 6-bit precision Furthermore, perform ablation studies the LO-BCQ configurations with quantization bitwidth ranging from 4.25-bits to 5-bits.
We compare LO-BCQ against previous block quantization works that have explored PTQ of both weights and activations such as VSQ (Dai al. 2021), MX (Rouhani al. 2023a), MXFP Rouhani et al. 2023b).
IIn our experiments this paper, we emulate ("fake") quantization representing the quantized values in BF16 format. Therefore, the computations performed BF16 precision.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: PTQ Perplexity (lower is better) on Wikitext-103 dataset with GPT3, Llama2 and Nemotron4 models.
Method
Bitwidth (W4A4)
Wikitext-103 PPL (A)
GPT3
Llama2
Nemotrond
SB
22B
7B
70B
15B
310B
BF16 (Pretrained)
7.38
6.54
5.06
3.14
5.87
3.48
MX4 (g16)
4.5
8.15 (0.77)
7.69 (1.15)
5.73 (0.67)
3.58 (0.44)
8.88 (3.01)
4.01 (0.53)
VSQ (g16)
8.17 (0.79)
(0.58)
(829)
4.96 (1.82)
7.58 (1.71)
4.19 (0.71)
MXFP4 (x:2)
4.2G
9.12 (1.74)
10.18 (3.64)
5.76 (0.70)
3.69 (0.55)
8.24 (2.37)
4.10 (0.62)
LO-BCQ (g64, N. = 2)
4.25
7.61 (0.23)
6.74 (0.20)
5.31 (0.25)
3.35 (0:21)
6.30 (0.43)
3.67 (0.19)
LO-BCQ (g64, N.
4.5
7.48 (0.10)
6.62 (0.08)
5.19 (0.13)
3.23 (0.09)
6.13 (0.26)
3.60 (0.12)
LO-BCQ (g32, No 16)
4.75
7.45 (0.07)
6.59 (0.05)
5.15 (0.09)
3.20 (0.06)
6.03 (0.16)
3.56 (0.08)
Bitwidth weights and activations including the overheads from per-block arry (group) scale and codebook selectors
QuaRot (Ashkboos et al. 2024), Atom (haoo et al. 2024), , OmniQuant (Shao et al. 2024p and SmoothQuant (iaoo et al. 2023). VSQ and MX perform per-block quantization of 16-element blocks with an 8-bit scale-factor per-block resulting in an effective bit-width of 4.5 bits. VSQ quantizes each scalar to INT4 format and per-block scale-factor to INT8 format. MX performs micro-scaling at per-block level with a 1-bit exponent shared by 2-clement blocks. Each scalar is quantized to INT3. In this paper, we overestimate accuracy of MX by allowing each scalar to have its own exponent, resulting in INT4 precision. The per-block array scale factors of MX are quantized to ESMO format. Therefore, our evaluation results in a bitwidth of 4.5 bits. Further, MXFP explores 32-clement blocks with 8-bit scale-factor per block resulting in effective bitwidth of 4.25 bits. The nunber format of scalars and per-block scale factors are E2M1 and E8M0, respectively. The quantization methodology with these block formats is detailed in A.4.3
Additionally, we compare woight-only (W4A8) LO-BCQ to other weight-only quantization proposals of equivalent bitwidth such as GPTQ (Frantar et al. 2023 AWQ (Lin et al. 2023), QuiP# (Tseng et al. 2024 and AQLM (Fgiazarian al. 2024). For this comparison, we choose a block-array length of 128 for LO-BCQ, matching the group-size of other works.
4.2 Accuracy studies on downstream tasks
We present our comprehensive accuracy evaluations across the Nemotron4, Llama2 and GPT3 models, on the Wikitext-103 103, LM evaluation harness and MMLU datasets. For convenience, we present select LO-BCQ configurations with La in this section. See ?? for accuracy studies other configurations.
4.2.1 Perplexity on Wikitext-100
As shown in Table 2 across large models such as Llamna2-7BB, Nemotronn43340 and GPT3-22B, 4.5-bit LO-BCQ achieves < 0.12 loss perplexity compared to the unquantized baseline on the Wikitext- dataset. MX, MXFP and VSQ perform per-block quantization by associating scale-factor to each block (or a block array) and with single mumber format (quantizer) across blocks. On the other hand, in addition to per-block array scaling, T.-BCQQ allows a block to flexibly map to a codebook that best represents it from set of codebooks. This flexibility allows LO-BCQ to achieve better perplexity. Furthermore, we find that with a larger quantization bitwidth, LO-BCQ achieves better perplexity across models as expected. We achieve these improvements during PTQ, i.e., without any additional training or finetuning.
The number format of per-group (or block array) scalc-factor has a significant impact on accuracy. VSQ is unable to sufficiently capture the range of activations with its INTS scale-factors as observed in Llama2-7B, while it outperforms the ESMO scale-factors of MX in GPT3-22B due to better resolution when representing large values. Across various models, we find that the E4M3 format of LO-BCQ provides sufficient range and resolution to represent the scale-factors.
Table 3 compares LO-BCQ to other PTQ methods that quantize both weights and activations. Here, all methods use group (block array) size of 128. As shown, LO-BCQ significantly outperforms the prior-art. While the prior-art proposes various techniques for suppressing outliers in both weights and activations, LO-BCQ calibrates a set of codebooks that captures various non-uniform value distributions.
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Comparing perplexity loss (lower better) of LO-BCQ to other 4-bit (W4A4) quantization works such as QuaRot, Atom, OmniQuant and Smooth:uant. Here, the perplexity loss Wikitext-103 dataset for LO-BCQ and Wiki2 for others.
Method
Bitwidth (W4A4)
A Wiki PPL
(g128)
Llama2-7B
Llama2-70B
Smooth
77.65
OmniQuant
4.13
9.14
QuaRot
0.46
0.29
Atom
4.13
0.56
0.36
LO-BCQ (N. = 2)
4.19
0.14
0.09
LO-BCQ (N.
0.12
0.07
LO-BCQ (N. 8)
4.44
0.09
0,08
LO-BCQ (N. 16)
Table 4: Comparing perplexity loss of weight-only (W4A16) LO-BCQ to other weight-only quantization methods.
Method
Bitwidth (W4A16)
#codebooks
Llama2-7B
Llama2-701
PQ
HS
Wiki PPL
PQ
WG
GPTQ (g12s)
0.37
76.61
68.19
55.44
0.23
81.23
75.61
63.47
AWQ (g128)
4.13
0.13
77.09
69.53
56.25
0.09
QuiP#
2
77.91
55.78
76.8
AQLM
4.14
2
2
0.09
78.24
67.32
55.99
0.07
81.5
76.48
63.69
4.19
0.14
78.29
68.11
56.55
0.09
81.18
79.24
64.57
LO-BCQ (g128)
56.75
4.44
0.09
77.69
68.75
56.75
0.06
81.50
79.79
65.11
68.90
80.43
Table 5: Comparing perplexity loss sub-4-bit weight-only LO-BCQ to other weight-only quantization methods.
Method
Bitwidth
#codebooks
Wiki PPL
Llama2-7B
Llama2-
Llama2-70f
BF16 (Pretrained)
16
5.47
4.88
3,32
QulP# (LDLQ, no FT) AQLM (FT)
2
22
5.91
5.23
3.61
W3A16
2
212
5.46
4.82
3.36
LO-BCQ (L.DLQ, FT)
3.375
4
5.79
5.12
3.53
5.72
3.49
QuIP# (L.D.., FT) AQLM (FT)
2
2
8.05
6.59
4.44
W2A16
6.59
5.60
3.94
LO-BCQ (LDLQ, no FT)
2.375
8.02
7.12
4.52
2.5
8
6.87
6.16
4.20
4.2.2
Accuracy on LM evaluation harness tasks
Across 0-shot LM evaluation harness tasks in Table LO-BCQ shows significant improvement in average accuracy compared MX, MXFP and VSQ al equivalent bitwidth. Further, across models during 4.5-bit. quantization, LO-BCQ achieves 1% loss in average accuracy compared the respective unquantized basclincs. When the bitwidth of LO-BCQ is increased by varying its configuration, we find that the average accuracy generally increases albeit with few exceptions. Although these variations are small (< 0.5%), we believe that they arise due to the universal calibration of codebooks. Our codebooks are calibrated on a batch of training data from the Wikitext-103 dataset and the GPT3-126M model and remain frozen across all datasets and models.
Although the focus of this work weight-+attvatinn quantization, we also compare to prior art weight-only quantization proposals for completeness. Table g compares weight-only (W4A8) LO-BCQ with block array size of 128 to other weight-only quantization proposals such GPTQ and AWQ of comparable block array size and effective bit-width. As shown, LO-BCQ with 2, 4, 8 and 16 codebooks with effective bitwidth of 4.19, 4.31, 4.44 and 4.56, respectively, achieves significantly lower perplexity loss. It is worth noting that we evaluate this loss on Wikitexl-103 dataset, which is a much larger dataset compared to Wikitex12 used by other works. Further, we compare LO-BCQ against other codebook- based quantization methods such as Quip# and AQLM. As shown, LO-BCQ achieves comparable accuracy using only small codcbook size (8 codebooks with 16 entries each) compared significantly larger codebook sizes (2 codebooks with 8 entries each) required by QuiP# and AQLM.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: LM evaluation Harness 0-shot accuracy (higher better) on Llama2 and GPT3 models.
Method
Bitwidth (W4A4)
Llama2-7B
IAA
BQ
WG
PQ
HS
Avg (A %)
BFI6 QuaRot [g128]
16
79.29
69.35
78.07
57.10
65.65
4.13
63.77
76.77
MX4 (g16)
(2.88)
VSQ (g16)
4.5
31.39
65.75
55.49
67.30
43.51
52.69
(12.96)
MXFP4 (g32)
4.25
11.34
74.00
67.48
77.53
54.22
62.91 (2.74)
LO-BCQ (g64, N,
4.25
42.49
77.58
68.90
77.00
56.93
64.40 (1.25)
LO-BCQ (g64, N₂ 8)
4.5
42.58
77. 43
69.77
77.09
56.51
64.68 (0.97)
LO-BCQ (g32, No 16)
(0.68)
Llama2-700 200
86.23
79.95
81.50
60.27
72.16
QuaRot (g128)
4.13
76.21
82,
MX4 (g16)
4.5
48.04
82.41
76.40
80.58
63.24
70.13 (2.03)
VSQ (g16)
4.5
47.85
82.29
77.27
79.82
61.40
69.73 (2.43)
MXFP4 (g32)
4.25
47.75
83 33.006
76.32
80.58
63.24
70.19 (1.97)
LO-BCQ (g64,
78.77
(0.91)
LO-BCQ (g64, N. 8)
4.5
49.28
84.03
78.37
81.45
64.76
71.58 (0.58)
LO-BCQ (g32, N. 16)
4.75
49.28
84.93
80.66
81.34
65.18
72.28
(40.122
GPT3-811
BFI6
16
41.34
68.35
67.88
78.78
64.16
62.10
MX4 (g16)
1.5
38.
06.27
65.11
75.63
50.77
59.21 (2.89)
VSQ (g16)
4.5
40.86
63.91
66.93
76.28
51.38
59.87 (2.23)
MXFP4 (g32)
4.25
39.71
65.35
67.01
76.12
50.22
59.68 (2.42)
LO-BQQ (g64, N.
4.25
40.48
69.20
66.85
77.31
53.06
61.38 (0.72)
LO-BCQ (g64, N. 8)
1.5
39. 43
69.45
67.72
77.76
53.71
61.61 (0.49)
LO-BCQ (g32,
16)
77.37
(0.74)
GPT3-225
40.67
76.04
70.64
79.16
57.11
64.82
MX4 (g16)
4.5
39.04
72.26
67.96
77.86
54.77
62.38 (2.44)
VSQ (x16)
4.5
40.57
65.81
69.61
77.20
54.82
61.60 (3. 22)
MXFP4 (g32)
4.25
69.61
61.17
75.68
47 60
59.24 (5.58)
LO-BCQ (g64, N. = 2)
4.25
40:48
75.41
69.14
78.
56.06
63.87 (0.95)
LO-BCQ (g64,
70.17
(0.44)
LO-BCQ (g32, N. 16)
4.75
39.62
75.35
69.30
78.89
56.64
63.96 (0.86)
Table 7: MMLU accuracy (higher is better) with Nemotron4 15B, Llama2-7B. 70B and GPT3-22B models.
Method
Bitwidth (W4A4)
Nemo
Llams2
GPT3
151
7B
7UB
22B
BF16
16
64.3
69.12
08.75
MX4 (g16)
4.5
58.15
41.38
65.73
37.07
VSQ [g16]
57.38
37.79
MXFP4 (g32)
4.25
58.28
37.64
66.
32.26
LO-BCQ (g84, N. 2)
4.25
63.17
43.90
68.07
36.71
LO-BCQ (gG4, N.
4.5
63.72
43.90
38.13
LO-BCQ (g32, N. 16)
4.75
64.33
41.50
68. 27
38.34
Table 8: Perplexity Wikitext- 103 dataset various LO-BCQ configurations
N
LA
Llama2-700 70B
(FP32
PPL
3.14)
61
3.35
3.25
3.23
3.21
3.31
3.22
3.27
32
3.22
3.20
16
3.25
3.22
3.20
3.19
3.23
3.20
3.20
GPT3-22B
(FP32
6.64)
61
6.74
6.64
6.62
6.63
6.71
6.61
6.61
32
6.67
6.64
6.61
6.59
6.65
6.64
6.60
16
6.67
6.63
6.59
6.61
6.66
6.63
6.62
Table 9: Perplexity Wikitext-103 dataset with universally calibrated locally calibrated codebooks
Llama2-BB (FP32 PPT. 5.06),
2
16
I.A
Uni verversllly Calibrated Codebooks
Layerwiae Calibrated Codebooks
64
5.31
5.26
5.19
5.18
5.29
5.22
5.19
5.17
32
5.23
6.25
5.18
5.15
5.23
5.19
5.17
5.15
16
5.23
5.19
5.16
5.14
5.20
5.17
5.15
5.14
11

---

Published in Transactions on Machine Learning Research (10/2025)
Table 10: Quantizing LO-BCQ codebook entries to INT4 vs INT6 vs INT8 on Wikitext-103. Perplexity is measured the Llama2-7B model (BF16 PPL 5.06).
Method
Bitwidth of codewords
INT4
INT6
INT8
LO-BCQ (g128, N: 22)
6.21
5.35
5.35
LO-BCQ (R128, =4)
6.21
5.27
5.25
LO-BCQ (g128, No 8)
6.20
5.21
5,21
LO-BCQ (g128, N. = 16)
6.18
3.21
3.19
Table shows the Wiki2 perplexity results on weight-only quantization of Llama2 models with LO-BCQ and compares against state-of-the-art 2-bit and 3-bit quantization proposals such as QulP# and AQLM We find that despite the significantly smaller number of codebooks (<- 8 with 16 entries each) utilized by LO-BCQ compared to about 2 to 2 codebooks with 8 entries each in the other methods, LO-BCQ achieves competitive results. During 3-bit. quantization, LO-BCQ with effective bitwidth of 3.375 bits achieves lower perplexity than Qul'# where LDLQ [Teng et al. (024)) is applied to both methods and no finetuning is performed. Similarly, when compared against 2-bit Qul we find that LO-BCQ with effective bitwidth of 2.375 bits achieves better Wiki2 perplexity. Further, 3.5-bit and 2.5-bit LO-BCQ without finetuning suffers only a small loss compared to finetuned AQLM during 3-bit and 2-bit quantization of weights, respectivcly.
4.2.3 Accuracy on MMLU tasks
Similarly, in 5-shot MMLU tasks LO-BCQ achieves < 1% loss in average accuracy with 4.5-bits per scalar compared to respective unquantized baselines across GPT3-22B and Llama2-70B models. Further, LO-BCQ achieves a significantly better accuracy compared to all of our block quantization baselines such as VSQ, MX and MXFP4 at equivalent bitwidth. Across Llama2 models, LO-BCQ with a smaller bitwidth (4.25-bits) outperforms VSQ and MX4 with comparatively larger bitwidth (4.5-bits). While the 0.5-bit overhead in VSQ and MX4 are used per-block array scale-factors, the 0.25-bil overhead of LO-BCQ is shared between scale-factors and codebook selectors. Therefore, the superior accuracy of LO-BCQ can be attributed to the better representation by selecting the best codebook for each block.
4.3 Ablation Studies
Table 2 shows the perplexity of LO-BCQ on Wikitext-103 dataset and across Llama2-70B and GPT3-22B models when its configuration is varied. For a given La (block length), larger number of codebooks results in better perplexity. This is intuitive since larger number of codebooks leads to better representation of the values each block since LO-BCQ allows it to map to the codebook with best representation. Further, when the block array size is reduced, we achieve better perplexity. The block array corresponds to the granularity of normalization. As discussed in scction 2.3, normalization improves convergence of LO-BCQ and results in better perplexity. When comparing configurations with same bitwidth, we find that the configuration with larger number of codebooks is better than smaller block array. This shows that the per-block metadata is better utilized for codebook selectors than scale factors. Furthermore, we find that reducing the block length (Lg) below 8 results in diminishing returns. This is because, the overhead of storing codebook selectors is larger for a smaller block. For a given bitwidth, configuration with smaller To has fewer codebooks. Thercfore, these configurations result larger loss perplexity. Table 10 compares perplexity achieved by LO-BCQ codebooks with INT4, INT6 and INT8 entries on Llama2-7B model and Wikitext- datatset. As shown, LO-BCQ with INT6 achieves negligible perplexity degradation compared to INT8, while INT4 suffers significantly larger degradation. As result, we quantize LO-BCQ codebooks to INT6.
Table g compares the perplexity with universally calibrated codebooks to codebooks calibrated layerwise pper-tensor) in Llama2-7B model. The layerwise calibrated codebooks achieve slightly better perplexity when the number of codebooks are small (e.g. N. = 2). However, they do not provide significant benefits when N₂ > 4 despite the comparatively larger calibration effort. Therefore, in our experiments in this paper, we have largely explored universally calibrated codebooks.
12

---

Published in Transactions on Machine Learning Research (10/2025)
5 Conclusion and Future Work
We propose a new iterative block clustering and quantization algorithm called LO-BCQ, that greedily minimizes quantization MSE for any operand (weights and activations) through locally optimal steps at each step of the iteration. We demonstrate that LO-BCQ achieves thte-offthhe-rt perplexity across a suite of GPT3, LLama2 and Nemotron4 models on various downstream tasks. Given its strong inference accuracy with W4A4 quantization, we believe LO-BCQ opens new research avenues for even more aggressive quantization of both weights and activations. Furthermore, our approach requires significantly fewer codebooks than prior codebook-based methods and allows these codebooks to be static (frozen) across models and layers within models. This creates new opportunities to improve inference efficiency, which we plan to explore in future work.
References
Saleh Ashkboos, Amirkeivan Mohtashami,, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-frer free 4-bit inference in rotated llms, 2024. URL https://arxiv.. org/ abs/ 24004004464
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Trwin King, and Michael R. Lyu. Towards efficient, post-training quantization of pre-trained language models, 2021.
Minsik Cho, Keivan Alizddeh-Vahid, Saurabh N. Adya, and Mohammad Rastegari. Dkm: Differentiable
k-means clustering layer for neural network compression. ArXie, Abs/2100.12659, 2021. URL https
//api. haaplesemmniicccolarr.ooo htt.://cc.....a..oooooooooooaaaa
Minsik Cho, Keivan A. Vahid, Qichen Fu, Saurabh Adya, Carlo C Del Mundo, Mohammad Rastegari, Devang Naik, and Peter Zatloukal. edkm: An efficient and accurale train-lime weight clustering for large language models, 2023.
Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference. Tn A. Smola, A. Dimakis, and Stoica (eds.), Proceedings of Machine Learning and Systems, vol- ume 3, pp. 873 884, 2021. URL https: stttps://prooeeddngg..t www.ss.....g.g...ee..eeee...eeee org/paper. http:/aaapfffeeaaeeeeeappp /tllss/apper/22///Ei (8664111044461111117777666666663
Paper. .df
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024. URL https://arxiv.or
abs/2401.06115
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference Learning Representations,
2023. URL httpss://operreviert/ net.://weee/eeeee
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https: //zendo org/recordss/1206600
Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. Olive: Accelcrating large language models via hardware-friendyy nutlier-victim pair quantization. ISCA '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700958, doi: 0..1145/35793713588000
URL https:///ol: http:://ddo.orr/1000 111.///711115/33777... 358803
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural network with pruming, trained quantization and huffman coding. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv. wttp://www..aa.....001....
13

---

Published in Transactions on Machine Learning Research (10/2025)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL http:::/aaxxvv.or///bbbsssss 20000000000000000000 03300
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and
Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2023a.
Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. Finequant: Unlocking efficiency with fine-grained weight-only quantization for Ilms, 2023b.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and accoleration, 2023.
S. Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2):129 137, 1982. doi: 10.
Yun-Chen Lo, Tuc-Kuang Lec, and Ren-Shuo Liu. Block and subword-scaling foating-point (BSFP): : An efficient non-uniform quantization for low precision inference. In The Eleventh International Conference on Learning Representations, 2023. URL
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jemnings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Dcopak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, et al. Nemotron-4 15b technical report. arXiv preprint arXie:2402. 16819, 2024.
Bita Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, Lai Shao, Gaurav Kolhe, Dimitry Melts, Jasmine Klar, Renee L'Heureux, Matt Perry, Doug Burger, Eric Chung, Zhaoxia (Summer) Deng, Sam Naghshinch, Jongsoo Park, and Maxim Naumov. With shared microcxponents, a little shifting goes a long way. In Proceedings of the 50th Annual International Symposium on Compuler Architecture, ISCA 223, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9798400700958, doi: 01114//5537737.5588885
URL https:///olt org/10. 10. 1145/3579371 9588933333911
Bita Rouhani, Ritchic Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakkoosroshhii, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradcop Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. Microscaling data formats for deep learning, 2023b.
Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. Analytical guarantees on numerical precision of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pp. 3007-3016. JMrR.org, 2017.
Charbel Sakr, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, William J. Dally, and Brucek Khailany. Optimal clipping and magnitudo-aware diffcrontiation for improved quantization-awarr training, 2022.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models, 2024. URL https://arxiv org/abs.////ooooooooooo.ooo.. 2208. 13137
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2020.
Thierry Tambe, En-Yi Yu Yang, Zishen Wan, Yuntian Deng, Vijay Janapa Reddi, Alexander Rush, David Brooks, and Gu-Yeon Wei. Algorathm-hardware co-design of adaptive floating-point encodings for resilient deep learning inference. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, 2020. doi: 10.100//DAC18072200000... 992.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv
org/
Albert Tseng, Jenry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks, 2024. URL https://arrs: /wwpvvxxvvvvvv brgg/bs/
04396
Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization, 2024.
URL https://aaaat .rg. wtt.///////...122222223 2402.15319
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models, 2023. URL
attpp://arxa http:///arxivv.rrrr//////////// wtgg 11453
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models, 2023.
Hao Wu, Patrick Judd, Xiaojic Zhang, Mikhail Isacv, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. CoRR, Abs//204.0096/ 2020a. URL
attps://arxiva orggabs////roo.ooogggoogggeer// /204. 09602
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep
learning inference: Principles and empirical evaluation, 2020b.
Xiaoxia Wu, Zhewei Yao, and Yuxiong He. Zeroquant-fp: leap forward llms post-training w4a8 quantization using floating-point formats, 2023.
Guangxuan Xiao, Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps://openreeieet wwtp://wwww...oorrriiittt-tt-e..
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2. Exploring post-training quantization in Ilms from comprehensive study to low rank compensation, 2023.
Ali Hadi Zadch, Mostafa Mahmoud, Ameer Abdelhadi, and Andreas Moshovos. Mokey: Enabling narrow fixed-point inference for out-of-the-box floating-point transformer models. In Proceedings of the 490h Annual International Symposium on Computer Architecture, ISCA '22, pp. 888-901, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450386104. doi: 10. wtt..////......444444449...000
URL
Yijia Zhang, Lingran Zhao, Shijic Cao, Wenqiang Wang, Ting Cao, Fan Yang, Mao Yang, Shanghang Zhang, and Ningyi Xu. Integer or floating point? new outlooks for low-bit quantization on large language models, 2023.
Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikei. Atom: Low-bit quantization for efficient and accurate 1mm serving, 2024. URL
15

---

Published in Transactions on Machine Learning Research (10/2025)
A
Appendix
A.1
Lloyd-Max Algorithm
For a given quantization bitwidth B and an operand X, the Lloyd-Max algorithm finds 2 quantization levels [a such that quantixing X by rounding each scalar in X to the nearest quantization level minimizes the quantization MSE.
The algorithm starts with an initial guess of quantization levels and then iteratively computes quantization thresholds [T and updates quantization levels { Specifically, at iteration thresholds are set to the midpoints of the previons iteration's 's levels:
(n-1) 2 i+1
(m)
for i = 1...2
.2 -1
2
Subsequently; the quantization levels are re-computed conditional means of the data regions defined by the new thresholds:
a!-T = x E (
7. for 1
where to satisfy boundary conditions we have TD - 0o and T2" = co. The algorithm iterates the above steps until convergence.
Figure compares the quantization levels of 7-bit floating point (E3M3) quantizer (left) to a 7-bit Lloyd-Max quantizer (right.) when quantizing layer of weights from the GPT3-126M model at per-tensor gramularity. As shown, the Lloyd-Max quantizer achieves substantially lower quantization MSE. Further, Table [ shows the superior perplexity achieved by Lloyd-Max quantizers for bitwidths of 7, 6 and 5. The difference between the quantizers is clear at 5 bits, where per-tensor FP quantization incurs a drastic and unacceptable increase in perplexity, while Lloyd-Max quantization incurs a much smaller increase. Nevertheless, we note that even the optimal Lloyd- quantizer incurs a notable (~ 1.5) increase in perplexity due to the coarse granularity of quantization.
Unquantized Operand
Quantized Operand
10
(E3M3)
Lloyd-Max (7-bits) MSE 0.0016
MSE = 0.0028
104
10
10a
10
10
10
10
107
-0.75
0.50
0.75
0.00
0.50
Figure 8: Quantization levels and the corresponding quantization MSE of Floating Point (left) Lloyd-Max (right) Quantizers for a layer of weights in the GPT3-126M model.
Table 11:
Comparing perplexity (lower is better) achieved by floating point quantizers and Lloyd-Max quantizers on GPT3-126M model for the Wikitext-103 dataset.
Floating-Poii Point Quantizer
Lloyd- Max Quantizer
Bitwidth
Best Format Wikitext- Perplexity
Wikitext. 103 Perplexity
E3M3
18.3
18.27
E3M2
19.07
E4MO
43.89
19.71
A.2 Proof of Local Optimality of LO-BCQ
For given block by, the quantization MSE during LO-BCQ can be empirically evaluated as 1 b, 6 where by is computed from equation 11 as (b,,))(b)- Further, for given block cluster B,, we compute
16

---

Published in Transactions on Machine Learning Research (10/2025)
the quantization MSE as [B. EOEB, t, --1bb-o - ci (b)lB) Therefore, al the end of iteration n, we evaluate the overall quantization MSE j(m) for given operand X composed of Ne block clusters
N
((m)
Σ DEB(
16 B. 1h (b)1
N
Lb
At the end of iteration n, the codebooks are updated from C(n-1) to C(a), However, the mapping of given vector by to quantizers ((") remains as (b)). At the next iteration, during the vector clustering step, (b,) finds new mapping of by to updated codebooks ((n) such that the quantization MSE over the candidate codebooks is minimized. Therefore, we obtain the following result for by:
Lb 1b, -C yurroa (b) - Lb 1 bb; - (( (by)12
That is, quantizing bj at the end of the block clustering step iteration n + results lower quantization MSE compared to quantizing at the end of iteration n. Since this is true for all b X, we assert the following:
N
j(n+1)
Σ
- Ibb LB
c. (b) J()
(10)
Bl++1)
BEB(H
where 1(+11) is the the quantization MSE after the vector clustering step at iteration
Next, during the codebook update step at iteration 1, the per-cluster codebooks (()) are updated to Cin+1) by invoking the Lloyd-Mas Max algorithm (Lloyd 1982). We know that for any given value distribution, the Lloyd-Max algorithm minimizes the quantization MSE. Therefore, for given vector cluster B; we obtain the following result:
Σ bEB(")1)
1
cf (m+ (b)
Σ DEB("t1)
Lb -
-16--00000 (b)1
(11)
B
B
The above equation states that quantizing the given block cluster B, after updating the associated codebook from c(a) to c(x+1) results in lower quantization MSE. Since this is true for all the block clusters, we derive the following result:
N No
((++))
Σ bER(" (1)
b c(++1) (b)ll < j(+1)
In
(12)
B
x+)
Following 110 and we find that the quantization MSE is non-increasing for each iteration, that is, (MM where M the maximum number of iterations.
j(l) ( (3)
Figure shows the empirical convergence of LO-BCQ across several block lengths and number of codebooks.
Also, the MSE achieved by LO-BCQ is compared to baselines such MXFP and VSQ. As shown, LO-BCQ converges to a lower MSE than the baselines. Further, achieve better convergence for larger number of codebooks (Nc) and for smaller block length (L,), both of which increase the bitwidth of BCQ (see Eq 3
A.3 Additional Accuracy Results
A.4 Number Formats and Quantization Method
A.4.1
Integer Format
An n-bit signed integer (INT) is typically represented with a 2s-complement format Yao al. 2022
Xiao
et al.
2023
Dai et. al. 2021 where the most significant. bit denotes the sign.
17

---

Published in Transactions on Machine Learning Research (10/2025)
NMSE vs Iterations during LO-BCQ
0.0200
Block length-64, codebooks-4
BCQ: Block longth-64, 64, Num todebooks-16
0.0175
length=10, LO-BCQ: Black m6ngh=11, Num codebooks 16
0.0150
0.0125
V504
0.0200
0.0050
0.0025
20
40
60
80
100
Iterations
Figure 9: NMSE vs iterations during LO-BCQ compared to other block quantization proposals
Lb +
8
2
Ne
2
4
16
2
4
2
LA
GPT3-1.35
(FP32
PPL
9.98)
64
10.40
10.23
10.17
10.15
10.28
10.18
10.19
10.25
10.20
10.15
10.12
10.23
10.17
10.17
16
10.22
10.16
10.10
10.09
10.21
10.14
10.16
GPT3-8B
(FP32
PPL
7.38)
64
7.61
7.52
7.48
7.47
7.55
7.49
7.50
32
7.52
7.50
7.46
7.45
7.52
7.48
7.48
16
7.51
7.48
7.44
7.44
7.51
7.49
7.47
Table 12:
Wikitext- 103 perplexity across GPT3-1.3B and 8B models.
Lb +
8
N
2
4
16
L
Tlama2-71 7B
(FP32 PPT.
5.06)
64
5.31
5.26
5.19
5.18
32
5.23
5.25
5.18
5.15
16
5.23
5.19
5.16
5.14
Nemotrond-159
(FP32 PPL.
5.87)
64
6.3
6.20
6.13
6.08
32
6.24
6.12
6.07
6.03
16
6.12
6.14
6.04
6.02
Nemotron4-3
340B
(FP32
PPL
3.48)
64
3.67
3.62
3.60
3.59
32
3.63
3.61
3.59
3.56
16
3.61
3.58
3.57
3.55
Table 13: Wikitext-100 perplexity compared to FP32 baseline in Llama2-7B and Nemotron4-155) 340B
models
18

---

Published in Transactions on Machine Learning Research (10/2025)
L
8
N₂
4
16
8
16
LA
Llama2-17
7B (FP32 Accuracy 45.8%)
Llama2-700 70B
(FP32 Accuracy 69.12%)
64
43.9
43.4
43.9
44.9
68.07
68.27
68.17
68.75
32
44.5
43.8
44.9
44.5
68.37
68.51
68.35
68.27
16
43.9
42.7
44.9
45
68.12
68.77
68.31
68.59
GPT3-222 22B
(FP32 Accuracy
38.75%
Nemotron4-15B
(FP32 Accuracy = 64.3%)
36.71
38.85
38.13
38.92
63.17
62.36
63.72
64.09
32
37.95
38.69
39.45
38.34
64.05
62.30
63.8
64.33
38.88
38.80
38.31
38.92
63.22
63.51
63.93
64.43
Table 14: Accuracy on MMLU dataset across GPT3--2B, Llama2-7B, 70B and Nemotron4- models.
Lb +
8
No
2
4
16
4
8
16
LA
Race (FP32
Accuracy = 37.51 17.51% 1))
Boolq
(FP32 Accuracy
64.62%)
64
36.94
37.13
36.27
37.13
63.73
62.26
63.49
63.36
32
37.03
36.36
36.08
37.03
62.54
63.51
63.49
63.55
16
37.03
37.03
36.46
37.03
61.1
63.79
63.58
63.33
Winogrande (FP32
Accuracy
58.01%)
Piqa
(FP32
Accuracy
74.21%)
64
58.17
57.22
57.85
58.33
73.01
73.07
73.07
72.80
32
59.12
58.09
57.85
58.41
73.01
73.94
72.74
73.18
16
57.93
58.88
57.93
58.56
73.94
72.80
73.01
73.94
Table 15:
Accuracy on LM evaluation harness tasks on GPT3-1.3B model.
La
+
8
N₂
2
4
8
16
2
4
8
16
LA
Race (FP32
Accuracy = 41.34%)
Boolq
(FP32
Accuracy
68.32%)
64
40.48
40.10
39.43
39.90
69.20
68.41
69.45
68.56
32
39.52
39.52
40.77
39.62
68.32
67.43
68.17
69.30
16
39.81
39.71
39.90
40.38
68.10
66.33
69.51
69.42
Winogrande
(FP32 Accuracy
67.88%)
Piqa
(FP32
Accuracy
78.78%
64
66.85
66.61
67.72
67.88
77.31
77.42
77.75
77.64
32
67.25
67.72
67.72
67.00
77.31
77.04
77.80
77.37
16
68. 11
68.90
67.88
67.48
77.37
78.13
78.13
77.69
Table 16:
Accuracy LM evaluation harness tasks GPT3-8B model.
19

---

Published in Transactions on Machine Learning Research (10/2025)
Lb 7
8
N₂
2
16
8
16
LA
Race (FP32
Accuracy 40.67%)
Boolq (FP32 Accuracy
76.54%)
64
40.48
40.10
39.43
39.90
75.41
75.11
77.09
75.66
32
39.52
39.52
40.77
39.62
76.02
76.02
75.96
75.35
16
39.81
39.71
39.90
40.38
75.05
73.82
75.72
76.09
Winogrande (FP32 Accuracy
70.64%)
Piqa
(FP32
Accuracy
79.16%)
64
69.14 14
70.17
70.17
70.56
78.24
79.00
78.62
78.73
32
70.96
69.69
71.27
69.30
78.56
79.49
79.16
78.89
16
71.03
69.53
69.69
70.40
78.13
79.16
79.00
79.00
Table 17:
Accuracy on LM evaluation harness tasks on GPT3-22B model.
Lb
+
8
No
2
4
16
4
8
16
LA
Race (FP32
Accuracy = 44.4%)
Boolq
(FP32 Accuracy = 79.29%)
64
42.49
42.51
42.58
43.45
77.58
77.37
77.43
78.1
32
43.35
42.49
43.64
43.73
77.86
75.32
77.28
77.86
16
44.21
44.21
43.64
42.97
78.65
77
76.94
77.98
Winogrande (FP32 Accuracy
69.38%)
Piqa
(FP32
Accuracy
78.07%)
64
68.9
68.43
69.77
68.19
77.09
76.82
77.09
77.86
32
69.38
68.51
68.82
68.90
78.07
76.71
78.07
77.86
16
69.53
67.09
69.38
68.90
77.37
77.8
77.91
77.69
Table 18:
Accuracy on LM evaluation harness tasks on Llama2-7B model.
La
+
8
N₂
2
4
8
16
2
4
8
16
LA
Race (FP32
Accuracy = 48.8%)
Boolq
(FP32
Accuracy
85.23%)
64
49.00
49.00
49.28
48.71
82.82
84.28
84.03
84.25
32
49.57
48.52
48.33
49.28
83.85
84.46
84.31
84.93
16
49.85
49.09
49.28
48.99
85.11
84.46
84.61
83.94
Winogrande (FP32
Accuracy
79.95%)
Piqa
(FP32
Accuracy
81.56%)
64
78.77
78.45
78.37
79.16
81.45
80.69
81.45
81.5
32
78.45
79.01
78.69
80.66
81.56
80.58
81.18
81.34
16
79.95
79.56
79.79
79.72
81.28
81.66
81.28
80.96
Table 19:
Accuracy on LM evaluation harness tasks Llama2-70B model.
20

---

Published in Transactions on Machine Learning Research (10/2025)
A.4.2 Floating Point Format
An n-bit signed floating point (FP) number x comprises a 1-bit sign (Zsign), Bm-bit mantissa (Imant) and Bą-bit exponent (Nexp) such that Bm + B, 1. The associated constant exponent bias (Fbias) is computed as (2B-11-1) We denote this format as ÆB, MBm.
A.4.3 Quantization Scheme
A quantization scheme dictates how a given unquantized tensor is converted to its quantized representation. We consider FP formats for the purpose of illustration. Given an unquantized tensor X and an FP format. EB MB.. we first, we compute the quantization scale factor sx that maps the maximum absolute value of X the maximum quantization level of the En, Mn. format as follows:
max (X)
8x max BeexxEB,,BB... MBm)
(13)
In the above equation, |. denotes the absolute value function.
Next, we scale X by sx and quantize it to X by rounding to the nearest quantization level of EB, MB.. as:
X
X round-to-ncaree
EB, MBm.
(14)
sx
We perform dynamic max-scaled quantization dynamically computed during runtime.
[Wu et al. 2020b), where the scale factor s for activations is
A.5 Vector Scaled Quantization
During VSQ (Dai et al. 2021. the operand tensors are decomposed into 1D vectors hardware friendly manner as shown in Figure Since the decomposed tensors are used as operands in matrix multiplications during inference, it beneficial to perform this de- composition along the reduction dimension of the multiplication. The vectorwise quantization is performed similar to tensorwise quanti- zation described in Equations 13 and 14 where scale factor 8, is required for each vector that maps the maximum absolute value of that vector to the maximum quantization level. While smaller vector lengths can lead larger accuracy gains, the associated memory and compulational overheads due to the per-vector scale factors increases. To alleviate these overheads, VSQ (Dai et al.) 2021 proposed a second level quantization of the per-vector scale factors to unsigned integers, while MX (Rouhani et al. 2023b) quantizes them to integer powers of 2 (denoted as (TT)).
Input Tensor
Input Tensor
Vector
size
One scale factor for each vector.
Figure 10: Vectorwise decomposition for per-vector scaled quantization (VSQ (Dai
et al. 2021
A.5.1 MX Format
The MX format proposed in (Rouhani et al. 2023a introduces the concept sub-block shifting. For every two scalar elements of b-bits each, there is a shared exponent bit. The value of this exponent bit determined through empirical analysis that targets minimizing quantization MSE. We note that the FP format E M, is strictly better than MX from an accuracy perspective since it allocates dedicated exponent bit to each scalar as opposed to sharing it across two scalars. Therefore, we conservatively bound the accuracy of a + 2-bit. signed MX format with that of a E, M, format our comparisons. For instance, we use E1M2 format as a proxy for MX4.
Figure 11 compares our 4-bit LO-BCQ block format MX (Rouhani al. 2023a). As shown, both LO-BCQ and MX decompose given operand tensor into block arrays and each block array into blocks. Similar to MX,
21

---

Published in Transactions on Machine Learning Research (10/2025)
LO-BCQ-4
MX-4
Operand Tensor X Block Arrey
Operand Tensor Arrays
Block Array Blocks
Block Array
Sa
A
Aa
A
Codebooks
A
A
02
N. (2.4. 8, LG) G
ANA
coditoo
(2,4.)
An
1-att soll
Co...
ontries
s-alt E4MBI
Array lenoth
B-bit scale
lnnth
Effective
INTO entry
S-hits
Coaes antra in
Figure 11: Comparing LO-BCQ to MX format.
find that per-block quantization (I& < LA) leads to better accuracy due to increased flexibility. While MX achieves this through per-block 1-bit micro-scales, associate dedicated codebook to each block through a per-block codebook selector. Further, MX quantizes the per-blocks array scale-factor to ESMO format without per-tensor scaling. In contrast during LO-BCQ, we find that per-tensor scaling combined with quantization of per-block array scale-factor to E4M3 format results in superior inference accuracy across models.
22


Reviews:
[{"review_id": "wGr9o3d6w7", "reviewer": "Reviewer_4WbB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written, and provides good background information on block quantization methods needed to understand the authors' approach\n- The approach is elegant and explained clearly: both the assignment of blocks to codebooks and codebook optimization happens in an alternating scheme, inspired by existing approaches to minimize MSE and using the LloydMax algorithm\n- The approach is benchmarked with large Transformer models, showcasing its robustness, applicability, and benefits at scale\n\nWeaknesses:\n- Most of the results are related to language modelling. Large-scale models are used for a variety of tasks outside of language, so while I would still expect it to work well, ultimately the authors' approach has not been tested as extensively on different areas such as vision tasks, where people are also interested in quantization. It could be that BCQ in general is a strong quantization approach\n- The work does not significantly explore and discuss the choice of hyperparameters (number of clusters/codebooks) beyond the evaluation of a few configurations\n- There is a lack of discussion about how the resultant quantization produces similar or different results from other approaches. If there are interesting features learned by the approach, it would be interesting to see an analysis of them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Recent work has sought to improve the performance of quantized neural network by quantizing operands by blocks, allowing for different quantization configurations for different parts of a model. The common approach is to optimize a set of code books such that the error of quantization for the blocks is minimal. Here, the authors propose improvements to the approach, where they also optimize the assignment of blocks to codebooks. By doing this, they develop a new clustering+quantization algorithm that performs block clustering with state-of-the-art performance on compressing large language models at 4-bit weights and activation (W4A4) precision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the work could be strengthened by further discussing the hyperparameter tuning. In particular, further discussion on the effect of and motivations for block size selection would be interesting. For example, why not have a block for every vector, so that every vector can be assigned independently to a codebook? \n\nI would also be curious to know more about the properties of codebooks and blocks learned by the authors' method. Ie. is it possible to analyze how they are quantizing operands differently from previous approaches?\n\nFinally, an analysis of the BCQ approach on tasks outside of the domain of NLP would be valuable. However, I acknowledge that the scope of the paper is explicitly LLMs.\n\nIn general, I think the paper is well-written and self-contained with convincing results as is, so I do not think any of the above suggestions are necessary for my recommendation for acceptance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uuKp7hzyCR", "reviewer": "Reviewer_uJtg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength : \n* Strong Empricial results across different variants of LLM on different standard metrics.\n* General two step approach to quantize any model.\n\nWeakness :\n* Paper is not clearly written, please refer to summary\n\nI don't see much weakness as the authors use numbers to justify the method. Numbers look so good, I really wanted to try the quantize model to double check the evaluation score. If they could open source the code or provide the steps for evaluation that would help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I found paper pretty challenging to read. Sections 2.1–2.2 do a good job of laying out the idea of block-clustered quantization, but they never say “codebook entries live in 6 bits” or “scale factors get 8 bits” until later.\nAs a result, readers naturally assume all quantized values use the same bit-width, Only in sec 3 (footnote 3) and sec 4.1 do they reveal. The bit width used in this paper all should be in section 2 clearly explained not in foot note.\n\nPaper can be summarized by its core Algorithm (LO-BCQ)\n1. Block to Codebook assignment\n    * For each normalized block, compute average MSE versus each of the N codebooks (each with 16 FP32 centroids de-quantized from 6-bit storage).\n    * Pick the codebook with the smallest FP32 MSE—store its selector index.\n2. Codebook update\n    * Gather all normalized scalars assigned to each cluster and run 1-D k-means to refine its 16 centroids.\n    * Warm-start from previous centroids.\n\n\nNovelty & Contributions\n* Dynamic block clustering  lets each block pick the codebook that best fits its distribution.\n* Alternating, locally-optimal assignment + centroid updates with k-means++.\n* Two-level scale quantization so per-block dynamic ranges don’t blow up the bit-budget.\n* Universal codebooks that generalize across layers and models, removing the need for per-tensor calibrations.\n* End-to-end system demonstrating weight+activation quantization on LLMs with <1 % perplexity or accuracy loss.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please clarify the bit width in the section 2, as the algorithm, and definition is explained in section 2.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L4drpYmdDd", "reviewer": "Reviewer_bj5M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper presents a well-executed and intuitive idea for weights and activation quantization that yields good empirical results.  The experimental evaluation is reasonable and largely supports the primary claims. \n\n- The proposed idea builds on methods like MX4 and VSQ by allowing selection from multiple quantizers. This seems reasonable and logical step. \n\n- Please consider discussing the choice of hyperparameters like codeword bitwidth, scale factor bitwidth and how they are selected.\n\n\nOverall, I found the paper provide a simple and effective idea. Since my knowledge in the area is limited, I might not be able to fully appreciate the contextualization of this work in the literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of post-training quantization (PTQ) for large language models (LLMs) to 4-bits for both weights and activations (W4A4). The proposed method, Block Clustered Quantization (BCQ), first groups contiguous scalars in a tensor into blocks. It then clusters these blocks and assigns a dedicated codebook to each cluster for quantization. The paper introduces an iterative algorithm, Locally-Optimal BCQ (LO-BCQ) to jointly optimize the block-to-cluster assignments and the per-cluster codebooks by alternating between a block mapping step and a codebook generation step using the Lloyd-Max algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)