Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations
Authors: Patrick Blumenberg
Thomas Graave
Tim Fingscheidt
Venue: ICLR 2026 Poster (2026)
Keywords: quantization
large language models
LLMs
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) demand extensive memory capacity during both fine-tuning and inference. To enable memory-efficient fine-tuning, existing methods apply block-wise quantization techniques, such as NF4 and AF4, to the network weights. We show that these quantization techniques incur suboptimal quantization errors. Therefore, as a first novelty, we propose an optimization approach for block-wise quantization. Using this method, we design a family of quantizers named 4-bit block-wise optimal float (BOF4), which consistently reduces the quantization error compared to both baseline methods. We provide both a theoretical and a data-driven solution for the optimization process and prove their practical equivalence. Secondly, we propose a modification to the employed normalization method based on the signed absolute block maximum (BOF4-S), enabling further reduction of the quantization error and empirically achieving less degradation in language modeling performance. Thirdly, we explore additional variations of block-wise quantization methods applied to LLMs through an experimental study on the importance of accurately representing zero and large-amplitude weights on the one hand, and optimization towards various error metrics on the other hand. Lastly, we introduce a mixed-precision quantization strategy dubbed outlier-preserving quantization (OPQ) to address the distributional mismatch induced by outlier weights in block-wise quantization. By storing outlier weights in 16-bit precision (OPQ) while applying BOF4-S, we achieve top performance among 4-bit block-wise quantization techniques w.r.t. perplexity.

Published as a conference paper at ICLR 2026
IMPROVING BLOCK- LLM QUANTIZATION BY 4-B BLOCK-WID OPTIMAL FLOAT (BOF4): ANALYSIS AND VARIATIONS
Patrick Blumenberg, Thomas Graave & Tim Fingscheidt Institute for Communications Technology Technische Universität Braunschweig Braunschweig, 38 38106, Germany
(patrick ck. bl umenberg, thomas. Eingese (etteeeddd@@uubbbeee (etuubs.a
ABSTRACT
Large language models (LLMs) demand extensive memory capacity during both fine-tuning and inference. To enable memory-efficient fine-tuning, existing methods apply block-wise quantization techniques, such as NF4 and AF4, to the network weights, We show that these quantization techniques incur suboptimal quantization errors. Therefore, first novelty, we propose an optimization approach for block- wise quantization. Using this method, we design family of quantizers named 4-bit block-wise optimal floar (BOF4), which consistently reduces the quantization enror compared to both baseline methods. We provide both a theoretical and data- driven solution for the optimization process and prove their practical equivalence. Secondly, we propose modification to the employed normalization method based on the signed absolute block maximum (BOF4-S), enabling further reduction of the quantization error and empirically achieving less degradation in language modeling performance. Thirdly, we explore additional variations of block- wise quantization methods applied to LLMs through an experimental study the importance of accurately representing zero and large-magnitude weights on the hand, and optimization towards various error metrics the other hand. Lastly, we introduce mixed- -eexcdd-reciiion quantization strategy dubbed outlerr-pessrviigg quantization (OPQ) to address the distributional mismatch induced by outlier weights in block-wise quantization. By storing outlier weights in 16-bit precision (OPQ) while applying BOF4-S, we achieve top performance among 4-bit block- wise quantization techniques w.r.t. perplexity.
INTRODUCTION
Driven by scaling the transformer architecture to billions of parameters, large language models (LLMs) have achieved remarkable performance in language modeling tasks. However, their size poses significant challenges for deployment, particularly in memory-comee conoory-conttrained settings. Numerous post-training quantization (PTQ) techniques have been developed to mitigate this, reducing the memory footprint of the weights and, in many cases, improving inference speed (Frantar et al. 2023
Lin al. 2024 Xiao al. 2023 Liu al.) 2024). Fine-tuning imposes even greater memory demands, making it challenging to adapt LLMs on consumer-grade GPU hardware. To address this, Dettmers al. (2023) introduced QLoRA, memory-effciint fine-tuning method that combines 4-bit quantization of pre-trained weights with low-rank adaptation (LoRA) (Hu et al. (2022). For quantization, [Dettmers et al. (2023) propose 4-bit NormalFloat (NF4), quantization method with fixed codebook. This method normalizes blocks of network weights by their absolute maximum (block-wise absmax normalization). Unlike calibration-data-brr free. This means that NF4 quantizes weights without computing network activations, making the quantization process itself far more efficient in terms both time and memory, while maintaining acceptable accuracy degradation at 4 bits per weight. Dettmers et al. 2023) claim that the NF4 codebook is information-theorrttcall theoretically optimal due to its equal utilization of the 16 reconstruction levels. However, Yoshida (2023) demonstrates that this claim is incorrect. We add that equal
PTQ methods, NF4 is data-

---

Published as a conference paper at ICLR 2026
utilization of reconstruction levels is not theoretically justified criterion for the optimality of quantizer. YYshhdda (2023) also proposes an alternative codebook (AF4) designed to address the shortcomings of NF4.
In this work, we show that neither NF4 nor AF4 minimizes the quantization error of the network weights. For the first time, we provide rigorous mathematical analysis of block-wise absmax quantization and explore multiple design variations through an experimental study. As a first contribution, we derive (Loyyd 1982) that computes the correct, informatiin-theoretieeee absmax quantization w.r.t. the mean absolute error (MAE) or mean squared error (MSE) criterion. Additionally, we propose an alternative normalization technique: Instead of normalizing blocks by their absolute maximum value, we normalize by the signed absolute maximum. This simple modification results in significant reduction of the quantization error. Using our EM algorithm, we compute a family of optimal codebooks which we refer as 4-bit block-wise optimal float (BOF4), or BOF4-S when signed normalization is used. Furthermore, quantization sensitive to outlier weights affecting the distribution of the normalized weights. We address this by introducing outlier-preservee preserving quantization (OPQ) that stores outliers in 16- precision. When combined with BOF4-S, OPQ substantially improves perplexity over NF4 and AF4.
appectaion-mmaimizaaiin (EM) algorithm inspired by Lloyd's algorithm
optimal codebook for block-wise
identify that block-wise absmax
The paper is structured as follows: Section reviews related work block-wise quantization. Section 3 outlines our mathematical analysis and novel quantization methods. Section details the experimental setup, and Section 5 presents and discusses the results. We conclude in Section 6
RELATED WORK
Block-wise quantization based blocks of input values normalized by their absolute maximum was introduced Detttmess et al. (2022) as method for quantizing optimizer states during neural
network training. Subsequent works (Dettmers al.] [0233 Yoshida 2023) Dotzel al.) (0224
applied this technique to LLM network weights for memory fine-tuning. We refer to this
quantization method as block-wise absmax quantization.
2.1 BLOCK- WISE ABSMAX QUANTIZATION
In block-wise absmax quantization network weights Wa, E R are first grouped into blocks, with block indices b = (l...., B). and indices weights within a block E I (1...., /}. where B E I is the number of blocks, and I N the block size. Then, the weights are normalized by the absolute maximum weight in their respective block:
w
max
max Vel
Whb
bEB
Whi
E [-11] max
iEIEEE
(2)
Next, each normalized weight Fa, quantized independently using scalar quantization. The absolute block maxima ugmax, commonly referred to quantization constants, stored in addition the quantized weights for later decoding. Overall, the block-dependent quantization function Q () for weights Wh,s à is defined as
Qo(un,)) wh
max
Q Woet
wh
(2b.i):
(3)
where Q0 is block-independem quantization function.
2.2 4-BIT BLOCK- WISE QUANTIZATION FOR LLMS
In this section, we discuss the previous block-wite wise absmax quantization that our work builds upon.
4-bit NormalFloat (NF4) NF4 (Dettmers et al. 2023) is 4-bit scalar quanitzer for block-wiss wise absmax quantization. The L = 2 16 reconstruction levels &(E), É = {1....,L) L} are computed based quantiles of the assumed Gaussian network weight distribution pw = _'(00 2).
2

---

Published as a conference paper at ICLR 2026
Dettmers et al. (023) claim that their construction leads equal utilization of the 16 reconstruction levels. However, this was already shown to be incorrect by Yoshida (2023). Furthermore, an equal probability for all codebook points is not general criterion for the optimality of a quantizer. Instead, quantization aims at rate-distortion optimality (Berger 2003). Accordingly, a codebook assigning equal probability each codebook point only optimal for uniformly distributed input data. This has been well-known for decades, most prominently through the necessary conditions for optimality that underpin Lloyd's algorithm (Lloyd 1982).
4-Bit AbnormalFloat (AF4): Yoshida )0023) analyzes the distribution of normalized network weights and performs direct minimization of the mean absolute error (MAE) to obtain block-wiss, absmax quantization codebook for normally distributed network weights, named AF4. This quantizer aims to correct oversight in the design of NF4 (Dettmers et al., 2023). which does not account for the dependence of the distribution of normalized weights on the block size. However, Yoshida's optimization method targets the minimum MAE of normalized weights MAF( (IT..)) instead of minimizing the end-to-end quantization error of the network weights MAE(wb, Qo(w,,)).
Both NF4 and AF4 contain reconstruction levels at 1, 0, and I, such that the weight of the largest absolute value in block is represented in full 16-bit precision, while the zero is represented without error. Not including these reconstruction levels leads to significantly worse MAE, mean squared error (MSE), and perplexity. We confirm this in Appendix
3
METHODS
In this section, we introduce our methods for optimizing 4-bit block-wise absmax quantization.
3.1 NOVEL BLOCK- WISE Signed ABSMAX NORMALIZATION
Instead of normalization by the absolute block maximum, as described in Section 2.1 and commonly used in existing quantization methods such NF4 and AF4, we propose a different normalization approach: block-wise signed absmax normalization. In NF4 and AF4, two reconstruction levels are intentionally constrained to ((1 -1 and a(16) 1, respectively, ensuring that the weight w/ ()/mm)) with the largest magnitude in each block & quantized without error. This empirically improves quantization performance. However, we notice that for network weights in general position, each block 6 of the normalized weights T6, contains only one of the two endpoints, either - or 1. This observation motivates the idea to normalize the weights by the signed absolute block maximum. By doing so, normalization always maps wi, ()fmaa)) 1, instead of to or -1 with equal probability. This preserves exact quantization of wn (max) while constraining only f(16) = 1, as illustrated in Fig. | The additional free reconstruction level 21) adds a degree of freedom for optimizing the non-umffrm codebook, thereby improving the achievable quantization error. Formally, in block-wise signed absmax normalization, the quantization constants wgmex from ①
selected by
w Wh.j"
max
with j = arg max |w6,l, be B.
Except for this modification, we proceed with quantization as before, using ( and (3) Note that no runtime overhead is incurred during inference or fine-tuning as decoding the quantized weights is unchanged.
3.2 NOVEL 4-BIT BLOCK- WISE OPTIMAL FLOAT (BOF4 BOF4-S)
To determine optimal quantization codebooks w.r.t. the MSE and MAE criteria, we design an expectation-maxinne mnnittiionmmiiizziitiinn (EM) algorithm based on Lloyd's algorithm (Looyd 1982). well-known algorithm for quantizer design. In each maximization step, the reconstruction levels f(() E R, l E
L (1... L}} set to the centroids of their respective Voronoi region R. ((((-11),(()) with decision boundaries {(), e (()) {0. 1, ., L), where {(0) -oo and {(L) co. However, in block-wise absmax quantization, the codebook is applied to normalized weights Tb whereas our goal is to minimize the quantization error of the quantized unnormalized weights (u(w,,) relative
3

---

Published as a conference paper at ICLR 2026
1.00
0.75
px
0.50
0.25
0.00
22
(a) BOF4 (MSE)
(b) BOF4-S (MSE)
Figure I: The blue histograms show the distributions of normalized weights px (r) for block-wise absolute absmax normalization (left) and block-wise signed absmax normalization (right) assuming Gaussian network weights. Also shown are the resulting reconstruction levels i(() ( fixed, optimized) and decision thresholds {(E) (dashed lines), after minimizing MSE( W, Q(W)) for normally distributed network weights W (Wb.s) with Wa, pw NO, 1) and block size / = 64. For absolute absmax normalization, we compute the 4-bit block-wise optimal float (BOF4, left), requiring three fixed reconstruction levels (-1, 0, 1). In contrast, when using signed normalization. we obtain BOF4-S (right), in which the largest absolute value in a block and zero are precisely represented by only two fixed reconstruction levels (0, 1), which reduces the quantization error.
to the original weights Wh.s- This introduces mismatch between the optimization target and the weight distribution directly used in Lloyd's algorithm. To resolve this, we mathematically derive an optimal solution for the centroid update. We name the resulting quantizer 4-bit block-wise optimal float (BOF4). When signed absmmxx normalization is used, we refer to it as BOF4-S. A complete derivation is provided in Appendix D resulting codebooks in Appendix major results follow here.
Let W be random variable representing continuous, zero-symmetric distribution of network weights. We further define two derived random variables X and M representing the normalized weights and absolute block maxima, respectively. Our goal is find a reconstruction level (()) that minimizes the MSE quantization eπror for those network weights that fall into fixed region R₂ after block-wise absmax normalization. By analytical optimization (Appendix D.2.I 26 we obtain the solution for the updated centroid as
MSE:
S0 m² Ex X M =m, X E Re) (M())) [Fx(x M= ((e dm
E(0) E(e-1)
a(E)
(5)
So m² PM (m)) [x M M=m)){(t-) dm
where the probability density function (PDF) px of the normalized weights, the cumulative distribu- tion function (CDF) Fx of normalized weights, and the expectation E[X M X E R22 can be computed directly from the known CDF Fw and PDF pw of the network weights, see Appendix D.2.1 (3). A detailed derivation and simplified solution for the special case of Gaussian network weights is also provided in Appendix D.2.1 (see (34)). Equation (5) can be solved by numerical integration. Alternatively, the centroid can be approximated by Monte-Care Carlo estimation based samples drawn from the distribution pw of network weights as (see Appendix D.3 64
#(f) EbEK, w Ib ΣtEK, wf
(6)
where F& E R, are the normalized weights that fall into region Re k E K2 (l....,K Ke} being their indices, and wy is the absolute block maximum upaxx of the block b containing zk.
Minimizing the quantization error of normalized weights Ta, leads worse perplexity; see Appendix
4

---

Published as a conference paper at ICLR 2026
MAE: A similar optimization can be performed for the MAE criterion, detailed in Appendix D.2.2 (59)
yielding
Par(m) (Fx()(() 1 M- M- Fx M M- ((e-))
E(O) c(e-1)) dm 0.
The zero the left-hand-sided monotonous function in (+() can be found using the bisection method in combination with numerical integration. Moreover, using the Monte-Carke method, the centroid
can be estimated as the weighted median (see Appendix D.3 (69
K Σ k=R+1
a(t) modianw (E1,.... FK, W1, "K,
wK,) ) max (x NEK:
wk
wk}.
(8)
To constrain certain reconstruction levels during Lloyd's algorithm specific values, e.g., -1, 0, 1, we initialize them with their predetermined values and skip their recomputation in each iteration. Note that for fixed weight distribution, the EM algorithm only needs be executed once, offline, and therefore adds no runtime cost during quantization of the network weights.
3.3 NOVEL BUTLIER-PRESERVIIE PRESERVING QUANTIZATION (OPQ)
Extreme outlier weights lead to suboptimal scaling of the associated block during block-wise absmax normalization. Therefore, block-wise quantization methods typically require small block sizes to limit the number of affected parameters. This increases the memory required store the quan- tization constants. To enable larger block sizes and accordingly smaller memory footprint, our autlier-preserving quantization (OPQ) approach stores outlier weights separately in bfloat16 6 and additionally uses 64-bit integer for each of them to address the outlier in the (flattened) weight tensor of the respective layer. We define outliers for each weight block independently as weights with an absolute value greater than the q-quantile of absolute block maxima after normalization of the block to unit standard deviation. Formally, a weight Wb, is classified as an outlier and only if
[6,a| > o. FF (q):
where T3 is the corrected sample standard deviation of the b-th block (see (73) in Appendix G FM () the quantile function of absolute block maxima (see (11) in Appendix D.I and q E [0. I is hyperparameter controlling the number of affected outliers. Before quantization, we exclude outliers from the tensor by replacing them with zero, so that they are not considered in the subsequent (signed) block maximum search. Note that OPQ can be combined with either BOF4 or BOF4-S. For an in-depth explanation of our OPQ design choices, see Appendix
EXPERIMENTAL SETUP
In this section, we discuss our choices for the experimental evaluation of quantization methods.
For evaluation, we apply quantization to three families of pre-trained LLMs: Llama-2 3. 1 1/3.3 3.2 2 (Dubey et al.]] 2024), Owen-2.5 (Yang et al. 2024), and Mist ra 1al---Bvv 3 (Jiang et al. 2023). By benchmarking across diverse set LLMs, we aim to demonstrate the generalizability of our method.
Quantized Models:
We evaluate our proposed BOF4 and BOF4-S approaches, optimized w.r.t. either MAE or MSE. For the optimization, we always assume Gaussian network weights to ensure comparability to the baseline methods NF4 (Dettmers et al., 2023) and AF4 (Yoshida 2023). However, we evaluate all approaches (of course) on real LLMs with weights. which are known to be only approximately Gaussian, see Appendix C For the evaluation of OPQ, we performed limited hyperparameter search, resulting in q 0.95, see Appendix G.2 The optimized BOF4 and BOF4-S codebooks are provided in Appendix
Evaluated Quantization Methods:
Fine- Fine-Tuning Method: In addition to inference with quantization, we benchmark LLMs fine-tuned
with quantization using the QLoRA method (Dettmers et al. 2023). The models are fine-tuned for instruction following using the Unnatural Instructions dataset (Honovich al.) 2023) or for code generation using the Crrrodddr-OS--nsttrere hyperparameters can be found in Appendix
dataset (Wei et al. 2024). Further details and
5

---

Published as a conference paper at ICLR 2026
MAEI (WW Q(W)) (x102)
(X10)
12
8
10
W 1SSS
7
NF4
8
NF4
AR4
ARI
(MAR)
(MSE)
BOF4S (MAR)
8OF4S (MSB)
6
25
26
27
2
29
200
211
212
25
26
27
2
29
2
2
212
block size
block size I
Figure 2: MAE (left) and MSE (right) quantization error of our quantization methods BOF4 and BOF4-S optimized for MAE (left, *) or MSE (right, o) compared to the baselines NF4 and AF4 for Gaussian network weights W (wy,; with Wh, ~ (0, 1) depending on the block size I.
Metrics: order to show that our approach incurs reduced quantization errors, we report the mean squared error (MSE) and mean absolute error (MAE) of network weights. Following prior work (Frantar et al. 2023 Lin et al., 2024, Xiao et al. 2023), we assess the language modeling abilities of quantized models based mainly on the perplexity (PPL) measured on the WikiText-2 (Merity et al. 2017 and LAMBADA (Paperno et al., 2016) datasets. The perplexity on WikiText-2 is computed using the rolling log- ligelikelihood with a maximum sequence length of 2048, it common in literature, Additionally, we evaluate the accuracy (ACC) in the NLP tasks MMLU (Hendrycks) al. 202T), ARC-Challenge (Clark al. 2018), HellaSwag (Zellers al. 2019). PIQA (Bisk et al.
2020 SIQA (Sap et al. 2019). and WinoGrande (Sakaguchi et al. 2021
RESULTS AND DISCUSSION
Quantization Error: In Fig. we compare the MAE and MSE quantization errors of our proposed BOF4 and BOF4-S quantization methods with the baselines NF4 and AF4, assuming ideally Gaussian- distributed network weights. Accordingly, the results shown are independent of any particular LLM the methods applied to. All compared quantizers constrain reconstruction levels such that 0 and the weight of the largest absolute value in a block are quantized without error, or in full 16-bit resolution, respectively. The error is computed empirically based 2 samples.
We observe that all investigated block-wise quantizers show increasing MAE/ MSE with increasing block size I. This is expected, larger block sizes will have larger block maxima, which in turn increases average error for the many non-maximum weights in the block. All of our proposed methods B-F,(-S), optimized w.r.t. both MAE and MSE, are equal baselines NF4 and AF4. Note that AF4 (Yoshida) 2023) was presented in some MAE-optimized form, which explains its poor MSE performance for medium- or large-sized blocks. Our signed normalization method BOF4-S achieves lower MAE and MSE scores than any other investigated quantization approach.
better than each of the two
Quantization Error and Perplexity: Tab. T shows comparison of the MAE and MSE quantization errors, well as perplexity (PPL), evaluated on the weights of three pre-trained LLMs: Llama-3 1.mma-3... -3.1 8B, Qwen 2 7B, and Mistra. 7B. Results for additional (smaller) models are provided in Appendix 0 Note that intention is not compare PPL between the various LLMs, but rather between the various quantizer options.
We observe that our basic BOF4 approaches equal to lower in quantization than the baselines NF4 and AF4 when optimized for the particular metric MAE/ MSE. The respective methods with signed normalization (BOF4-S) clearly outperform the non-signed BOF4 approaches in all cases,

---

Published as a conference paper at ICLR 2026
Table Quantization error (MAE and MSE) and perplexity (PPL) on WikiText-2 of quantization methods applied to the network weights of three LLMs with block size / = 64. Best result in each column in bold, second best underlined.
Llama-3. -31 8B
Qwen-2.5 7B
Mistral 7B
MAE 1e-3
MSE
PPL 4
MAE
MSE J
PPL 4
MAEJ
MSE
PPL J
1e-6
le-4
le-8
1e-3
1e-6
NF4
0.977
1.637
8.53
1.202
2.391
9.89
2.256
8.439
8.90
AF4
1.006
1.762
8.51
1.234
2.562
9.91
2.324
9.085
8.90
BOF4 (MAE)
0.976
1.621
8.52
1.202
2.370
9.89
2.256
8.360
8.90
BOF4 (MSE)
0.994
1.566
8.51
1.228
2.310
9.94
2.296
8.075
8.89
BOF4-S (MAE)
0.936
1.508
8.49
1.152
2.204
9.87
2.162
7.777
8.90
+ OPQ
0.918
1,457
8.46
1.121
2.101
9.82
2.121
7.514
8.89
BOF4-S (MSE)
0.954
L.411
8.46
1.179
2.126
9.88
2.204
7,430
8.88
+ OPQ
0.932
1.367
8.43
1.140
1.981
9.83
2.153
7.052
8.87
9.2
NF4
9.2
NF4
AF4
9.0
IMAE)
9.0
BOF4-S (MAF) +OPQ
BOF4-5 (MSF)
.OPQ
8.8
8.8
8.6
8.6
8.4
8.4
26
26
27
28
29
20
211
212
25
26
27
28
29
200
211
212
block size I
block size I
Figure 3: Perplexity of Li amaam-3. 8B on WikiText-2 after quantization with NF4, AF4, and our BOF4-S optimized w.r.t. MAE (left, *) or MSE (right, o) for different block sizes I, with and without outierrpreserrvng quantization (OPQ, dashed line).
and accordingly, the baselines NF4 and AF4 as well. We emphasize that MAE- and MSF-optimized BOF4(-S) schemes show the lowest quantization errors for their respective optimization metric. This empirically confirms derived centroid update rules (7) and 5). along with the underlying Gaussian distribution assumption of the LLM network weights. Analyzing perplexity, BOF4-S is equal to (in single case) or better than each of the baselines, indicating that the lower quantization error also pays off in terms of an improved language modeling accuracy. Our proposed audiee-prresevingg quantization (OPQ) variant provides a further consistent performance improvement, it lowers MAE and MSE quantization errors and perplexity in all cases.
Comparative Effect MAE and MSE Optimization: Tab. also shows the language modeling perplexity of the quantization methods on the WikiText-2 dataset (Merity 2017). This allows us to compare the effectiveness of BOF4(-S) optimized for MAE and MSE. We report both error metrics (MAE, MSE) for the quantized model weights w.r.t. the original model weights.
We observe the tendency of MSF-optimized BOF4(-S) methods to yield better (i.e., lower) perplexity than the MAE-optimized version, with only Qwen-2.5 2.5 7B being exception with 0.01 point perplexity advantage for MAE optimization. Overall, the best-performing of our proposed schemes is BOF4-S (MSE) with OPQ, as it ranks either first second among all other investigated methods in each metric.
Fig. 3 shows the perplexity of Llama-3 8B on the WikiText-2 (Merity et al. 2017) and LAMBADA Paperno et al., (016) datasets after quantization with NF4, AF4, and our BOF4-S optimized w.r.t. MAE (left) and MSE (right). Furthermore, Fig. reports the effect of utilizing the proposed outlier-

---

Published as a conference paper at ICLR 2026
Table Inference results of 4-bit scalar quantization methods evaluated using multiple LLMs with block size I = 64. The evaluated metrics are the perplexity on the WikiText-2 and LAMBADA dataset, and the accuracy on the MMLU (few-shot), ARC-Challenge, HellaSwag, PIQA, SIQA, and WinoGrande benchmarks. Best result in each column in bold, second best underlined, BF16 excluded.
Model
Quantizer
WikiTex12 PPL L
Lambeda PPL
MMLU
ARC-C ACC 1
HelloSwag ACC1
PIQA ACC1
SIQA ACC
WinoGrando ACC1
NAV
ACC 1
ACC 1
BF16
10.12
4.90
54.0
55.3
76.7
47.2
69.0
35.7
NF4
10.72
5.45
52.3
41.0
54.4
76.3
47.0
68.3
34.4
L 1.iama33. 3.2
AF4
10.74
5.51
52.8
40.5
54.4
76.6
47.4
69.3
35.0
3B
BOF4 (MSE)
10.73
5.35
52.6
42.1
54.0
76.7
46.4
68.8
34.8
OPQ
10.67
5.17
52.9
42.1
54.1
76.9
46.4
34.8
BOF4-S (MSE)
10.67
5.32
52.6
42.0
54.3
76.1
46.3
68.5
34.5
OPQ
10.64
5.25
52.5
41.8
54,2
76.2
46.5
69,5
34.9
BF16
12.42
5.91
65.1
44.6
55.0
78.1
49.6
68.5
39.5
NF4
12.36
7.16
63.0
43.1
53.6
77.7
50.8
67.2
38,2
Twen-2.5
AF4
13.08
6.82
63.3
43.5
54.2
78.1
50.6
68.5
38.9
3B
BOF4 (MSE)
12.46
6.84
53.8
77.7
39.2
OPQ
12.48
6.90
63.1
54.1
77.5
67.6
38.9
BOF4-S (MSF)
12.50
6.53
63.5
46.5
53.8
77.3
50.0
68.2
39.1
OPQ
12.35
6.43
63.5
46.6
54.0
77.5
50.8
69.1
39.7
preserving quantization (OPQ) in combination with BOF4-S, A corresponding figure including our BOF4 is given in Fig. 13 in Appendix
Fig. (left) shows that our MAE-optimized BOF4-S methods reveal lower PPL than both baselines up to block sizes of I S 29, The MAE-optimized baseline AF4 shows some strengths for very large block sizes I 2 2 which, however,
practically relevant.. When comparing Fig. F (right), we observe that our MSE-optimized BOF4-S methods generally achieve lower perplexity than both baselines and also than their MAE-optimized counterparts the left. This trend becomes more pronounced with increasing block size The overall better performance of our MSF-optimized BOF4(-S) approaches leads us focus on these in the following experiments.
Comparison to NF4 and AF4 for Inference: Tab. shows the perplexity and accuracy of various quantized LLMs in the 3B regime common NLP benchmarks. In addition, normalized average accuracy (NAV ACC) is computed that accounts for the chance-level accuracy in each benchmark; for details about this metric, Appendix K Results for smaller and larger models are provided in Appendix J Furthermore in Appendix [[> we provide additional inference evaluations demonstrating an application of our methods to baarrattidddttabaasseeaa GPTQ quantization (Frantar al.. 2023).
Analyzing accuracy over the various benchmarks reveals that rank orders of models can be quite different in different benchmarks. Accordingly, such accuracy results should be interpreted with care. Our normalized average accuracy metric (last column) helps in identifying overall trends. For the Llama-3.
model we see only slightly varying NAV ACC results, with AF4 and our BOF4-S +OPQ approach being close-by on first and second rank. On Qwen 2 3B, our favored BOF4-S +OPQ method has the overall best NAV ACC, even outperforming the BF16 reference. As we hardly claim to be better than 16 bit weight representation, metric in general. Among the two benchmarks reporting perplexity, our proposed BOF4-S +OPQ method ranks three times first and time second, outperforming both baselines. NF4 and AF4.. In addition, OPQ incurs only a small runtime overhead during inference, as shown in Appendix G.3
note once more the variance in the accuracy
Note that the perplexity advantage of BOF4-S (+OPQ) achieved despite the only partly valid Gaussian weight assumption in the design of our codebook. Our method also supports codebook optimization w.r.t. better-fiting distributions, which might even further improve performance.
Fine- Fine-Tuning with Quantization: Tables and 4 show the results for quantized fine-tuning using QLoRA (Dettmers et al. 2023) with various quantizers. Llama-3.2 3B fine-tuned for instruction following and code generation, respectively, and evaluated on corresponding task-specific benchmarks. For comparison, we apply LoRA fine-tuning (Hu al. 2022) to the original, unquantized weights in
8

---

Published as a conference paper at ICLR 2026
Table 3: Prompt-level and instruction-level accuracy (%) on IFEval after fine-tuning Llama-3 3.2 3B for instruction following us-
Table Accuracy (%) on the HumanEval+ and MBPP+ benchmarks after fine-tuning L. Llama-3.2 3B for code generation using
ing 4-bit quantization with block size I = 64.
bit quantization with block size I = 64.
Prompt-level ACC 7
Instr-level ACC
AVG ACC
MBPP+ ACC
HumanEval+ ACC 1
AVG ACC 1
Base Model
21.1
33.6
27.3
Base Model
34.9
17.1
26.0
BF16
23.5
34.2
28.8
BFI6
37.8
30.5
34.2
NF4
24.4
35.0
29.7
NF4
34.1
24.4
29.3
AF4
23.3
34.1
28.7
AF4
32.8
23.2
28.0
BOF4 (MSE)
26.8
36.5
31.6
BOF4 (MSE)
34.4
24.4
29.4
+OPQ
25.0
34.8
29.9
+OPQ
35.4
29.9
BOF4-S (MSE)
24.4
35.1
29.8
BOF4-S (MSE)
35.7
26.2
31.0
OPQ
25.0
35.0
30.0
OPQ
36.5
27.4
32.0
bfloat 6 representation (BF16). We find fine-tuning be stable with all quantization methods. In addition to the task-specific accuracy metrics in Tables 3 and 4 we also report the average accuracy (AVG ACC).
From a bird's-eye view over both tasks (tables) we observe the strength of BOF4-S +OPQ being confirmed: For instruction following, ranks second in AVG ACC, and for code generation, ranks first-in both cases being better than NF4 and AF4.
Table interestingly, reports our BOF4 approach as by far the best for instruction following. The OPQ variant for this particular downstream task is not the best. Accordingly, we keep in mind that for fine-tuning towards specific task might be advised to investigate which of our four proposed MSE-optimized BOF4 quantizers (signed vs. unsigned, with without OPQ) performs best.
In Table 4 we observe for the code generation task that the previously best BOF4 is still equal to or better than the NF4 and AF4 basclines. The other three of our BOF4 variants are, however, even better in this case, with BOF4-S +OPQ being clearly ahead of investigated approaches. The second rank is clearly taken by BOF4-S without OPQ. This again confirms the recommendation that BOF4-bad based quantization of fine-tuned LLMs is best done after a small ablation study among the four MSE-optimized BOF4 quantizers. For limitations of our work, Appendix
CONCLUSIONS
In this paper, analyzed block-wise absmax quantization for large language models (LLMs) and derived an expectation-maxiizzaiiio algorithm to minimize the quantization error. The resulting family of quantizers, termed 4-bit block-wise optimal float (BOF4), reduces the weight quantization eиror over previously published biock-wise absmax quantizers such as NF4 (Dettmers et al. 2023) and AF4 (Yoshida) 2023). We also presented an improvement the normalization technique by normalizing blocks of weights using their signed absolute maximum rather than the absolute maximum, which further reduces the quantization error and empirically mitigates the negative effect of quantization perplexity. Our experimental study confirmed the importance of precisely representing zero and outlier network weights, and found that optimization w.r.t. the mean squared error (MSE) criterion results in lower perplexity compared to mean absolute error (MAE) optimization. Finally, we introduced outlier preservieeeeeeeeee quantization (OPQ), a mixed precision strategy for block- wise absmax quantization, which yields significant perplexity advantage, especially at larger block sizes. We find that our methods can outperform NF4 and AF4 not only for inference, but also when used for fine-tuning with quantization, achieving higher accuracy the target tasks.
Overall, our proposed methods can enable improved fine-tuning and inference for LLMs on consumer- grade hardware by boosting performance without increasing the memory footprint, thereby facilitating broader participation in both the scientific investigation and the application of LLMs.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
For the sake of reproducibility, we make the source code for fine-tuning and inference evaluations, as well as codebook optimization, available at
com/inssamm//oo/ In Appendix [ we provide some of the quantization codebooks resulting from our optimization algorithm. Additional codebooks are also provided in the GitHub repository. For our theoretical results, we state all assumptions and provide a detailed derivation Appendix
ACKNOWLEDGMENTS
This work was partially funded by the German Federal Ministry for Research, Technology and Aeronautics (Bundesministerium für Forschung, Technologie und Raumfahrt, BMFTR) under the KI4ALL project (funding code: 16DHBKI055).
REFERENCES
Saleh Ashkboos, Amirkeivan Mohtashami. Maximilian L. Croci, et al. QuaRot: Outlier-Fred 4-Bit Inference in Rotated LLMs. Proc. of NeurlPS, pp. 100213-100240, Vancouver, BC, Canada, December 2024.
Toby Berger. inte-Distottion Theory. Wiley, 2003.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, et al. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proc. of AIlI, pp. 7432-7439, New York, NY, USA, February 2020.
Richard L. Burden and J. Douglas Faires. Numerical Analysis. Brooks/Cole, 2010.
Peter Clark, Isaac Cowhey, Oren Etzioni, et al. Think You Have Solved Question Answering? Try
ARC, the A12 Reasoning Challenge. arXiv, March 2018.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit Optimizers via Block-wise Quantization. Proc. of ICLR, pp- 1-19, virtual, April 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLORA: Efficient Finetuning of Quantized LLMs. In Proc. of NIPS, pp. 10088-10115, New Orleans, LA, USA, December 2023.
Jordan Dotzel, Yuzong Chen, Bahaa Koth, et al. Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs. In Proc. of ICML, pp. 11573-11591, Vienna, Austria, July 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 Herd of Models. arXiv, July 2024. arXiv: 2407.21783.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In Proc. of ICLR, pp- 1-16, Kigali, Rwanda, May 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, et al. The Language Model Evaluation Harness, December 2024. JtR/hhps//dooi..r/11..333 02000000
Dan Hendrycks, Collin Burns, Steven Basart, et al. Measuring Massive Multitask Language Under- standing. In Proc. of ICIR, pp. 1-27, virtual, May 2021.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. In Proc. ACL, 14409-14428, Toronto, ON, Canada, July 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, et al. LoRA: Low-Rank Adaptation of Large Language Models. Proc. ICLR. pp. 1-13, virtual, April 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7B. arXiv, October 2023. arXiv: 2310.06825.
10

---

Published as a conference paper at ICLR 2026
Ji Lin, Jiaming Tang, Haotian Tang, al. AWQ: Petivation-Awae Weight Quantization for On- Device LLM Compression and Acceleration. In Proc. of MLSys, pp. 87-100, Santa Clara, CA, USA, May 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Proc. of NewrIPS, pp. 21558-21572, New Orleans, LA, United States, December 2023.
Zechun Liu, Changsheng Zhao, Igor Fedorov, et al. SpinQuant: LLM Quantization with Learned Rotations. arXiv, May 2024. arXiv: 2405.16406.
Stuart P. Lloyd. Least Squares Quantization in PCM. IEEE Transactions on Information Theory, 28 (2):129-137, March 1982.
Iya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In Proc. of ICLR, pp. 1-10, New Orleans, LA, USA, May 2019.
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, et al. HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem. In Proc, of NAACI, pp. 10857-10886, Albuquerque, NM, United States, April 2025.
Stephen Merity, Caiming Xiong, James Bradbury, et al. Pointer Sentinel Mixture Models. In Proc. of ICLR, pp. 1-15, Toulon, France, April 2017.
D. Owen. A Table Normal Integrals. Communications in Statistics -Simulation and Computation, 944:3899419, 1980.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, et al. The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context. In Proc. of pp. 1525-1534, Berlin, Germany, August 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67.jam 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, al. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9)999-106, August 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, et al. Social IQa: Commonsense Reasoning about Social Interactions. In Proc. of EMNLP-IJCNIPP, pp- 4463-4473, Hong Kong, China, November 2019.
Mart van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, et al. GPTVQ: The Blessing of Dimensionality for LLM Quantization. arXiv,, February 2024. arXiv: 2402.15319.
Yuxiang Wei, Zhe Wang, Jiawei Liu, et al. Magicoder: Empowering Code Generation with OSS- Vienna, Austria, July 2024.
INSTRUCT. In Proc. of ICML, pp.
Guangxuan Xiao, Ji Lin, Mickael Seznec, et al. SmoothQuant: Accurate and Efficient Post Post-Training Quantization for Large Language Models. In Proc. of ICML, pp. 38087-38099, Honolulu, HI, USA, July 2023.
An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 Technical Report. arXiv, December 2024. arXiv: 2412.15115,
Davis Yoshida. NF4 Isn't Information Theoretically Optimal (and That's Good), arXiv, 6 2023. arXiv: 2306.06965.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, et al. HellaSwag: Can a Machine Really Finish Your Sentence? In Proc. of ACL, pp. 4791-4800, Florence, Italy, July 2019.
11

---

Published as a conference paper at ICLR 2026
APPENDIX
A LIMITATIONS
Our evaluation focuses on comparisons with data-free quantization techniques and does not include post-training quantization (PTQ) methods that rely on calibration data. While quantization based on calibration data typically achieves better accuracy, we believe that data-free techniques are valuable because they are significantly more efficient in terms of time and memory required for quantizing the weights. Note that our contributions are compatible with and can be effectively applied to calibration- data-based PTQ methods, demonstrated in Appendix Furthermore, the accuracy results of the utilized language modeling benchmarks may not be sensitive enough to reflect minor differences between quantization methods. We partially mitigate this issue by computing normalized average accuracy score and relying more on perplexity as our primary metric. While we choose to evaluate our method to optimize quantization under the assumption of Gaussian weight distribution ensure a fair comparison with NF4 and AF4, our algorithm does not rely on particular distribution. When applying double quantization as proposed by Dettmers al. (2023), i.e., additionally quantizing the quantization constants beyond BF16, signed normalization would require an extra bit per block to encode the sign, raising memory consumption from 4. 127 to 143 bits per weight. Without this, the improvement in the quantization error by using BOF4-S may be slightly diminished, since the input range of the quantizer for the quantization constants is doubled.
In future work, we want combine our approach with Hadamard transformations of network weights used by recent quantization methods to ensure a Gaussian weight distribution (Ashkboos et al. 2024 Malinovskii al. 2025). Furthermore, we would like investigate in more detail how contributions are best applied to calibration-data based PTQ.
B ABLATION ON CONSTRAINED (I.E., FIXED) RECONSTRUCTION LEVELS
In Tab. we evaluate the importance of precisely representing zero weights and absolute block maxima. We use the term "precise" in this context for error-free representation of zero weight and for a 16-bit representation of absolute block maxima. We measure the perplexity on WikiText-2 (Merity et al. 2017) of Llama 3. 1B8B quantized with BOF4 for all four possible combinations of fixed (i.e., constrained) reconstruction levels (() from O and 1
Table 5: The quantization error (MAE and MSE) and perplexity (PPL) WikiText-2 of BOF4 with block size using different combinations of fixed reconstruction levels applied to network weights of 1lama-3. 1-B.. Best result in each column in bold.
Constrained
MAE (1e-1)
MSE L (1e-6)
PPL |
reconstruction levels
Ø
9.881
1.506
8.81
{0)
9,904
1.516
8.57
-1--1 (0, -1
9.914
1.555
8.78
9.936
1.566
8.51
We observe that fixing all (-1, 0, and 1) yields the best performance w.r.t. perplexity, even though the additional constraints on the codebook inevitably increase the quantization error. This confirms that the design choice of NF4 (Dettmers et al. 2023) and AF4 (Yoshida 2023) to include these values as reconstruction levels is sound.
C WEIGHT DISTRIBUTION OF LARGE LANGUAGE MODELS
Dettmers et al. (0233) analyze the distribution of weight rows in a 7B LLaMa model to support the assumption of Gaussian weight distribution for quantization. They perform Shapiro- Shapiro-Wilk Wilk test on the rows and find that only 7.5% are non- Geussian-distribenn
We conduct a similar analysis for 1.1 a.lama-3.. 3B and Qwen-2.5 3B evaluate whether this assumption holds for the models
12

---

Published as a conference paper at ICLR 2026
used in this work. Instead of entire weight rows, use blocks, since these the units to which the quantizer is applied, and because the number of elements per row often exceeds the sample size at which the Shapiro-Wii Wilk test produces reliable p-values. Using block size 1 = 64 and significance for the p-value, the test only identifies 6.9% and 10.5% of blocks as non-Gaussian filtered prior the test, as in OPQ (according to (9), the fraction of non-Gaussian blocks decreases further to 4.6% and 6.5%, respectively. This indicates that assuming Gaussian-distributed weights for codebook optimization reasonable for large majority of the quantized blocks.
threshold of 0.05
for I.1 11aam33. 8B and Qwen-2.5 83, respectively. When outliers
D FULL DERIVATION OF THE CORRECT OPTIMAL RECONSTRUCTION LEVELS
D.I DISTRIBUTION OF NORMALIZED WEIGHTS
To derive an optimized code for quantization, we first characterize the distribution of normalized weights. Yoshida performs similar analysis, assuming zero-mean, unit-variance normally disttrbbudd network weights Wh. ~ N(0, 1) (Yohhida 2023). In the following, we generalize this analysis to weights distributed as any symmetric, zero-mean probability distribution. The weights Wh, considered i.i.d. samples from random variable W with the probability density function (PDF) pw and cumulative distribution function (CDF) Fw. Similarly, X is random variable describing the distribution of the normalized weights with PDF px and CDF Fx. A third random variable M describes the distribution of absolute or signed block maxima w Fig. (4shows an estimation of Px(r (r = 25,;) in case Wh, Pw (O0 1). We observe that the distribution of normalized weights concentrates around zero with increasing block size Furthermore, the discrete fraction representing the absolute block maxima at the edges (-1 and 1) is inversely proportional to the block size T, as we
will formalize later in (16).
PDF for Fixed Absolute Block Maximum: First, we analyze the distribution of normalized weights X for fixed absolute block maximum M = m = witux (I). The PDF px(x) (r) with x E R assigns a probability mass of 27 to both x -1 and x = +1 since in the non-degenerated case there is exactly one value of maximum magnitude in each block of 1 weights. The remaining probability mass of I-1 forms continuous, non-uni non-uniform probability distribution on the interval (-1,
For a fixed absolute block maximum m =w E R the continuous portion of the CDF of X is
Foout (x M= = P[x <a/ X 1, M = mm] =[[ maal [W| m
P(W mar A W m]
P[[ ml
Fw (m (w)m))
(m) Fww(-m)
FW (mx),
(10)
for x E [0, 1}, where Fw. is the CDF Fw of weights truncated to the interval [-m, mmm.m....
Distribution of Absolute Maxima: We continue with the distribution the absolute block maxima, defined by the random variable M. First, due to the statistical independence of weights wh,i within block, the CDF of M is given by
(m) = P[we,, S m, |/0,2| S m...., |wb.r| S m]
II Flw((m), I= = 21,
iEI
= (m),,
(LI)
13

---

Published as a conference paper at ICLR 2026
block size
64
block size / 1024
1.0
Px(r))
0.5
0.0
1.0
-0.5
0.0
0.5
1.0
-1.0
00.5
0.0
0.5
1.0
Figure 4: Empiric estimation of the PDF px. resulting from block-wise absmax normalization, in case of Gaussian network weights based on 2 samples for different block sizes J.
1.00
abs. norm.
signed norm.
0.75
Fx
0.50
0.25
0.00
1..
-0.5
0.0
0.5
1.0
Figure 5: Example CDF (z) for absolute and signed block-wise absmax normalized Gaussian network weights vb, and block size 8.
where Fl the CDF of [W]. Taking the derivative of FM using the chain rule, we obtain the PDF of M:
d dmm
PM((m)
FM (m)
d (m)
(l)((m) dm
d (2Fw(m) 1)
F'rl (m) dm
21 Flwl (m) pw(m) (m)
(12)
In the derivation of (12). we use the fact that the PDF is the derivative of the CDF, and we use the equality
Fw (m)) = Pl-m S W < m]
FFw (1 Fw (m))
2F (m) - 1.
(13)
exploiting the symmetry of pw
14

---

Published as a conference paper at ICLR 2026
CDF of the Normalized Weights: Now, the continuous part of the CDF Fx(x), - < 1, can be calculated using the law of total probability by integrating over all possible values of the block-wise absolute maximum M
and weighting each block-maximum-depennee the corresponding probability density pas (m) as follows:
CDF with
F Fx ()) = P P S [X| <]
pM(m). Fx M -=m) dm.
(14)
By substituting the terms PM (m) and Fx MM using 12 and (10 obtain
[ (m)pw (m) Fmr (ma) dm.
FX (x) 21
(15)
Considering that Fx,com contains the fraction 7 of the probability mass, whereas +1 and -1 each occur with probability Z7. the CDF of X can be characterized as
0, 1 +
if < <1 (z),
Fx(x)=
27 PX"
- x <1 if 2
(16)
1,
For signed wise absmax quantization, the continuous part of the distribution remains unchanged due to the symmetry of pw w.r.t zero. Only the discrete probability mass of 1/1 is now allocated entirely to X resulting in the CDF
0. (z),
if if-1
Fx(x) = 4 (
(17)
1.
if
The CDF (x) for an example Gaussian weight distribution Wh, pw (0, 1) with block size = 8 for both absolute and signed block-wise absmax quantization is shown Fig. 5 We observe that, in the case of absolute block-wiss absmax normalization, the CDF Fx exhibits two discontinuities at æ = and x = 1, whereas, for signed absmax normalization, there is only one discontinuity x
D.2 CENTROIDS BLOCK- WISE NORMALIZED WEIGHTS
In the following, we show how Lloyd's algorithm Lloyd 1982) can be modified to minimize the MSE(W, Q(W)) or MAE(W, Q(W)) quantization eгrоr when using block-wise absmax quantiza- tion, where Q I - R denotes the overall block-wise absmax quantization function. Lloyd's algorithm is applied to the normalized weights distributed according to X, whereas the quantization error should be minimized end-to-end for the distribution of original network weights W. We demon- strate how the centroid criterion for a reconstruction level (()) of region R2 = [<(E-1), can be reformulated to account for this discrepancy. Specifically, we derive mathematical formula enabling the direct computation of the updated reconstruction level (()) for any continuous distribution of network weights that is symmetrical w.r.t. zero and has known PDF pw and CDF Fw. Furthermore, we consider the special case of Gaussian weights pw = (0, 1), which allows further simplification in the case of the MSE criterion.
should also be noted that the assignment of regions according to the nearest neighbor criterion remains unchanged from Lloyd's algorithm. The proof that the nearest neighbor criterion is still necessary condition for optimality, even when applied to normalized weights, is trivial. Therefore, showing that our modified centroid criterion is second necessary condition for optimality is sufficient to prove the local optimality of any solution to which our algorithm converges.
D.2.1 MSE OPTIMIZATION
First, we minimize the MSE quantization error MSE(W.Q(W)) = Ew [( Q(W))²] with Ew [ being the expectation w.r.t. the weights W.
15

---

Published as a conference paper at ICLR 2026
General Centroid Criterion: Our goal is to find reconstruction level #(() that minimizes the MSE quantization error for normalized weights that fall into region Rx:
+(E) - arg min MSE, (W, Q(W)) = arg min [ Q(W)) X E R.). SER
(18)
&EB
where å represents a candidate value of the reconstruction level utilized for normalized weights falling into region R₂ within Q, (W, is the MSE quantization error of weights that fall into region R₂ after normalization.
Next, enable analytical minimization of the reconstruction error, we use the law of total expectation
E[A] = Eg A
B]
B[A B p(b) db.
(19)
express the expectation 18) as integral over expectations conditioned fixed block maximum
M = m:
Ew [WW Q(W))² X E RR]
PM (m X R2) Ew [W Q(W)) M=m, E Re] dm.
(20)
This reformulation allows us to express the expectation terms of the random variable X and the sought reconstruction level i:
Ew [W - Q(W)) M m,, X E R₂]
Ex (m X 2² M - m, X E Rd
m² Ex [[X = M-m, X R.]
(21)
Substituting 21 into 20).
obtain the formulation of an MSE-optimal reconstruction level
a(()
arg min EER
pM X RR) m² Ex [[X - +)2 M -m, X RR] dm..
(22)
To find the optimal reconstruction level 2, we set the derivative w.r.t. & equal to zero:
d
pm(m X Re) m² 2 Ex [(X 2) M M X E Rr] dm
di
J
m² 2 - Ex|X = X E Rol): PM(mm X E Rs) dm = 0
(23)
Rearranging for yields
=(() -= Jo m² Ex[X M- X Re] PM (m |
X E Re) dm
(24)
So m PM X E Re) dm
To compute this, we must express all quantities in terms of the known PDF pw and CDF Fw To accomplish this, we analyze the PDF (m X E Re) of the weight maximum conditioned on the region R. Using Bayes' theorem, we can express this as
(m E R2) = PM (m)) P(X E RR M=m]
P[X E Re]
(m)) (Fx(E(()
M M=m) - Ex(E((-1) M=m))
P[X E Re]
(25)
The PDF of the weight maximum (m))m) is known from 12 and the CDF Fx (x M= for the continuous part of the distribution known from (OO The special case of the non-continuous outermost regions is considered separately. With this and (24). the updated reconstruction level can be expressed as (major analytical result for MSE-optimized codebook reconstruction levels)
2(C) = 10 m² Ex[X = X E RR) PM [Fx(x M M=m)] E(E)
dm
E(e-1)
(26)
fo m² PM (m) [Fx (x M ==m))
((e-1) dm
E(P)
16

---

Published as a conference paper at ICLR 2026
Here, the notation (xx M M(-)) (x(((() M= Fx(E(e-1)) M =)). Using the known PDF pw and CDF Fw. PM (m) is obtained by 12) while Fx M - m) is
Elt)
obtained by (10). Hence, only the conditional expectation Ex [X] M = m, X E Re] of normalized weights X requires further analysis.
Expectation of Normalized Weights: To compute the optimal reconstruction level s()) according to (6). we require a method to determine the expected value of the normalized weights, conditioned on the absolute block maximum M and the region R, [s(e- 1),{())) First, we analyze Ex [X] M X E Re] under the assumption that the region is contained in the continuous part of the distribution R. (-1, The conditional mean is given by
JR. xpx (z
M m) dr
Ex[X M m, X Rd]
JR. Px M mm) dr
SR. xpx (2r M M=
(27)
(x((( M=m) Fx (x(((((())
We know from (10) that Fx M) for the continuous part of the distribution Rş E (- -111) 1) can be expressed as Fw.-m.. (mx) using the CDF of W truncated to the interval [-m, ml. The derivative w.r.t. x represents the corresponding PDF
d
px px -=)
Fw. (ma) mmpm-m. (mxx), dx
(28)
where [) for a, b E R and a < b is the PDF pw truncated to the interval [a, 6). formally defined as
PW (w) = Pw (w) I(a.0)(1))
(29)
(w)(b) Fw (a)
where [/..8= R - [0, is the indicator function with [|0.5] (w) = 1 a w < Note that
our assumption R2 C (-1,1) implies -1 < X < 1, and therefore 1 m (mz) = Using this observation and (29). we get
SR. m x PW m.... (mz) dx
X X Re] Ex X M-
(30)
Fwi (mE(1)) Fw- (mg(e-1)) This expression can be simplified, since pW. (mx) and Fw,- (max), shown in 10 share the common denominator Fw (m) - Fw (-m), yielding (to be used in (6)
SR. m x Pw (mx) dr Ex(x M= X E RR] Fm((m(()) - Fw (m
(31)
Simplified Solution for Gaussian Network Weights: In the following, we adopt the common assumption that the network weights are distributed according to zero-mean unit-variance Gaussian pw(w) = g(w)= == / (w; 0, 1), enabling further simplification. Furthermore, we denote the CDF of the zero-mean unit-variance Gaussian G(w) = Fw (w). Additionally, we utilize the solution to
the indefinite integral (Owen 1980, equation (101))
zg(mx)
m 9 (mx) + C,
(32)
with C E IR. By applying this solution to (31 we obtain
- Ex[X M= X E R.] m(G(mE(() G(mE((-1))
(33)
Substituting this into the centroid criterion
26 results in
a(f) So m (C(m()))))
-f Pas (m) (x M= E(t)
dm
1(
(34)
So m² PPM(m) [Fx M M=m)
E(E-1) ] dm
((8)
Lo0 m [9(mz)) ((1)
((t-1) (2G(m) .g(mm dm
x(()
(35)
Jo m² [G(m)) E(E)
- - g(m) dm.
<(E-1)
for R₂ E (-1,1) 1). The integrals can be solved using numerical integration.
17

---

Published as a conference paper at ICLR 2026
Expectation of the Outermost Reconstruction Levels: Next, we consider the edge case, where
the centroid of the outermost region is to be computed, c.g., -1 E(e-1) < S E(E), for the
rightmost region. In this case, decompose the overall expectation in (26 follows
Ex[X M=m,X Ry]
PlE(e-1) s X <1 M m, X E Rd E[X M=m, + P|X =1 M m, E Rd]
X <]
(36)
The expectation [ M - X E R₂] is decomposed into the expectation over the continuous m, their respective probability. For the continuous part,
part [x[X M = Ele -1 - X < 1) and the complementary case X = 1, each weighted by
compute the expectation with (33). The
fraction of the probability mass in R. that allocated to = 1 can computed as
P M m, X Re]
P 1, E R4 M=m]
PIX E R₂ M=m)
P M M=m]
P M M=m]
P[X=1| M=m]
(37)
P(g((-1) < X <1 M M=m] P[X=1] M-m]
For block wise absmax normalization, without signed absmax. a fraction of 27 of the probability mass is allocated to 1, whereas the 1 is contained in the continuous part of the distribution. We can use the CDF Fx (r M = m) = Fwi-m.,. (mx) to find the fraction of the probability mass contained in the continuous part of R.:
P[g(e-1) < X <1 M mm
= P|-
M =m] P[g(e- < < - < -
1-1
(1 oon (E(6-1) (
M m=m)
1-1 (-E(--1) M M=m),
(38)
utilizing the symmetry of ggont( w.r.t. in the final step. Concerning 37).
obtain
P[X=1
M m. X E R2
27
7-1 7-1 FronE (E(E 1) M M=m)) + 21
1
(39)
2(1 1) (- -C((E- M M=m))+1
where 1 (x M = m) = Fw (mx) is the CDF of X on the continuous part of the distribution derived in (TO). The derivation for the leftmost reconstruction level symmetric. When using signed absmax normalization, probability of & is assigned to 1, and we obtain instead
P[X = M = m, X RR]
1
(40)
(I - 1) Fxont (-E(E-1) M=m)) +1'
for the rightmost reconstruction level, while the leftmost reconstruction level requires no special
treatment.
Additionally, for the centroid according to we require Fx (r M=m). For the continuous part of the distribution (-1 1 < X < 1) the solution is provided by (IO). Accounting for the probability mass fraction distributed to -1 and 1, we obtain for absolute block-wise absmax normalization:
0.
if < -1
Fx(x M -=M)-
coaa (x M M=m),
i <1
(41)
1,
1
18

---

Published as a conference paper at ICLR 2026
and for signed block-wist absmax normalization:
0,
if <-1 - Fx (x M=m),
Fx M= M=m)
ff-1sa<l
(42)
1,
1
It should also be noted that the computation for the continuous part of the distribution with CDF "x =mm) is identical for both signed and absolute block-wise absmax normalization. This is because the distribution of normalized weights only differs in the probability each of the endpoints -1 and 1. Moreover, these edge cases are typically not evaluated, since the outermost reconstruction levels are usually constrained and and are not updated during Lloyd's algorithm.
assigned to
D.2.2 MAE OPTIMIZATION
We derive condition for the centroid that minimizes the MAE quantization error MAE(W, Q(W)).
We show that the condition minimizes the MAE by reducing the problem to the well-known fact that the median minimizes the mean absolute deviation from set of points. Beginning analogously to Section D we arrive at the following criterion for optimality:
((f) = arg min ZER
m Ex [[X - + =mm, X pM(mm) X dm.
(43)
Thus, we define the objective function we aim to minimize as
g
m Ex||x al M- X X Par(m) EE)2) dm
(44)
Using the definition of the expected value, we have
Ex ||X a) M= XERo = /
px(x M=m, . 18--00 z/dz.
(45)
so that
g 1
Pas XER2) px(x M=m, X | r/ dm.
(46)
R2
We swap the order of integration:
g(t) = 1 R2 [ m Pm(m XER;) px(x M=m, X E dm 1 dr.
(47)
Now, we define a "re-weighted" PDF
p()) 00 PMd (m X (x ER) dm
(48)
So m Pas (m X X dm
with which the objective function can be rewritten as
g 100 m (mr(m) X
[ p(x) 1/ - x dr.
(49)
Since search for arg ming (()), ignore the first integral in 49 it is only constant factor, and our objective function becomes
/
g(t) =
p(x)
l x dx.
(50)
Re
19

---

Published as a conference paper at ICLR 2026
It is well-known fact that for any PDF p(), the expected absolute deviation
p(x) 1 dx
(51)
minimized by the median the distribution with PDF p(x). That is, if x(() is the minimum of g()), then it satisfies
{ 2())
/ p(z) dr.
p(z) dx
(52)
2 R.
The denominator Ie m (m X E Rx) dm in (48) can be eliminated from both sides of 52)
allowing redefine p
f(x) / m par (m ( px( M X ER)) dm.
(53)
Now, substituting the definition of p(z) into the left-hand side of 52 we have
(E)
((() [ m (m X XER)) px(x M=m, X RR) dm dr.
((x) dr
[ px(x M M=m,
((()
m (m X
X R2) dr dm.
(54)
The inner integral is the conditional CDF of X given M =m and X ER, Thus.
[ 2(E) p(x) dx So0 par(m) X E m Fx(((() M m X (R) dm.
(55)
Similarly, the total mass in Re, the right-hand side of 52 is
dr [
p(x)
(m X RR) dm.
(56)
Re
Substituting 555 and 56 into 52
we obtain the new condition for optimality:
m (m XERe) Fx(((() M M=m, X RR) dm
1 Pm(mm EE);
(57)
2 J
Rearranging yields
Pm (m X (Px(a(0) M M=m,X - 1) dm 0.
(58)
Finally, the definition of the truncated CDF and the characterization of pas (as(m)) X E R2) from to obtain
25
E(6) m (m)) (Fx(#(0) M =)m - 2 [Fx(x M=m)]][([) g((-))))))))))))))))) dm = 0.
(59)
All expressions can be computed directly from the known PDF pw and CDF Fw of W. using 12). (11, (42) Thus, integration in combination with some method for finding the root of the monotonous function in @(() on the left-hand side of (59). such as the bisection method. (Burden & Faires 2010 pp. 48 ff.)
this equation in Lloyd's algorithm to determine the centroid by numerical
20

---

Published as a conference paper at ICLR 2026
D.3 EMPIRICAL CENTROID COMPUTATION
In Section D.2 we have derived theoretical solutions for the centroid computation in block-wise absmax quantization. However, computing the resulting integrals numerically might suffer from precision issues preventing a straight-forward implementation. Therefore, we additionally provide a simpler method compute the centroid (({) of a region Rr using Monte-Carlo estimation based on samples drawn from the network weight distribution. This method has the additional ad advantage that it can be applied to empirically collected weights from existing pre-trained networks rather than an assumed parametric distribution.
We first sample weights W = (W&) (ws.s) E R grouped into B blocks each with I weights from the distribution of network weights pw. Then, we normalize the weights by their respective block maxima wg obtaining the normalized weights X E R 13xx== = (X+) = (. The objective now becomes to minimize the quantization error of the sampled weights J(W, Q(W)), where either MAEO or MSE() and Q(W) is the result of applying the quantization function Qx() clement-wiss wise to each row W, of W. We apply Lloyd's algorithm based on empirical data to the generated weights with a modified centroid criterion, which is derived in the following for both MAE and MSE optimization.
Our first goal is to minimize the MSE of the network weights based on the sampled weights W. We can express MSF(W. ((W)) in terms of MSE of the normalized weights as follows:
MSE Optimization:
(WW(W))(WW)
1
ΣΣ
(mb Qo(w).)))
B
BEB IET
(wb) max (2(2))))
B
bEB EE
( (#0,1 - (22o.)))
B
BEB
iEI
(w) maxy MSE(X,, Q(X+)),
(60)
B
BEB
where whaxx for the 6th block Wa (wr.s) is computed according to Lp. and Q0 is the block-
independent quantization function 33) that is utilized to quantize normalized weights.
Next, we show how the centroid computation using Lloyd's algorithm must be modified update the reconstruction level ((() in a specific interval weights X (2b,s), such that the MSE of the weights MSE. (,, Q(W)) for that interval is minimized. Let Dk, k E Kr, (1...., K2), ). be those normalized weights 2s,i that fall into the interval ( 1), E()) in the ith block, and wv wita the absolute or signed block maximum corresponding to the normalized weight xg- Using (60). we can conclude that the contribution of the fth interval to the overall MSE is
(()) using empirical samples of normalized
MSE, ((W)) Ke keR.
(wx)2 (II -
(61)
We minimize by computing the derivative w.r.t. & according to
d
2
MSE: W, Q(W))
w (xx - 2),
(62)
de
KP
KE
and setting it equal to 0, allowing to ignore the constant factor:
0 =
w (xt - +()) = Σ
W
Σ F +
(63)
keK,
k
kEK,
Rearranging for å we obtain the optimal reconstruction level in the Eth interval as
a i ΣkEK, wi 23
(64)
ΣnERe w7
21

---

Published as a conference paper at ICLR 2026
which is the weighted mean of weights 23 in the interval (E(L-1).E(0) weighted by their corre sponding squared absolute or signed block maxima w Tab. 8 (discussed Appendix E further supports the equivalence of this Monte-Carlo method with the theoretical solution given 5).
MAE Optimization: A similar derivation can be made for the optimization the MAE. Given fixed normalized network weights æg, Ke, with associated absolute block maxima wky Kr, we are searching for
x (0)) = arg min MAE W, Q())) arg min
Σ kEK
l2 - $
(65)
Therefore, we define the function we aim minimize as
f(i)
Σ
2/.
(66)
kEK2
where TR. h E Ke, are those normalized network weights contained in the interval [c(e=1), Further, we assume, w.l.o.g. that the normalized network weights z are in ascending order: Ii S
T2 S S TKe
Obviously, the minimum must satisfy #1 S [()) S TKe. Consider two distinct, consecutive normalized weights 2x, Tx+1 with 2x # Zx+1- Lr S < t+t S Ix+1 for some R+. Now, we can show the monotonicity of f() on the interval (Ixs xx+1) as follows:
f(z + Σ wp (a+e-
Σ
Wk (xx (o+e))
k=x+1
Wy +
wk xkk
K2
Kr
Wh +
Σ
Wk
(xx -
k=x+1
k=*+1
Wk
Σ
wk
E f(t)
(67)
k=1
k=x+1
Therefore, ((0) is monotonously decreasing on the interval (F+,)a+1)) if
K
Wk
Wk:
(68)
K==+1
and monotonously increasing otherwise. Let medd be the largest index k for which (68 holds. Then, f(t) is monotonously decreasing for å < Fuue and monotonously increasing for ggg Therefore, f()) must be minimal at 2 E,enns. The point n, is known as the weighted median of Ik: K,,, with weights W'.-
In conclusion, the optimal reconstruction level (l)) under the MAE criterion during the iteration of Lloyd's algorithm for normalized network weights is computed as the weighted median
K
2(() f modianw ( (2,,...,, Kei wi,..., wK.) max { [M
WR
Σ
wy
(69)
WEA
k=R+1
for Ty. k E Kr, in ascending order S.z S...... SIK,, both in the case of block-wise absolute and signed absmax normalization. both cases, wy represents the weight with the largest absolute value in the block containing Zg.
E OPTIMAL QUANTIZATION CODEBOOKS
Tab. and Tab. 7 display the codebooks that were computed using the EM algorithm outlined in Section 3.2 In Tab. 6 the reconstruction levels of BOF4 and BOF-S optimized w.r.t. both MAE and
22

---

Published as a conference paper at ICLR 2026
Table Reconstruction levels (()
BOF4 and BOF4-S optimized w.r.t. MAE and MSE for block
size / = 64.
BOF4
BOF4-S
ê MAE-opt. +(()
MSE-opt. a(()
MAE-opt. #(()
MSE-opt. a({)
1 -1.0
-1.0
0.8018796232078552
018364463921546936
2 0..7026305794775588
0,7535245418548884
0.60760551592826643
0.6692874431610107
3 10.5272703766822815
0.579203724861144 145
0.468828022480011
0.5235266089433392
4 10.3946738243103007
0043859988451004003
-.3359960277703
0.400488257408114
5 (12832114796848222
0.3116779905889418
10.25661693716649999
1029100331
6 -0.18353 15888380814 14
0.2255924453499693
0.1677481383988555
-0. 1900992959403992
7 0.090308666229248
0.1015387225501182
0.0827366262643332
0.093852959557326699
8
0.0
0.0
0.0
0.0
9 0.07896600016686584
0.0887245312333107
0.0789434835314331
O 0.0887671 018876717090000 709066669
10
1.15987725231456676
0. 0.1799779598007222
0.15979666498803925
0.1 7998026655112772
11 0.2449863555044118
0.2741499841213226
0.2448495477437973
0.2743096053600311
12
0.375821 1433887482
0.3371480107
0.3760197460651338
13
0.441359281539917
0,48849377036096677
0,4412573873996735
0,4886530041696611
14
0.565777063369751
0,6187058687210083
0.565681934356660
0.61 0,6886603639602666
15
0.7299178222683411
0,779045224189558
0.7298068404197693
0.77 0.7791395783424377
16
1.0
1.0
1.0
1.0
Table 7: Reconstruction levels s()) BOF4-S optimized w.r.t. MSE for various block sizes 7.
(()) for BOF4-S (MSE)
32
1
64
128
256
1 -088732797503471375
00856646392115469366
0.83739173412323
0.8146629009056091
2 0..697744334429932
0.6692874431610107
0.6462452411651661
-0.6221 03859333934
3 -0.54370391 136886597
0.5235266089439392
0.502863467336555
-0. 1820544190044440
4 -0.4173701 1044173701703548431 703548431
0.4004882574081427
0.3836244622966666
1.3666650852680266
5
1
0.29106338
-027183775009119988
-0,26598718762277
6 1821989917222112999
-0. 0.1190092959403992
-01815571394055004
-0. 171333223366000
7
0.09811552220225566
9.09382299995:326899
0.0896477326755755
1.0855777593008910
8
0.0
0.0
0.0
0.0
9 0.09259384122547122
0,0887671709 70996666999
0.08 5091509523093000
10 0.187048003077700
0.1794802695512772
0.1720834821 462631
0.1649 49656295776
11
0.28551977892266677 0,2743090053600311
0.2632072865962982
0,25243920087814433
12
0.3907126188278198
0.376019746065
0.36132931 70928955
0.3470274209976196
13
0.506233164024353
094886330046694411
0.4707452654838562
0.453153331415577
0.6379748582833996
0,61886633339006661
0.5988966822622207
0.5788486599922218
15
0.7956376671911077
0.7791395783342377
0.761027991771698
0.7418596744537354
1.0
1.0
1.0
1.0
MSE are shown for an example block size / 64. Tab. a shows the reconstruction levels of our top-performing quantizer BOF4-S optimized w.r.t. MSE for additional practical block sizes / < 256.
Furthermore, Tab. presents a comparison of the BOF4 (MSE) reconstruction levels computed with two different implementations. In the first solution, the centroid computed based on an empirical approach by the Monte-Carlo method using second (theoretical) solution is computed data-independdenly using our implementation on numerical integration. The variance in the finite number of Gaussian samples on the one hand, and numerical inaccuracies on the other hand, cause minor differences in reconstruction levels. The MSE between the theoretical and empirical solution is computed as (in dB)
data according to (6), while the (5) based
Etec PIX E Re] (5theo (f) jemp (0) dB,
MSF 10 log10
(70)
Eiec P E Re] (t)
23

---

Published as a conference paper at ICLR 2026
Table 8: Reconstruction levels of BOF4 (MSE) for block size 64 using either the empirical method (f)))(() or the theoretical solution (e)) for the computation of centroids. The third column shows the absolute deviation between corresponding reconstruction levels.
Empirical solution
Theoretical Solution
Deviation (emp(() - 5 tth (2)
(c)
itheo (()
1
-1.0
-1.0
0.0
2 -0.75552454185445544
-0.75356892038895577
0.00004437853320993
3 -0.579203724861144
0.57722681492533122
0.0000644243922563
4 0.4385998845100403 0044886700884478466
0.0000721239377865
5 40.3167679905891411 -0.2060291 109696588
0.316819103979144
0.0000511133990002
6 0.22599944453999933
0.0000366656119653
7
0.10053386225001188
0.1015640796456477
0.0000253171954289
8
0.0
0.0
0.0
9
0.0387245312333117
0.8887646748673226
0.0000401436340100
10
0.17937695988002202
0.1794535266886747
0,0000765668879545
11
122744499842213226
0.274249773841407
0,0000997897220443
12
0.375821 1433887482
0.37595 1029286055
0.0001298858972968
0.48849377033694477
0.48859255268366
0.0000987564759645
14
061187558882110883
0.61877155462888
0.000065633977922
0.779045241189588
0.7790823367844242
1.09003761259466599
16
1.0
1.0
0.0
where L = {l...., 16}. With the results from Tab. we obtain MSE dB..34 This
demonstrates the practical equivalence of both implementations.
F OPTIMIZING THE QUANTIZATION ERROR OF NORMALIZED WEIGHTS
Instead of minimizing the end-to-end quantization error MAE(W, Q(W)) or MSE(W, Q(W)) of the network weights Wb, as in BOF4(-S), see Section 3.2 equations and ( one could alternatively minimize the quantization eтror MAE(X, (X)) or MSE(X, (.) (X) of the normalized weights
Th,i. comparison to BOF((-S,, optimizing the quantization error of normalized weights is more straightforward and can be achieved using Lloyd's algorithm (Looyd 1982) with standard centroid update rules. For MAE minimization, the centroid of Voronoi region R, is computed based on samples from the network weight distribution with PDF pw, the median of normalized weights Ik E I. with & E K = {l,.... Ke}:
a() median (2 (......,, FK,)
(71)
For MSE minimization, the optimal centroid is the mean
a(() = Ki
Σ Fk.
(72)
kEK
Note that BOF4(-S) modifies these centroid conditions by introducing an additional weighting of the normalized network weights 24 depending on the absolute block maxima Wy of their respective block (see (8) for MAE and (6) for MSE).
We empirically compare the two optimization strategies. A 4-bit codebook minimizing MSE(X, (x(X)) is computed with Lloyd's algorithm using centroids as defined in 772) Then, the perplexity of Ll ama 3. 8B on WikiText-2 is measured for both this codebook (72) and BOF4 (MSE) (6) Figure 6 shows the difference in perplexity PPL PPLNORM between the two optimization approaches, with PPLBOF referring to the perplexity achieved by BOF4 (MSE), and PPLNORM referring the perplexity when using the codebook that minimizes MSE(X, (X)). For all values of I, the difference is negative, indicating that BOF4 (MSE) consistently achieves lower perplexity than the codebook minimizing the MSE of normalized weights.
24

---

Published as a conference paper at ICLR 2026
0.00
PPLBOF PPLNORM
-0.01
-0.02
-0.03
-0.04
23
26
27
2
29
20
2
2
block size
Figure Difference in perplexity on WikiText-2 of Llama 1.mma-3.1 1. 8B quantized with BOF4 (PPLBOF) codebook minimizing MSE of normalized weights (PPLNORM). Lower val- ues indicate better performance of BOF4.
G FURTHER DETAILS ON
QUANTIZATION (OPQ)
G.1 DESIGN CONSIDERATION.
We use a method to identify outliers that depends on the standard deviation d of weights within a block b rather than on fixed threshold, as the scaling of individual blocks within neural network layer's weight tensor can vary greatly. Accordingly, we normalize the weights in each block to standard deviation of dividing by the sample estimate
Σ ubb w.) b B,
(73)
where W, = 7 Wh.s denotes the sample mean of weights Wb,i in block Furthermore, to the
make the method generally applicable to different distributions Pw of network weights, expected distribution of absolute block maxima waaax (pM from (12) to determine the threshold at which normalized weight Wh, is classified distribution of absolute block maxima with PDF pM for some value q close to as the threshold. Intuitively, this means that normalized weight counts an outlier its absolute value is larger than fraction of all absolute block maxima, assuming that the actual distribution of network weights would ideally adhere our distribution assumption pw.
outlier. Specifically,
the q-quantile of the
Fig. iillustrates the detection of outliers. The blue histogram represents a block of absolute network
[wa.sl weights de normalized by the standard deviation σg- The PDF par (see 12 describes the theoretical distribution of absolute block maxima, indicating where the largest absolute non-outlier weight expected. In this example, we define outliers as absolute weights exceeding the 95th percentile of the expected absolute block maxima, denoted by (772(0 the inverse of the CDF F'M taken from (II). An example outlier is highlighted by red hatching.
Fig. illustrates the advantage of applying OPQ the network weights W, which almost with only a small fraction of outlier weights that are highly unlikely to occur in Gaussian Gaussian-distributed data. While OPQ stores the outlier weights (red color) 16-bit precision, the non-outlier weights (blue color) are subject to normalization. On the right side in Fig. 8 the resulting normalized weights X without and with OPQ are shown for the weights that are no absolute block maxima, i.e., (-1, 1,11, 1). Without applying OPQ, the outliers affect the scaling of their blocks during normalization, resulting in a distribution
normalized weights X that is more concentrated around the mean than the distribution poont for which the quantizer was optimized. This is because during normalization, each block is divided by its absolute maximum winxx If W contains outliers, wi is larger than expected for many blocks, leading smaller normalized weights, which lets quantizer operate in the underload regime, thereby being suboptimal w.r.t. its rate- distorrtottion characteristics. On the other hand, when OPQ is used, the outlier weights replaced with the placeholder value of 0 in
25

---

Published as a conference paper at ICLR 2026
Bist(lws..l/og
1.0
outliers
PM (w)
0.8
Fa (0.95)
0.6
0.4
0.2
0.0
2
5
w
Figure 7: Illustration of OPQ outlier detection. The histogram of absolute weights [wa.l of example block with block size 64 normalized to a unit standard deviation shown in blue. Weights are identified as outliers (red hatching) iff they are greater than Fis (0.95), i.c., expected be greater than q 95% of the absolute block maxima ww according to the assumption of cuussian-distributed network weights Wh,s. The corresponding PDF of absolute block maxima |ugaax shown as black solid line.
Hist W)
Hist (X) w/o OPQ
Hist (X)
non-outlier weights outlier weights normalized weights X thcoretical PDF x
w/ OPQ
block-wise absmax normalization
1
W
Figure Effect of outlier-preserving quantization (OPQ) on the distribution of normalized network weights. The histogram of original network weights W containing some outlier weights (red) and non-outliers (blue) is shown on the left. The normalized network weights X that are not -1 are shown on the right with and without OPQ. The theoretical PDF px of the continuous part of normalized weights shown for comparison. The PDF p is computed under the assumption of Gaussian-distribute network weights, whereas the true network weights W contain (non-Gaussian) outliers.
the weight tensor W before normalization. Consequently, the distribution normalized weights X is much more similar to the theoretically expected PDF ponnt We chose this method for managing outliers, instead abandoning the assumption of Gaussian network weights, because we observe that most rows of weight matrices in LLMs are very close to Gaussian, whereas only some blocks follow a Gapps-Ga distribution with a small number of large- magnniude outlier weights. This observation is also supported by Dettmers et al. (2023, Appendix I). In practice, the design of OPQ enables one to control the expected number of weights stored in high precision via the choice the hyperparameter 4-
26

---

Published as a conference paper at ICLR 2026
1.0
*
4 0.95
relative PPI. improvement (%)
q 0.99
q 0.9
L1 Llama 3.1 70B
q 0.97
Llama 11ama-3.2 3.2 1B
Llama 3 1
8B
3B
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
relative memory overhead (%)
Figure 9: Relative perplexity (PPL) improvement WikiText-2 validation split and relative memory overhead in relation to memory utilized by 4-bit quantized weights for OPQ at various model sizes and values of the hyperparameter q. All results use BOF4-S (MSE) with block size T= 64.
Note that the reconstruction levels of BOF4 or BOF4-S, shown in Tabs. 6 and remain unchanged when OPQ is used.
G.2 HYPERPARAMETER ABLATION
We perform a hyperparameter ablation to analyze the influence of the OPQ hyperparameter g on the quantized model's performance and the additional memory cost compared to block-wise absmax quantization without OPQ. Fig. presents the results for various Llama models ranging from billion to 70 billion parameters and hyperparameter choices {0.9, 0.95, 0.97, 0.99, 1.0). The relative perplexity improvement from OPQ is calculated as
APPL(gg PPL - PPL(g)
(74)
PPL.,f
where PPL(q) is the perplexity of the quantized model on the WikiText-2 (Merity et al. 2017) vali- dation split using BOF4-S (MSE) with OPQ specific value for the hyperparameter q, and PPL the reference perplexity resulting from BOF4-S (MSE) without OPQ. The memory overhead is calculated in relation to the memory usage by 4-bit quantized layers, excluding layers that quantized, such as the embedding and final layers.
We observe that OPQ yields substantially larger relative perplexity improvement for the Llama-3.1 70B model compared to the other significantly smaller LLMs with 1...8 billion parameters. Notably, this improvement is achieved without any increase relative memory over- head. This suggests that, in very large LLMs, preserving weight outliers is even more critical for maintaining performance. For fixed value g, the relative memory overhead similar between model sizes. For instance, at y 0.95 the overhead ranges from 1.32% and 1.79% for all evaluated models. In smaller I.LMs, decreasing q below 0.95 provides only marginal perplexity gains while increasing memory overhead. For very large models, however, choosing lower value of q 0.95 may be viable if the deployment hardware can accommodate the additional memory overhead, since perplexity continues improve beyond this point. Based these observations, recommend q = 0.95 as a robust hyperparameter choice across model sizes.
Additionally, analyze how the block size I affects the choice of the hyperparameter q by measuring the additional memory cost (Fig. 10) and the perplexity on the WikiText-2 validation split (Fig. TT). using I. aalama-33 8 example.
27

---

Published as a conference paper at ICLR 2026
4=09
9=09
4=096
Perplexity (WikiText-2)
9=095
Memory overhead
0-097
9=097
2.0%
4-099
OPQ
8.8
9-099
web
1.0%
8.6
0.0%
2
26
27
2
2
2
21
2
2
27
2
2
2
2
block size I
block size I
Additional OPQ memory overhead of BOF4 (MSE) applied to Llama-3.2 8B as fraction of the total memory required by the quantized model af- ter block-wise absmax quantization, includ- ing the quantization constants.
Figure 10:
Figure 11: Perplexity of Llama 33 on the WikiText-2 validation split after quan- tization with BOF4 (MSE) using OPQ with various values of the hyperparameter q.
We observe that the memory overhead decreases as the block size I increases. Moreover, the positive effect of OPQ perplexity increases with increasing block size. For instance, when setting q 0.9, the memory overhead at block size / = 32 is approximately 3% while the effect on perplexity is low. Meanwhile, larger block sizes, the effect on perplexity becomes more pronounced and OPQ only incurs a minimal memory overhead, even for = 0.9. Across block sizes, we find that 0.95 Hight-green curves) leads an acceptable fraction weights stored in high precision, even for small block sizes, and consistently improves perplexity every tested block size. This confirms our earlier observation from Fig. 9 that g = 0.95 is robust hyperparameter choice.
G.3 RUNTIME OVERHEAD
additionally evaluate the runtime overhead of OPQ. Figure shows the time required to generate 1000 tokens with Llama-3.1 8B NVIDIA RTX 4070 Ti Super GPU using block-wise absmax quantization without and with OPQ. We use a batch size of and start generating from an empty context. Note that the particular block-wise absmax quantization method that is used does not influence the decoding runtime, since NF4, AF4, BOF4, and BOF4-S all utilize the same implementation of decoding, only differing in the values of the reconstruction levels ((f). We measure runtime overhead of 2.88% on average for OPQ, which is nearly constant in the block size with the largest measured overhead being 3.84%. This shows that OPQ only incurs a minimal runtime overhead.
H TRAINING AND EVALUATION DETAILS
Our hyperparameter choices align closely with those used by Dettmers al. during the original evaluation of the LooRA method (Dettmers et al.
Hutter 2019 with constant learning rate 2023). We use the AdamW optimizer (Loshchiloy 10 configured with the exponential decay rates 31 = 0.9 and Bg 0.999. We perform supervised fine-tuning for 1875 steps using batch size 16. Furthermore, we use gradient clipping with a mwx_grad_noo.r..o....a parameter of 0.3. dropout with 10% dropout rate is applied to the LoRA layers. In contrast to Dettmers et al. (202)). do not perform double qunaitzation, i.e., the quantization constants are not further quantized.
We the LM Evaluation Hamnsss by EleutherAl (Gao al. 2024) (MIT License) for the evaluation on the NLP benchmarks. The code generation benchmarks HumanEval+ and MBPP+ are provided by the EvalPlus framework (Liu et al. 2023) (Apache 2.0 License).
28

---

Published as a conference paper at ICLR 2026
115.0
112.5
runlime [s]
110.0
107.5
105.0
-
w/ OPQ
102.5
w/o OPQ
100.0
26
27
28
29
210
211
block size I
Time to generate 1000 tokens starting from an empty context with and without OPQ, depending on the block size evaluated on Llama-3. 8B using a batch size of
Figure 12:
All fine-tuning runs were conducted on a single A100 40GB GPU. Each run finished in less than 8 hours. For perplexity and accuracy evaluations, either an NVIDIA RTX 3080 with 10GB of memory or an A100 40GB was used. Each evaluation run of (quantized) model on our set of NLP benchmarks required most 24 hours.
I APPLICATION TO CALIBRATION DATA BASED QUANTIZATION
For most experiments, we evaluate BOF4 in a data-free setting, where quantization is determined solely by the network weights rather than by network activations computed with the help of calibration data. This ensures fair comparison with the data-free data-free methods NF4 and AF4, which we improve upon. While data-free quantization offers this comparability, post-training data-awan aware methods often achieve higher accuracy. Data-aware approaches, such as GPTQ (Frantar et al., 2023), AWQ (Lin et al. 2024). and SmoothQuant (Xiao et al. 2023), leverage calibration data to optimize the assignment of network weights to reconstruction levels. Most of these methods rely on uniform quantization, yet the uniform codebook can be readily replaced with a non-uniform one. In fact, the use of non-uniform codebooks in GPTQ has already been explored by van Baalen al. (2024). Furthermore, GPTQ natively supports operating on normalized blocks of weights (referred to as groups by (Frantar et al.] 2023)), making it natural framework evaluate our BOF4 method in a data-aware setting. Moreover, GPTQ can be trivially modified to use our signed block-wise normalization (Section) 3.1 method and OPQ (Section 3.3
As proof of concept, we experimentally evaluate this approach. Frantar et al. (023) demonstrate that GPTQ already works well with larger block sizes between 128 and 1024. Accordingly, evaluate all methods at block size 128. As calibration data, we use 512 samples from the C4
dataset (Raffel et al. 2020).
The results are shown in Tab. We observe that using GPTQ with non-uniform BOF4 (MSE) quantization significantly improves perplexity and average accuracy compared the uniform quan- tization from the original GPTQ proposal. Further improvements are achieved by adding signed normalization (BOF4-S) and autler-prreservngg quantization (OPQ). Overall, perplexity is reduced from 8.60 to 8.34 on WikiText2 and from 4.32 to 3.93 on Lambada, while normalized average accuracy improves from 42.0% to 42.8%. This significantly closes the gap to full-precision (BF16) inference. Furthermore, we observe that non-uniform quantization generally improves perplexity and accuracy significantly: non-unifon uniform, data-fie free quantization performs better than uniform, data- aware GPTQ both perplexity metrics and competitively in terms of accuracy. This highlights that non-unite non-uniform quantization is a viable option that should be considered in cases where memory is the main concern and improving inference speed is secondary. Our results demonstrate that
29

---

Published as a conference paper at ICLR 2026
Table 9: Inference with GPTQ quantization (Frantar 2023) with block size I 128 and various underlying codebooks evaluated using Llama-33 3.1 8B. Best- Best-performing data-free methods are shown for comparison. The evaluated metrics are the perplexity on the WikiText-2 and LAMBADA dataset, and the accuracy on the MMLU (few-shot), ARC-Challenge, HellaSwag, PIQA, SIQA, and WinoGrande benchmarks. Best result in each column in bold, second best underlined, 6F16 excluded.
Quantizer
WikiTeat2 PPL
Lambada PPL
MMLU ACC1
ARC-C
HellaSwag ACC
PIQA
SIQA
WinoGrande ACC
NAV
ACC
ACC
ACC
ACC1
BF16
7.94
3.96
63.0
51.3
60.0
80.0
47.0
73.8
43.4
BOF4-S (MSE)
8.57
4.04
61.4
48.3
59.3
79.7
46.5
72.4
41.5
+OPQ
8.48
4.05
61.7
48.9
59.3
79.8
46.7
73.0
42.0
GPTQ, uniform
8.60
4.32
60.9
50.2
59.3
79.2
47.2
72.9
42.0
GPTQ, BOF4 (MSE)
8.41
3,96
62.0
49,9
59.3
79.2
47.4
73.0
42.3
GPTQ. BOF4-S (MSE)
8.36
3.92
62.3
49.8
59.5
79.0
48.1
72.5
42.3
+ OPQ
8.34
3.93
62.2
50.5
59.3
79.7
47.3
73.7
42.8
contributions- codebook optimization, signed normalization, and OPQ-cur can be naturally incorporated into calibration-dater data-based quantization methods, yielding clear benefits.
J ADDITIONAL EVALUATIONS
and the Qwen 2. 7B models are shown. Note that Tab. [10 corresponds to Tab. the latter just showing larger models. Similar to the larger models, we observe that our BOF4(-S) quantizers perform at least as well and usually better than the baseline methods in the quantization error metric (MAE or MSE) for which the particular codebook optimized. Furthermore, the BOF4-S quantizers using our signed absmax normalization significantly improve all metrics over BOF4 with absolute absmax normalization. When additionally applying OPQ to the BOF4-S quantizers, performance all metrics improves further. The lowest errors achieved by BOF4-S +OPQ, using the codebook optimized for the particular error metric. Interestingly, for the Qwen-2... 3B model, our MAE-optimized methods generally achieve better perplexity, suggesting that the target error metric for optimization, which leads to the best performance, may vary depending on the LLM.
In Tab. I0 the quantization error and perplexity results for the Llama-33
Fig. 13 shows perplexity results for BOF4 the WikiText-2 and LAMBADA datasets. Note that Fig. 13 corresponds to Fig. the latter reporting on BOF4-S, however. We observe that for most block sizes the perplexity of BOF4 optimized w.r.f. MAE MSE similar that of the best-performing baseline method. Adding OPQ significantly reduces perplexity, particularly when used with MSE optimized codebooks and at large block sizes.
Tab. II displays additional perplexity and accuracy measurements for the larger allam-3 BB, owen-2.5 7B, and the tiny Qwen 2.5 0. 5B. Note that Tab. IT corresponds to Tab, 2 the latter reporting on small 3B models. Our best BOF4 quantization method consistently outperforms the baseline methods, NF4 and AF4, in terms of perplexity on WikiText-2 and LAMBADA, except when applied to the Qwen-2.5 2. 7B model, where NF4 achieves surprisingly low perplexity on LAMBADA-SuUE surpassinn even the performance of the unquantized BF16 model. Note, however, that for the Owen-2... 7B model, each of our four proposed BOF4(-S) methods performs as well as or better than NF4 in the NLP benchmarks' normalized average accuracy (NAV) metric. Overall normalized average accuracy (NAV) results from the language modeling benchmarks do not indicate a single quantizer or approach that consistently performs best.
The OPQ variant proves particularly effective for the smaller Qwen-2.5 0. SB model, where significantly improves perplexity over the respective quantizer without OPQ and both baselines NF4 and AF4.
30

---

Published as a conference paper at ICLR 2026
Table 10: Quantization error (MAE and MSE) and perplexity (PPL) on WikiText-2 of quantization methods applied to the network weights of two 3B regime LLMs with block size I = 64. Best result each column in bold, second best underlined.
Llama-33 1 3B
Qwen-2.5 3B
MAE 1e-3
MSE
PPL 4
MAE
MSE J PPL 4
1e-6
1e-3
1e-6
NF4
1.399
3.333
10.72
1.822
5.722
12.13
AF4
1.441
3.588
10.71
1.862
6.118
13.48
BOF4 (MAE)
1.399
3.302
10.72
1.821
5.670
12.16
BOF4 (MSE)
1.424
3.191
10.73
1.862
5.526
12.46
BOF4-S (MAE)
1.341
3.071
10.68
1.746
5.274
12.07
+ OPQ
1.316
2.971
10.63
1.689
5.026
12.05
BOF4-S (MSE)
1.367
2.936
10.66
1.788
5.087
12.41
+ +PQ
1.336
2.791
10.64
1.719
4.739
12.36
9.2
NEI
9.2
NF4
AF4
9.0
BOF4 (MAF)
9.0
BOF4 (MSF)
+0PQ
.OPQ
8.8
8.8
8.6
8.6
8.4
8.4
26
26
27
28
29
10
211
212
25
26
27
28
29
200
211
212
block size I
block size I
Figure 13: Perplexity of ama-3.1 8B on WikiText-2 after quantization with NF4, AF4, and BOF4 optimized w.r.t. MAE (left, MSE (right, o) for different block sizes I, without and with outlier-peserrvng quantization (OPQ, dashed line).
K DEFINITION OF THE NORMALIZED AVERAGE ACCURACY METRIC
To determine an overall accuracy score for model over multiple benchmarks, we employ normal- ized average accuracy that accounts for the chance level accuracy achievable by random guessing on each benchmark. For example, some benchmarks use multiple-choice format with four answer options. In this case, random guessing would yield an accuracy of 25%. To ensure that no benchmark disproportionnaely influences the average accuracy, we normalize the accuracy of multiple-choice benchmarks such that random guessing expected yield 0% and answering all queries correctly yields 100%. The normalized accuracy is calculated
ACC ACC ACC ACCchance
(75)
ACCchance
where ACCehance is the chance-level accuracy. This normalized average accuracy ACCuoru is reported in Tabs. and abbreviated as NAV ACC.
31

---

Published as a conference paper at ICLR 2026
Table 11: Inference results of 4-bit scalar quantization methods evaluated using additional LLMs with block size 64. The evaluated metrics are the perplexity on the WikiText-2 and LAMBADA dataset, and the accuracy on the MMLU (few-shot), ARC-Challenge, HellaSwag, PIQA, SIQA, and WinoGrande benchmarks. Best result each column bold, second best underlined, BF16 excluded.
Model
Quantizer
WikiTex12 PPL
Lambada PPL
MMLU
ARC-C
HellaSwag ACC
PIQA ACC1
SIQA ACC f
WinoGrande ACC
NAV
ACC f
ACC f
ACC
BF16
7.94
3.96
63.0
51.3
60.0
80.0
47.0
73.8
43.4
NF4
8.53
59.1
78.9
73.6
42.0
3.1
AF4
8.51
4.38
61.6
49.9
59.1
79.5
47.0
73.6
42.4
8B
BOF4 (MSE)
8.47
4.25
61.7
50.4
59.3
78.9
46.4
73.1
42.0
OPQ
8.47
4.25
61.7
50.4
59.3
78.9
46.4
73.1
42.0
BOF4-S (MSE)
8.47
4.29
61.7
48.5
59.5
79.2
46.2
72.8
41.6
OPQ
8.43
4.29
61.9
49.2
59.5
79.7
46.5
72.5
41.9
BF16
9.50
4.53
71.5
48.2
60.0
78.7
54.8
45.8
NF4
9.91
4.48
70.7
46.7
59.0
78.9
54.2
71.7
44.6
Qwen-2.8 2.
AF4
9.90
4.70
70.6
47.2
58.9
78.3
54.5
70.2
44.0
7B
BOF4 (MSE)
9.95
4.83
70.7
48.2
59.2
78.7
54.1
71.3
44.8
OPQ
9.85
4.73
70.6
47.4
59.2
78.9
54.2
72.2
45.0
BOF4-S (MSE)
4.79
59.2
78.6
70.6
44.6
OPQ
9,83
4.67
70.6
48.5
59.3
78.6
54.4
71.1
44.8
BF16
19.64
16.95
47.5
40.6
70.2
NF4
22.24
25.20
44.8
28.3
38.8
69.5
44.4
56.6
19.7
Owen-2.5
AF4
22.14
27.17
43.5
28.5
39.0
68.9
43.3
56.8
19.1
0.58
BOF4 (MSE)
22.22
27.28
45.1
29.9
39.1
69.5
42.9
54.5
19.1
OPQ
21.72
24.61
45.0
39.0
69.4
BOF4-S (MSE)
23.02
26.64
44.2
30.4
39.1
68.0
43.7
55.6
19.1
+ OPQ
21.88
22.90
44.2
29.6
38.8
68.4
43.4
56.7
19.2
32


Reviews:
[{"review_id": "3uhjJa0v0H", "reviewer": "Reviewer_M1dB", "summary": "This paper proposes a novel approach to improving block-wise training for large language models (LLMs) by introducing refined optimization and scheduling mechanisms that better capture inter-block dependencies during training. The authors demonstrate that their method enhances both convergence stability and downstream task performance, offering a computationally efficient alternative to full end-to-end fine-tuning. The study is well-motivated, clearly written, and supported by extensive experimental validation across multiple benchmarks.", "questions": "How sensitive is the performance to the choice of block partitioning (e.g., number of layers per block)?\n\nCould the proposed inter-block dependency modeling be integrated with adapter-based fine-tuning approaches?\n\nHave the authors considered evaluating the method’s stability when applied to continual learning or streaming data scenarios?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed method provides an original perspective on block-wise training, addressing the often-overlooked issue of gradient inconsistency across blocks.\n\n2. The formulation is mathematically rigorous, with theoretical justification for the proposed scheduling strategy.\n\n3. The experiments span several model sizes and datasets, showing consistent improvement over baselines such as layer-wise and progressive tuning.\n\n3. The approach maintains efficiency advantages (reduced memory and training cost) while achieving comparable or superior results to full fine-tuning, highly relevant for real-world large-scale model adaptation.\n\n4. The paper is well-structured, with clear motivation, methodology, and ablation analysis that enhances understanding.", "weaknesses": "1. While the approach performs well on benchmark datasets, it would be useful to see how well it generalizes to non-language tasks (e.g., multimodal or code models).\n\n2.  The method involves scheduling parameters whose influence is only briefly discussed; more detailed robustness analysis would strengthen the contribution.\n3. Although the method is efficient, the paper could provide clearer quantification of the additional cost introduced by the new scheduling mechanism relative to vanilla block-wise training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zALIzhE9rx", "reviewer": "Reviewer_sXWd", "summary": "This paper presents a comprehensive study on improving 4-bit block-wise quantization for Large Language Models (LLMs). The authors identify a fundamental issue in existing methods (e.g., NF4, AF4): they optimize the quantization error of the *normalized* weights rather than that of the *original* weights, leading to suboptimal results. To address this, the authors propose an improved Expectation-Maximization (EM) algorithm that directly optimizes the error of the original weights, resulting in a family of quantizers termed BOF4. They also introduce signed absolute maximum normalization, which frees up one codebook entry to enhance representational capacity. Additionally, a mixed-precision scheme is proposed to identify and handle outlier weights. The paper is strongly supported by rigorous mathematical derivations and extensive experiments on multiple LLMs and tasks (both inference and fine-tuning), demonstrating consistent and significant performance improvements over strong baselines.", "questions": "- **Q1: Gaussian Distribution Assumption**: The EM algorithm used for BOF4 relies on an assumed Gaussian distribution. How sensitive is the final performance to deviations from this assumption? If a model's weights do not follow a Gaussian distribution, what would be the magnitude of deviation this algorithm might cause?\n\n- **Q2: Selection of Hyperparameter q**: Regarding OPQ, the hyperparameter `q=0.95` was chosen via a limited search. Could you discuss the sensitivity of the results to the choice of `q`? Is this value generally robust across different models and sizes, or does it need tuning?\n\n- **Q3: The Special Sign Bit:** The signed normalization in BOF4-S requires storing the sign of the block maximum if double quantization is applied, as noted in Appendix A. Could you quantify the potential performance degradation if this extra bit is not used and the standard double quantization scheme is applied naively?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Theoretical Soundness and Novelty:** The paper's core insight—optimizing the end-to-end quantization error of the *original* weights rather than that of the *normalized* weights—is profound and well-articulated. The derivation of new centroid update rules for Lloyd's algorithm (applicable to both MSE and MAE) constitutes a solid theoretical contribution. Clear and comprehensive mathematical proofs further solidify the theoretical foundation.\n- **Holistic Methodological Framework:** The paper extends beyond a single idea by introducing a suite of complementary techniques: an optimal codebook based on reconstruction loss (BOF4), a signed normalization scheme (BOF4-S), and a practical outlier-preserving mechanism (OPQ). This integrated approach effectively enhances quantization accuracy from multiple perspectives.\n- **Thorough and Convincing Experiments:** The evaluation is extensive, covering multiple model families (Llama, Qwen, Mistral), both inference and fine-tuning (QLoRA) scenarios, and a wide range of benchmarks (perplexity, NLP tasks, code generation). The results consistently show that the proposed methods, especially BOF4-S with OPQ, outperform the baselines.\n- **Practical Impact and Reproducibility:** The methods are directly applicable for memory-efficient LLM deployment and fine-tuning. The paper provides optimized codebooks in the appendix and discusses integration with data-aware PTQ methods like GPTQ (Appendix I), enhancing its practical utility and reproducibility.", "weaknesses": "- **Assumption of Gaussian Weight Distribution:** The optimization of the BOF4 codebook relies on the assumption that network weights are Gaussian-distributed. Although Appendix C provides justification that most blocks are indeed Gaussian, especially after OPQ, the performance on models or layers with significantly non-Gaussian weight distributions remains less explored. This could limit the generalizability to certain architectures.\n\n- **Overhead of OPQ:** Although OPQ is shown to have minimal runtime overhead (Appendix G.3), it introduces additional memory overhead for storing the outlier indices and values. A more detailed analysis of this memory-cost/accuracy trade-off, especially for very large models, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTOGYSrUZ9", "reviewer": "Reviewer_dVvQ", "summary": "This paper introduces a novel 4-bit block-wise quantization algorithm called BOF4-S. The proposed algorithm employs an enhanced EM algorithm to obtain the optimal reconstruction levels. Additionally, it normalizes the weights using the signed absolute block maximum, which effectively saves one reconstruction level, thereby adding an extra degree of freedom to the codebook and further reducing quantization error. Moreover, this algorithm excludes outlier weights and store them separately to further enhance the quantization accuracy. Experimental results demonstrate that the proposed algorithm achieves less quantization error and perplexity compared to the conventional methods, while also achieving superior model performance before and after fine-tuning.", "questions": "1. Is it possible for both endpoints of the normalized weights (-1 and 1) to appear at the same time? If so, how is the sign handled?\n2. When outliers are removed, their corresponding values in the tensor are replaced with 0. Is this tensor modified before or after normalization? Does directly replacing values with 0 impact the distribution of weights within the current block, potentially interfering with the selection of the optimal codebooks?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper provides a thorough and clear explanation of the method, making it easy for readers to understand. The experimental results are comprehensive, which strongly supports the effectiveness of the proposed method. \n\n2. The method addresses several practical issues in current quantization algorithms, such as the wastage of reconstruction levels and the impact of outliers on quantization accuracy. By cleverly saving reconstruction levels, designing an optimized codebook algorithm, and eliminating outliers, it provides an ingenious approach to optimizing quantization algorithms, further enhancing the quantization accuracy.\n\n3. This method achieves improvements in both quantization error and perplexity with minimal time overhead. Furthermore, the LLMs quantized using this algorithm demonstrate better task performance before and after fine-tuning compared to traditional methods, showcasing the method's strong applicability.", "weaknesses": "Although selecting one of the two endpoints as the reconstruction level for the maximum absolute weight provides an additional degree of freedom for the codebooks, it also requires an extra bit to store the sign of this maximum value. Is this overhead justified? Especially since, when using Llama-3.2-3B as the base model, there is no performance improvement on most tasks (Table 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mFwmXVOihO", "reviewer": "Reviewer_SU8o", "summary": "The paper proposes a new 4-bit data format called FOF4, which leverages the asymmetric property of maximum/minimum values within a block to reduce the waste of degrees of freedom in the quantization codebook. This method can be regarded as a type of non-uniform quantization.", "questions": "Methods that previously determined normalization constants based on MSE have also achieved good results. Why is the weight based on the maximum absolute value chosen instead? What advantages does this approach offer?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This method leverages the asymmetry of the boundaries in weight quantization to reduce quantization error.", "weaknesses": "1. This method can be regarded as a special type of non-uniform quantization method (1D codebook). Additionally, compared with other KMeans-based methods such as GPTVQ and RPTQ, it may not have advantages in terms of accuracy and speed (without hardware support).\n2. This method lacks a comparison with similar basic codebook-based methods like GPTVQ，VPTQ...\n3. Current LLMs can achieve W4A4 quantization with almost no loss, which has greater advantages in reducing computational overhead. This method seems unable to achieve this, and it also requires additional unstructured outlier storage and computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)