LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid
Authors: Tianyi Zhang
Anshumali Shrivastava
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
quantization
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Large language models (LLMs) have shown immense potential across various domains, but their high memory requirements and inference costs remain critical challenges for deployment. Post-training quantization (PTQ) has emerged as a promising technique to reduce memory requirements and decoding latency. However, recent accurate quantization methods often depend on specialized computations or custom data formats to achieve better model quality, which limits their compatibility with popular frameworks, as they require dedicated inference kernels tailored to specific hardware and software platforms, hindering wider adoption. Furthermore, many competitive methods have high resource requirements and computational overhead for quantizing models, making it challenging to scale them to hundreds of billions of parameters. In response to these challenges, we propose LeanQuant (Loss-error-aware network Quantization), a novel quantization method that is accurate, versatile, and scalable. In the existing popular iterative loss-error-based quantization framework, we identify a critical limitation in prior methods: the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals. To overcome this fundamental issue, we propose learning loss-error-aware grids, instead of using non-adaptive min-max affine grids. Our approach not only produces quantized models that are more accurate but also generalizes to a wider range of quantization types, including affine and non-uniform quantization, enhancing compatibility with more frameworks. Extensive experiments with recent LLMs demonstrate that LeanQuant is highly accurate, comparing favorably against competitive baselines in model quality, and scalable, achieving very accurate quantization of Llama-3.1 405B, one of the largest open-source LLMs to date, using two Quadro RTX 8000-48GB GPUs in 21 hours. Our code is available at https://github.com/LeanModels/LeanQuant.

remain crit- Published as a conference paper at ICLR 2025
LEANQUANT: ACCURATE AND SCALABLE LARGE LANGUAGE MODEL QUANTIZATION WITH LOss-
AEROR-AWARD GRID
Tianyi Zhang
Anshumali Shrivastava
Dept. of Computer Science, Rice University
Dept. of Computer Science, Rice University
xMAD.ai
xMAD.ai
Houston, TX
ThirdAI Corp.
t tz srrrce.eee
Ken Kennedy Institute
Houston, TX
anshumali@ metmmali@rrcce...m edu
ABSTRACT
Large language models (LLMs) have shown immense potential across various domains, but their high memory requirements and inference ical challenges for deployment. Post-training quantization (PTQ) has emerged promising technique to reduce memory requirements and decoding latency. However, recent accurate quantization methods often depend on specialized com- putations or custom data formats achieve better model quality, which limits their compatibility with popular frameworks, as they require dedicated inference kernels tailored to specific hardware and software platforms, hindering wider adoption. Furthermore, many competitive methods have high resource require- ments and computational overhead for quantizing models, making it challenging scale them to hundreds of billions of parameters. In response to these chal- lenges, we propose LeanQuant (Loss-error-wwan network Quantization), a novel quantization method that is accurate, versatile, and scalable. In the existing pop- ular iterative loss-error-based quantization framework, we identify a critical lim- itation in prior methods: the min-max affine quantization grid fails preserve model quality due to outliers in inverse Hessian diagonals. To overcome this fundamental issue, we propose learning boss-error-awand grids, instead of using non-adaptive min-max affine grids. Our approach not only produces quantized models that are more accurate but also generalizes to a wider range of quanti- zation types, including affine and non-uniform quantization, enhancing compati- bility with more frameworks. Extensive experiments with recent LLMs demon- strate that LeanQuant is highly accurate, comparing favorably against compet- itive baselines in model quality, and scalable, achieving very accurate quanti- zation of Llama-3.1 405B, one of the largest open-source LLMs date, ing two Quadro RTX 8000-48GB GPUs in 21 hours. Our code available at https://githuut com M.aaom//TTeeMeeeee s Tea TuanQuant.
INTRODUCTION
Large language models (LLMs) have demonstrated impressive reasoning (Wei al.] 2022) and problem solving abilities (Kojima et al. 2022), and have shown the potential to bring transforma- tive changes various fields such as law (adddour al.) (023). education (Kasneci et al.) (023).
and medicine (Thirunavukarasu et al. 2023). However, deploying LLMs in a cost-effcetive manner presents significant challenges due to their substantial memory and computational demands (Chen
al. 2023). which hinders the accessibility and democratization of artificial intelligence (AI) (Kad-
dour et al. 2023
Post-training quantization (PTQ) Krishnamoorthi 2018) is a promising technique for reducing the memory footprint model inference by lowering the precision of a pre-trained model's parameters and storing them in compact, low-bit-width format. PTQ offers the additional benefit of reducing

---

Published as a conference paper at ICLR 2025
the decoding latency of LLMs by reducing memory reads, since LLM inference is often bottle- necked by memory bandwidth (Kim et al.) 2023). Although quantization causes a certain amount of precision loss in the parameters, the model quality can be reasonably preserved even in lower bit widths (Frantar et al. 2022; Chee et al. 2024). For many tasks, a quantized model is preferred over full model due its better size-aceuracy trade-off (Dettmers & Zettlmoyeer 2023). As open- source foundational models continue to scale up in size (Dubey et 2024). accurate and efficient quantization becomes essential for making AI accessible Llama-3.1 405B (Dubey et al. 2024) with its original 16-bit weights requires cluster of two nodes, each equipped with 8x80GB GPUs. In contrast, a 4-bit quantized version can be deployed on a single node with 8x48GB GPUs, eliminating inter-node communication overhead.
wider audience. For instance, serving
Challenges of Deploying Quantized Models One of the biggest challenges of successful deploy- ment of quantized models is implementing optimized kernels for quantized GEMM (general matrix multiply) that are tailored to various hardware platforms and software frameworks. In order to accel- erate inference of quantized models, fused kernels, which fuse dequantization and matrix multiplica- tion in the same subroutine, have to be implemented and tuned for the specific hardware accelerator. These kernels require specialized designs and tunings for different hardware accelerators to be fully optimized (Park et al.] 2022). Recent quantization algorithms have chosen to employ specialized computations or custom data formats to reduce the impact of quantization on model quality, but they require more sophisticated kernel designs for efficient inference. For example, AQLM (Egiazar- ian et al. 2024) and QUIP# (Tseng et al., 2024) perform dequantization through look-ups from multi-dimensionl or multi-bit codebooks, and Dotzel et al. (2024) proposed new data types such as Student Float to reduce quantization errors. While these approaches demonstrate promising re- sults, their reliance on specialized operations and data formats can hinder their widespread adoption due to the need for optimized inference kernels for each hardware platform and software framework. For example, llama.cpp (Gerganov] (003), a popular LLM inference engine that supports mobile de- vices, only supports affine and non-uniform quantization formats. Consequently, instead of focusing on developing better quantization methods with specialized operations, it may be more worthwhile to investigate improving the accuracy of existing widely adopted quantization formats, such as affine integer quantization and non-uniform quantization, which are supported by popular deep learning libraries (Paszke et al. 2019) and deployment frameworks (Kwon et al. 2023).
Scalability Challenges of Accurate Quantization To improve the quality of quantized models, isting approaches often incur higher computational overhead and require more hardware resources. As foundational models scale up in size (Hoffmann et al. 2022), these quantization approaches may struggle to scale to very large models, such as Llama-3. 405B (405 billion parameters) (Dubey al. 2024). For instance, LLM-QAT (Liu et al., 2023) uses 100K samples of training data and hundreds of GPU-hours to recover the performance of a quantized LLaMA-133 model (Touvron) ct al. 2023a). For AQLM (Egiazarian et al. 2024), the time needed for quantizing a 7B to 70B LLM ranges from to 14 days of an A100-80GB GPU. For SqueezeLLM (Kim et al. 2023), due to its use of the gradients of model parameters, quantizing a 70B LLM requires at least 240GB of total GPU memory. Given the significant hardware resources and lengthy optimization times of these quantization approaches, developing accurate yet efficient methods is crucial for ensuring accessibility of larger foundational models.
Our Proposal In this work, we propose LeanQuant, an accurate, versatile, and scalable quantization approach. We build upon the iterative cet-eerrorbbeedd quantization framework (Frantar & Alistarh 2022 Frantar et al. 2022) and identify one of the biggest limitations of such methods: the min-max affine quantization grid introduces high loss errors due to the existence of outliers in the inverse Hessian diagonals. We introduce techniques for learning loss-error-awane quantization grids, which mitigate this issue and greatly improve the quality of quantized models. We empirically demonstrate that LeanQuant compares favorably against competitive baselines in the 4/3/2-bit bit regions. Our ap- proach versatile, able to generalize to multiple commonly used quantization formats, such as affine and non-uniform quantization, allowing our quantized models to be directly compatible with existing highly optimized inference kernels (Frantar et al., 2024 Park et al., 2022) for maximum accessibility. Furthermore, our method is scalable and efficient. By designing and implementing a fused GPU kernel for LeanQuant grid learning, we achieve the accurate quantization of LLMs up to 123B in size using a single L40s-48GB GPU in 4 hours, and Llama-3.1 405B using 2 Quadro RTX 8000-48GB GPUs in 21 hours.
2

---

Published as a conference paper at ICLR 2025
2
BACKGROUND
In this section, we introduce the relevant background for our proposal including quantization grids and iterative cos--error-based quantization.
2.1 QUANTIZATION GRID
Quantization enables model compression by representing full-precision floating-point parameters with a limited set of grid points on quantization grid. The number of available grid points is de- termined by the bit width: a b-bit code allows for 2 distinct values, meaning 2-bit quantization results in four grid points. A visual explanation of quantization grid can be found Appendix [A] The placement of these points is crucial, as inaccurate placements can degrade model quality. To mitigate this, various quantization grids- such as affine, non-um con-unfform, and normal have been pro- posed. We provide an overview of these approaches below.
Affine Grid In an affine quantization grid (Krishnamoorthi 2018), the grid points are evenly spaced between the minimum and maximum of set of weights. To achieve finer quantization precision, the network's s weights are divided into groups (e.g., every 128 contiguous parameters). asymmetric affine quantization, each weight group is associated with scaling factor S and zero- point Z (with Z omitted in the symmetric case). The i-th weight w; in group w is quantized to a b-bit integer ii as follows:
min-max
Wi 5 + 1 Z, 1), where S = max min(w) and = min
w clip(1
2b
S
quant (w; S, (w. int 2)S
where 1-1 denotes rounding, clip(-) ensures the value remains within the b-bit integer range, and quant op (i, 2) is the quantized value of we.
Non-uniform Grid The grid points on a non-uniform grid are placed in a non-equidistant manner (Li et al., 2019). The motivation behind non non-uniform quantization is allow for finer precision in regions where model parameters are more concentrated or sensitive. Each row in weight matrix has distinct set of non-uniform grid points G. where |9| 2 for b-bit quantization. The weight
w; is quantized to the nearest grid point in G as follows,
quant nu (w,, G) arg minlg wal
9EG
Other Grid Types Previous works have observed that the distribution of LLM parameters often resembles Normal Student T's Distribution. Consequently, grid types such as NormalFloat (Dettmers et al., (024) and Student Float (Dotzel et al.] (2024) have been proposed, which align grid points with quantiles of these distributions. Our proposed method can be extended support them.
2.2 ITERATIVE LOSS-ERROR-BASED QUANTIZATION
Iterative loss-error- based quantization (Frantar & Alistarh 2022) a promising framework for quan- tizing deep neural networks to low bit widths while preserving model quality. In particular, Optimal Brain Quantization (OBQ) [Frantar & Alistarh 2022). which is based on the seminal works by Le- Cun et al. (1989) and Hassibi et al. (1993), aims to minimize the impact of weight perturbations introduced by parameter quantization on the network's task loss. Let L(wx) be the task loss of network V evaluated its weights wy (flattened to vector). Then, the OBQ objective is to minimize the loss error e, which is defined as
c= L(ww + 5w) L(ww)
where 5Nr is the weight perturbation introduced by quantization. The loss error can be approxi- mated with Taylor series (LeCun et al. 1989) as
0L OwN
8C Sx ((15vl1)
TT
2
Ow
negligible
nogligible
3

---

Published as a conference paper at ICLR 2025
where the first term is omitted due to own 2 0 in converged network, and the third and higher terms be ignored due to small norms. Computing the exact Hessian H
ac
8 c 8w 2 in deep network
difficult, hence OBQ leverages an approximation of loss error proposed by Nagel et al. (2020).
B(c) x
Σ WEN
|| - W
where W, W, X are the weight matrix, quantized weight matrix, and the input matrix to a linear layer in the network N As a result, the OBQ objective can be decomposed into layer-wise indepen- dent convex problems,
arg min WX- W
W
which can be further decomposed into row-wise independent problems, since Equation written as a sum of squares over the rows of W.
| can be
OBQ employs an iterative quantization approach, in which a single weight in a row w is quantized in each step, and then the remaining not-yet-quantized weights in the same row are updated to compensate for the introduced error. Given the constraint that the parameter w,, indexed by i in row w, is being quantized, the optimal weight perturbation to the remaining weights can solved with the following Lagrangian,
L(8, -
T HÒ
X x 8 (quant(w:: - w:)
where e; is the i-th standard basis vector and H = 2xX is the Hessian from Equation 1 (computed on small sample of input data). Solving Equation 2 yields the optimal weight perturbation 6; and loss eπror after quantizing wi.
8: quant (.)) H
1 (quantt (w:) - w;)
6,
where H and H. denotes the i-th diagonal entry and the i-th column of the inverse Hessian, respectively.
The loss error quantifies the degradation in model quality caused by quantizing parameter w, and always non-negative. OBQ leverages e, as heuristic for greedy optimization. In each iteration, OBQ computes for all weights in a row and greedily selects the parameter with the smallest G for quantization. The selected parameter is then rounded to the nearest value on the quantization grid, and the remaining weights are updated as w < w 5, This iterative process continues until all weights are quantized.
Scaling to Billion-Parameter LLMs Using Cholesky and Dampening OBQ produces accu- rate post-training quantized models for calllion-prrmmteer networks, but fails to scale billion- parameter LLMs due to two primary reasons: the inefficient time complexity and the accumulation of numerical inaccuracies during updates. To improve its computational efficiency, Frantar et al. (2022) propose to quantize the weights in a fixed non-greedy order for all rows, and keep the weight updates within a block B columns at time. To prevent model quality collapse from the accumu- lation of numerical inaccuracies by repeated weight updates, Frantar et al. (2022) propose to apply a mild dampening (1% % of the average diagonals) to the diagonal Hessian and leverage a Cholesky decomposition compute the inverse Hessian H-1 The resulting algorithm is GPTQ, which can efficiently quantize billion- billion-paramenn parameter LLMs.
3 METHODOLOGY
In this section, we introduce our proposed approach Loss-error-awarg network Quantization (Lean- Quant), for accurately and efficiently quantizing LLMs.
3.1 REVISITING THE Loss ERROR
To motivate our proposed approach, we first revisit the loss error e; in Equation 3 which approx- imates the (detrimental) increase in the network's task loss, introduced by quantizing weight w;-
4

---

Published as a conference paper at ICLR 2025
Liama-3-88
Liama-3-00
Loyer MLP Linear
Layer MLPUpp Weights
23"
digght-
Outtiers, which
1.0
0.
Lagee
10
Weight
Frequenicy (lgg
Min-max Affine Quantization Grid
Altention
-0/04
-002
0.00
Loss-error-wware
Non-uniform Grid (ours)
Learned Equation (5)
3.26
Layol 90 Aourrtor-Quurry Linear
0,22
0.00
012
337-
Loss-error-wware Affine Grid (ours)
Leamed using Equation (71
lused GPU kernel
0.04
0:10
Value Sug
Outfien, oP3 may ligh quanitationn
Figure 1: (Left) The empirical distributions of inverse Hessian diagonals, computed on 262K tokens from the C4 dataset for the Alama-3-8B model, contain outliers that can cause high loss errors. (Right) Our proposed loss-error-aware non-uniform and affine grids better preserve the quantized precision outliers, leading to more accurate quantized models.
This error c has been used as heuristic multiple previous works (LeCun et al., 1989 Hassibi ett al. 1993 Singh & Alistarh 2020 Frantar & Alistarh, 2022) for choosing the next best weight to prune or quantize. has been shown to be highly informative metric for measuring the impact of quantization.
By examining Equation| 3 one finds that the loss enror 6, is proportional to the square of weight quantization error and inversely proportional to the diagonal entry of the inverse Hessian, i.e.
1
(quant(ws) w:)² and 4
H
Hence, we further examine the empirical distribution of Hi which is proportional to c, the loss error of entire row. We obtain the empirical distributions on layers of Llama-3-80 (Dubey ctt al. 2024) with 128 sequences of length 2048 tokens from the C4 dataset (Raffel et al,, (020), and compute the inverse Hessian H (2XX )-1 where X is the layer input matrix. As shown in Figure [ The majority the inverse diagonals are concentrated in low-magnitude regions, with few outliers having high magnitudes. Quantizing the weights corresponding these outliers can lead to high loss errors if these weights are not well-aligned with the quantization grid points. Preserving the quantized precision of the weights corresponding these inverse-diagonal outliers is especially important because the loss егтог increases quadratically with their quantization error (Equation 44). Iterative loss-error-based quantization approaches (OBQ, GPTQ, etc.) employ min- max affine quantization grid, which is suboptimal for preserving the quantized precision of the livers-diagooal outliers, leading to high loss errors and model quality degradation. Our idea is to learn quantization grids that minimize the loss error
3.2
LOSS-ERROR-AWAN NETWORK QUANTIZATION
Existing iterative loss-error-baase based guantization methods rely on min mmx affine grids, which fail to account for outliers in the inverse Hessian diagonals. These outliers cause significant degrada- tion model quality. To address this limitation, we propose loss-error-aware quantization grids that preserve the precision of weights corresponding these outliers, thereby improving model quality, Our approach introduces techniques for learning loss-error-awart grids across various quantization formats, including non-umi con-uniform and affine. Additionally, accelerate grid learning for large models, we developed fused GPU kernels that enable efficient and scalable quantization.
3.2.1 NON-UNIFORM Loss- SOISS-EROR--E SERRR--WWRR GRID
For non-uniform quantization, we perform clustering on the model parameters, weighted by their corresponding exponentiated inverse Hessian diagonals, derive set of loss-error-wware grid
5

---

Published as a conference paper at ICLR 2025
points. The proposed objective aims to shape the learned grid to minimize quantization error for weights corresponding to avvese-diagonal outliers, as these outliers can disproportionately affect model quality. Concretely, we determine the set of grid points & for b-bit quantization by optimizing the following objective:
arg min Σ ) Iquant (w,, 9) uy
(5)
-:19|=2*
Here, p hyperparameter that balances the strength of precision preservation between inverse- diagonal outliers and non-outliers. Higher values of prioritize the precision preservation of outliers, while p treats all weights equally. In our experiments, we set p 4 for all models. A sensitivity analysis for p is provided in Section 4.3 To optimize this objective, we employ the k-means algorithm ([loyd] 1982). incorporating careful centroid initialization as described below. Once the quantization grid G is established, the weights are iteratively quantized to the nearest grid points within G.
Grid Initialization The quality of clustering relies heavily on initialization (Arthur et al. 2007).
as Lloyd's Algorithm (Lloyd, 1982) converges to a locally optimal solution. This sensitivity is especially critical in low-bit-we sea-bbitwiit bit-width settings (3-bit or 2-bit), where standard methods like random or k-means++ (Arthur et al. (007) often undersample extreme values due to the distribution of weights, which are densely concentrated near the center and sparse at the extremes.
To address this, we propose uniformly spaced grid initialization, which evenly spaces initial grid points between the minimum and maximum weights ensure both central regions and extremes well represented. The initial grid points for clustering, Gma. are defined as:
Gimi = min(w) + max w) min(w (w),
(o....., 2 13}
(6)
26
This lightweight and robust initialization improves representation across the entire range of weights. We present an ablation study confirm its effectiveness in Table [5]in the Appendix.
3.2.2 LOSS-ERROR-AWARE AFFINE GRID
The goal of learning an affine grid is determine an optimal scaling factor S and zero-point Z that minimize the loss error. Unlike non-uniform grids, where clustering strategies can be applied, affine grids require the grid points to be uniformly spaced over an interval, making clustering- proaches inapplicable. While gradient descent could be used to learn S and Z, it is computationally intensive, memory-demandinn
ap-
and susceptible to local minima.
To address this challenge, we adopt an enumerative search approach to learn the affine grid. Specif- ically, we enumerate candidate pairs of S and Z from a constrained search space and select the pair that minimizes the following objective, which is similar to Equation 5
(W1, arg min Σ (H (- quant (W:, S, 2) - Wi where
(8,2)
8 (max(w)- frnaxe 7)
(mnn(w w) Emin 4)
min(w) (w)
R + fyin T
fimis tmax E (0.....9) (7)
2
scaling factor
zaro point 2
Here, = max (w)-min(w)) (w) is the range of the weights, T is the number of partitions within Rt and t E (1,.... ... T is the number of partitions to enumerate over. By iteratively enumerating candidates of S and Z and evaluating their corresponding losses, we identify the optimal pair that minimizes the loss error. The parameter T' determines the granularity of the search; in our experiments, we set T 2048. To prevent overfitting, t controls the amount of shrinkage of the range, which explain in Appendix B
2
Efficient Fused GPU Kernel for Grid Learning The enumerative search for S and Z involves evaluating t² candidate pairs, which can be computationally expensive if performed sequentially. To accelerate this process, design and implement fused GPU kernel that leverages parallel

---

Published as a conference paper at ICLR 2025
Algorithm LuanQ for LLM quantization
Imput: weight
Rr*se imput matrix X. bit width block size dampening factor outlier preservation strength p Output: Quantized matrix W
1: W Orxe
2: E < D
3: H + 2XX
4: H +- Cholesky [H + df avR(ding(H)) 1)-
D apply dampening, inversion, and Cholesky decomposition
5: using non-umiform grid then
6: Go + arg min (diag(H " P) Iquant W.,., 9) Wa.: forall & (u.....r- r- 1}
b
9:101-2"
7:
else if using affine grid then
8: Sk. Zu arg min (diag(H 2) ") Iquant W.... W... forall & E (0,...,-- > apply block wise quantization
(8,2)E5
9:
end
0, 2B. do 11: for ...... B do
for i
12:
if using non-uniform grid then
W + quant o (Wr.5. St) forall ( (O....,r- else if using affine grid then
b quantize to non uniform grid
WE. J + quant Wa,5, Sa, Ze) forall E (U....,T-
quamtize to afline grid
16:
end if
17:
E
W. -W. W
3.3
18:
W W..:t 2)
W W. E.g-1 H. Hast
19:
end for
20:
(.++0) W (((++)) E. H
21:
end for
22:
return W
processing. Each thread block is assigned group of weights, and individual threads within the block evaluate all combinations of specific fmin and all possible fmax. The threads compute the loss for their assigned combinations, and the results are aggregated at the block level to determine the optimal S and Z for the weight group. This parallelized approach enables simultaneous computation of S and Z across all weight groups, achieving speedup of over 50x for the end-to-end quantization process. An analysis of the kernel's efficiency presented Section 4.3
3.2.3 LEANQUANT
Our proposed loss-error-aware quantization grid can be seamlessly integrated with any iterative loss- error-based quantization method to enhance the quality of quantized models. Figure [ illustrates a comparison between the min-max affine quantization grid and loss-error-aware grids (both non- uniform and affine) applied a layer of Llama-3-88 (Dubey et al. 2024). We introduce LeanQuant, which combines loss-error-aware grids with GPTQ (Frantar et al., 2022), and detail the method in Algorithm Additionally, for quantizing which integrates loss-error-aware grids with OBQ (Frantar & Alistarh 2022), with details presented in Algorithm?] 2 in the Appendix. To specify the grid type used within Lean- Quant, subscripts such LeanQuantag for affine and LeanQuant,, for non-uniform grids.
models more accurately, we propose
4 EXPERIMENTS
We conduct extensive experiments validate LeanQuant's effectiveness and scalability in LLM quantization against competitive baselines. We first introduce the baselines, models, evaluation metries, datasets, and hardware. Then, we present results, analyze efficiency and scalability, and conduct ablation studies to further validate our approach.
Baselines We compare LeanQuant.g against competitive affine quantization approaches AWQ (Lin et al.) 2024), GPTQ (Frantar et al. (002). and OmniQuant (Shao et al., 2024). and LeanQuant against the existing state-of-the-art non-uniform method SqueeZeLLM (Kim et al., 2023). For the baselines, we the quantized models provided by their official repository where possible, and quantize the unavailable models using their official codebase and recommended hyperparameters. More details baseline reproduction and evaluation methods Appendix. For LcanQuant models, we use a small calibration set of 128 sequences 2048 tokens from the C4 dataset (Raffel al. 2020) for computing the Hessian H. and set p 4.
be found in Section E of the

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot accuracy of quantized L.MSs benchmarks. The results of more models can be found in Table 10 of the Appendix. '2-bit quantization is unsupported by the SqueezeLLM codebase.
Method
Bits
ARC
LAMBADA
MMLU
HellaS
PIQA
WinoG
Avg.
Easy
Chg
Sid
OpenAl
STEM
Human.
Social
Other
Llama-3-8B
BF16
16
80.30
50.17
68.85
75.82
53.82
54.88
73.29
70.42
60.11
79.71
73.56
67.36
GPTQ
4,00
74.83
14.11
63.42
70.75
47.29
52.28
66.04
64.89
57.98
77.26
71.82
61.58
OmnniQuant
4.00
76.89
47.35
61.05
69.16
49.38
49.05
66.62
64.40
58.25
78.84
71.98
63.00
LeanQuantar
4.00
76.60
46.93
66.89
74.07
51.89
52.96
70.04
68.43
58.47
77.91
72.77
65.18
Affine
GPTQ
3.00
50.84
24.32
24.16
38.89
26.23
29.16
34.38
30.00
45.07
64.64
60.89
37.75
OmiiQQuant
3.00
60.90
30.12
21 .0.0
27.63
26.32
27.80
29.51
29.90
46.
68.17
59.9%
38.95
LeanQuantar
3.00
69.44
35.75
46.81
65.42
42.59
44.78
58.17
56.97
52.72
74.86
69.93
56.13
GPTQ
2,00
25.46
22.53
0.00
0.00
21.06
23.95
21.16
23.78
25.66
52.77
51.34
24.25
OriniQuant
2,00
26.81
21.67
0.00
0.00
24.21
21.71
47.43
24.26
TeanQuant,r"
2,00
35.06
18.26
11.33
14.71
21.31
27.17
21.71
24.01
31.43
59.30
51.85
28.47
Non unffomm
SqueeseLLMM
4.05
79.59
49.32
18
73.24
53.32
70.78
59.10
73,80
TeanQuantav
4.05
79.50
49.15
67.36
74.95
52.17
53.16
71.40
68.75
59.19
78.89
74.11
66.24
SqusezeLLM
3.02
73.19
43.52
58.22
66.58
43.61
46.57
61.91
60:03
56.17
77.64
69.
59.70
LearQuantas
3.02
77.74
47.01
63.32
72.17
48.84
49.05
65.45
62.79
56,42
78.24
71.67
62.97
SqueezeL.LM
2.01
-N/A-
LeanQuantt
2.01
58.21
26.62
31.22
39.16
25.98
25.48
27.01
26.65
40,78
68.01
60.38
39.05
Llama-2-7B
FP16
16
76.26
43.43
68.33
73.88
34.38
39.79
47.32
47.12
57.10
78.07
68.98
57.70
GPTQ
4.00
74.16
40.78
65.38
71.94
32.67
36.92
42.61
42.61
55.99
77.48
68.32
53.47
OmniiQuant
4.00
74.12
40.70
64.10
70.62
28.80
32.18
34.71
35.79
55.37
76,93
68.67
52.91
LuannQuantay
4.00
75.00
41.21
72.02
36.94
46.77
44.54
55.32
77.15
68.75
56.14
Affine
GPTQ
3,00
66.29
"4.222
46.46
58.
28.20
26.99
32.11
29.90
49.05
73.23
62.83
42.12
OmniQuant
3.00
70.12
37.29
53.27
66.66
29.05
31.05
30.61
30.38
52.58
74.05
66.46
49.23
LearQuantar
3.00
69.28
37.12
59.77
67.73
30.32
30.22
35.26
33.34
50.59
74.81
66.14
50.42
GPTQ
2,00
25.97
21.67
0.00
0,00
21.31
23.25
21.11
23.01
25.76
51.74
48.78
23.66
OnniQQuant
2,00
37.42
21.76
1.28
3.24
21.47
24.14
21.74
23.91
29.59
$7.18
51.93
26.70
LeanQuant,a"
2.00
41.08
20.99
16.98
21.93
21.25
24.06
21.77
23.88
31.94
61.64
56.51
31.09
Non- unifoom
SqueezeLLM
4.05
75.59
41.9%
72.79
38.94
44.96
56.77
LeanQuant,.
4.05
75.97
42.66
68. 14
74.25
34.35
39.06
46.05
46.51
56.03
77.86
69.38
57.30
Sq
3.02
73.06
40.27
61.96
71
33.75
35.22
43.35
43.
54.15
76.50
67.6
54.49
LeanQuantt.
3.02
73.74
40.19
66.12
73.16
32.25
35.54
43.40
43.39
53.24
76.44
68.35
55.07
Squeezel
2.01
-N/A-
LeanQuant,s
2.01
51.81
23.98
28.68
38.21
22.26
23.89
22.49
24.01
35,88
66.38
58.17
35.98
Mistral-7B
BF16
16
80.77
50.09
69.38
75.63
50.46
53.48
69.35
68.01
61.26
80.58
73.88
66.62
GPTQ
4.00
79.00
46.25
66. 99
73.67
46.24
50.82
66.20
64.66
59.36
79.65
72.93
62. 68
OmniiQuant
4.00
78.49
46.25
63.28
71.20
45.96
$1.35
65 68
64.76
60.19
79.87
71.90
63.54
LeanQuaatt.r
4.00
79.71
48.04
75.70
51.84
68.05
66.43
59.65
73.48
65.37
Affnn
GPTQ
3,00
70.54
38.65
52.63
62.10
36.31
38.89
49:20
47.86
54.76
77.58
67.96
$2.60
OmniQQunt
3.00
70.54
35.07
35, 49
46.54
33.71
32.88
40:23
37.
52.35
75.19
63.93
47.62
LearQuantay
3.00
77.65
44.71
60.51
71.94
43.99
46.14
60.97
59.35
55.61
78.51
71.59
61.00
GPTQ
2,00
26.73
22.27
0.00
0.00
23.31
24.46
23.86
23.42
25.35
51.52
49.72
24.39
OmniQuant
2.00
27.06
21.67
0.00
0.00
21.25
24.29
21.71
23.98
25.89
51.25
51.54
LeanQuant,
2.00
56.02
28.33
19.23
23.17
23.72
24.48
24.21
25.36
34.45
62.57
57.14
34.43
Non- unifom
SqueezeLLM
4.05
79.73
28
74.93
$2.73
68.87
66.98
59.80
73.56
63.73
LeanQuant.,,
4.05
79.80
48. 89
69.03
76.03
48.84
$2.86
68.87
66.69
60.19
80. 14
74.59
65.99
SM
3.02
77.54
45.93
64.00
71.43
43.96
47.93
62.69
59.
58.76
79.43
71.98
62.08
LeanQuant..
3.02
77.74
45,99
67.59
76.07
44.24
47.97
62.14
62.47
57.28
79.27
72.22
63.00
SqueezeLLM
2.01
.NAA
LeanQuant,a
2.01
63.47
30.55
41.01
$4.61
31.34
29.97
32.14
33.96
42.29
71.38
64.01
44.97
Models We consider the following recent, popular LLMs for quantization: Llama 1/2/3 series mod-
els (Touvron et al. 2023a b) Dubey et al. (024). Mistral-7B-vO. (Jiang et al. 2023). Mistral-Large
In5truct-2407 (123B) (Mistral Al Team 2024), and Llama-3. 3. 1 IIntruutt (Dubey et al. 2024).
Evaluation Metrics and Datasets We evaluate quantized LLMs using the perplexity metric on the datasets WikiText2 (Merity et al., 2016) and C4 (Raffel et al., 2020). and zero-shot accuracy on the benchmarks ARC (Clark et al. | (018)) LAMBADA (Paperno et al. (0166), MMLU (Hendrycks et al. 2020 HellaSwag (Zellers et al.]] 2019), PIQA (Bisk et al. 2020), and WinoGrande (Sakaguchi et al.,
2021 We also quantize and evaluate the -estruction-folloiing Llama-3-8B ---Intrruct using OpenAl
GPT-4o (2024-05-13) as a judge on the MT-Bench (Zheng et al. 2023). and the results are presented
in Section G in the Appendix.
Testbed Hardware LeanQuant models are quantized using a machine quipped with an L40s-48GB GPU, an AMD EPYC 7R13 48-Core CPU, and 370GB of RAM. To fit Llama-3.1 --0oa....---------ss----- in RAM, which is around 800GB in size, we use a machine equipped with 2 Quadro RTX 8000 GPUs, an AMD EPYC 7742 64-Core CPU, and 1.48TB of RAM.
8

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot accuracy of the quantized 123B MistraH-argge-nssruue--4077 model.
LAMRADA
MMLU
Model
Method
Bits
Easy
Chg.
Std.
OpenAl
STEM
Human.
Social
Other
Avg.
GPTQ
80.52
77.23
85.23
78.95
Mistral-I Ingee-nstreeet--2 20srm
LeanQuant,rr
4,00
85.14
63.99
74.99
81.14
76.56
77.32
89.21
85.87
79.28
LeanQuant,
87.67
64.59
76.63
81.51
76.50
78.00
89.35
85.08
79.99
Table 3: Zero-shot accuracy of the quantized Llama-3 3.11-415B-Insttie model.
Model
Method
Group
Easy
Are
Clg.
Std.
LAMBADA
OpenAl
STEM
Human.
MMLU
Social
Other
Avg.
GPTQ
128
4.25
88.21
65,10
73.41
76.96
82.34
82.64
90.45
87.51
80.83
Llame-3.1
LeanQuanter
4.25
88.26
64.76
73.32
77.08
82.68
83.21
90.58
87.71
80.95
LeanQuant...
4.05
88.22
63.65
74.56
78.52
82.52
83.40
90.51
87.64
81.13
4.1
MAIN RESULTS
Accuracy and Perplexity The zero-shot accuracy of quantized models on benchmarks are presented Table well as in Table i0 in the Appendix, and the perplexity results are shown Table in the Appendix. At the same bit width, LeanQuant achieves significantly better (lower) perplexity than GPTQ and AWQ, and performs on par with OmniQuant and SqueezeL.MM, However, perplexity may not be a representative metric for evaluating the accuracy of quantized models. In terms of zero- shot accuracy on various benchmarks, LeanQuantarf mostly outperforms GPTQ and OmniQuant, and LeanQuant,, similarly performs better than SqueezeLLM in most cases. We highlight that LeanQuantag improves the average zero-shot accuracy on tasks over OmniQuant by 17.18% for 3-bit Llama-3-8B, and by 13.38% for 3-bit Mistral-7B. Compared to GPTQ, LeanQuantar improves the average zero-shot accuracy by 18.38% for 3-bit Llama-3-30 3-88, and by 8.40% for 3-bit Mistral-7B.
Effectiveness Very Large LLMs We quantize the 123B Mistral-Large- Instructttt and the 405B Llama-3.1 model using LeanQuanteg: LeanQuanteu. and GPTQ, and present their zero-shot accuracy in Table / and respectively. OmniQuant and SqueezeLLM fail quantize these models due to GPU out-of-memory errors. LeanQuant models mostly outperform GPTQ in zero- shot accuracy. For affine quantization, we employ row-wise quantization for Mistral-Larg. and group-wise quantization (with size 128) for Llama-3.1 405B. This showcases that our method is effective for both row-wise and group-wise quantization.
4.2 MEMORY AND TIME EFFICIENCY
We report the maximum GPU memory consumption of LeanQuant and the baselines during quanti- zation on models of different sizes in Table LeanQuant is significantly more memory efficient than OmniQuant and SqueezeLLM: it successfully scales to 123B Mistral-Large using a single 48GB GPU, and to 405B Llama-3.1 models using two 48GB GPUs, while OmniQuant fails to quantize Llama-3-70B and LL fails to quantize Llama-3-8B on single 48GB GPU. The time cost of LuanQuant for different sized models are reported in Table 13 in the Appendix. LeanQuant can quantize 7B/8B models in less than an hour, the 123B model in 4.2 hours, and the 405B model in 20.7 hours.
4.3 ABLATION STUDY
Does LeanQuant effectively reduce the loss error compared to other iterative loss-error- based methods? Yes, LeanQuant effectively reduces loss errors compared to GPTQ, as shown in Figure 2 well in Figure 5 in the Appendix. The sum of loss errors computed Equation 3 Moreover, non-uniform LeanQuant generally achieves lower loss errors than affine LeanQuant, due to more degrees of freedom in the grid point placements, which also explains why LeanQuant,.. achieves higher accuracy than LeanQuant,g on benchmarks in Table
Q1:
Q2: Is LeanQuant sensitive to the hyperparameter p? No, found LeanQuant be not very sensitive to p. sensitivity analysis on the hyperparameter p is given in Table 14 in the Appendix. LeanQuant works well with p values of 3 or 4.

---

Published as a conference paper at ICLR 2025
GPTQ
LeanQuiitaar
LnanQuarts.
Liama-3-88 (4-br))
Llama-3-88 (3-bR)
Llama-3-45 3-B8--3-----88 (2-br))
15000
500000
230000
00000
200000
Sun Loes Enore
500000
26000
100000
Mistral-78 (4-a1)
Mistral-88
Mistral-78
100000
430000
50000
Oreeiin Kay
Amenicn Door
mennap-cco Cue
Sate Cake
mendowstyy Kay
Atecion Kabe
Paeriza Qoorr
Anente Dul
mrnnon Aay
martzon Vade
seectet Doer
mentow Da
Figure 2: Comparison of loss errors 6, summed over each layer, for GPTQ and LeanQuant (affine and non-uniform) during iterative quantization.
Q3: Is uniformly spaced grid initialization beneficial for model quality? Yes, uniformly spaced grid initialization consistently outperforms k-means++ (Arthur et al. 2007) initialization on different models in 3-bit and 2-bit regions, shown in Table 15 the Appendix.
Q4: Does the fused GPU kernel for LeanQuantar accelerate quantization? Yes, our fused kernel for learning affine grids accelerate the end-to-end quantization process by more than 50x, as shown in 3 which enables LeanQuant to be scaled to very large models.
Table 4: Peak GPU memory consumption of differ- Table 5: Comparison of total time needed for ent algorithms during 4-bit quantization. "OOM" quantizing Llama-3-8B with and without our indicates out of memory on single 48GB GPU, fused kernel for loss-errow ewwrrrar-- affine grid except for Llama-3.1-405B where we use 2 48GB learning.
GPUs.
Fused Kemel
Group Size
Bits
Quant. Time
Model
OmniQuant Squeezel.IM
GPTQ
LeanQuant
4.00
Llama-3-S8 Llama 70B
25.3 GB
0OM
GB
GB
4.00
0.27 hes
OOM
0OM
17.1 GB
17.2 GB
Mistral-L Large (1238)
0OM
0OM
328
GB
128
4.25
>> 100 hrs
Llama-3.1--A0 1-405B
0OM
0OM
0OM
65
128
4.25
0.40 hes
5 RELATED WORKS
Iterative Loss-error-based Compression Optimal Brain Damage (LeCun et al., 1989) introduced
saliency-score-basee iterative pruning algorithm for neural networks, and Optimal Brain Surgeon
(Hassibi & Stork 1992 Hassibi et al. 1993) extended to apply a weight update to compensate for
the error introduced in each iteration. These methods inspired number of works on model pruning
(Guo et al., 2016] Singh & Alistarh 2020 Yu et al. 2022) and weight quantization (Li et al. 2021
Frantar & Alistarh 2022 Frantar al.) 2022).
Efficient LLM Inference LLM inference is computationally and memory demanding, and existing
works explore improving inference efficiency through quantization (Dettmers et al. 2022b Lin
et al.
2024; Frantar et al.) 2022 Chee et al., 2024) Kim et al. 2023 Shao et al.] 2024 Hgiazarian
al.
2024 Tseng al. 2024), pruning (Frantar & Alistarh| [2033] Ashkboos 2024). weight-
activation quantization (Xiao et al. (2023), offloading Sheng et al. (2023), etc. A survey of more
relevant literature can be found in Appendix M
CONCLUSION
In this work, we propose LeanQuant, an accurate, versatile, and scalable quantization method for LLMs. Motivated by the finding that the min-max affine grid causes large errors the network's task loss in iterative loss-error-based methods, we propose to learn loss-error-aware grids to enable more accurate quantized models, and design fused kernels for efficient and scalable quantization. Our method generalizes multiple quantization formats enable greater accessibility. Exten- sive empirical evaluations reveal that our quantized models compares favorably against competitive baselines in accuracy, and scale to 1lama-3.1 405B, one of the largest open-source L.LM to date.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by National Science Foundation SHF-2211815 and Ken Kennedy Institute Cluster Grants.
REFERENCES
David Arthur, Sergei Vassilvitskii, et al. k-means++: The advantages of careful seeding. In Soda, volume 7, pp. 1027-1035, 2007.
Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024, 2024.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213- 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.055/ 07700,055767,,. 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.0:800.00444 2018.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35:16344-16358 2022.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pp. 7750-7774, PMLR, 2023.
Tim Dettmers, Ruslan A Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: sparse-quantived representation for near-lossless I1m weight compression. In The Twelfth International Conference on Learning Representations.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q) 8-bit matrix multiplication for transformers scale. Advances in neural information processing systems, 35: 00318-30332, 2022a.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-30332, 2022b.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in Neural Information Processing Systems, 36. 2024.
Jacob Devlin, Ming- Wing Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Jordan Dotzel, Yuzong Chen, Bahaa Kotb, Sushma Prasad, Gang Wu, Sheng Li, Mohamed Ab- delfattah, and Zhiru Zhang. Learning from students: Applying t-distributions to explore accurate and efficient formats for llms. arXiv preprint arXiv:2405.088 03/00/10333 103, 2024.

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXhv:240..17783, 2024.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.0611 18, 2024.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 09223-1037.. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. /7323, 2022.
Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed-precision auto-regressive parallel inference on large language models. arXiv preprint arXiv:2408. 11743, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL/httts:///enooe ortp recr///ecrd/////
10256836
Georgi Gerganov. llama.cpp. cessed: 2024-10-01.
2023. Ac-
Shinya Gongyo, Jinrong Liang Mitsuru Ambai, Rei Kawakami, and Ikuro Sato. Learning non- uniform step sizes for neural network quantization. Proceedings of the Asian Conference on Computer Vision, pp. 4385-4402, 2024.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEEE, 1993.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 155566 2022.
Yongkweon Jeon, Chungman Lee, Fulrang Cho, and Yeonju Ro. Mr. biq: Post-training non-uniform quantization based on minimizing the reconstruction error. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12329-12338, 2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307. 10169, 2023.
12

---

Published as a conference paper at ICLR 2025
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103: 102274, 2023.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2221 222113,, 2022.
Raghuraman Krishnamoorthi.
Quantizing deep convolutional networks for efficient inference: A 2018.
whitepaper. arXiv preprini
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint wrXiv:223.00255) 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 1-66,66 2023.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non- uniform discretization for neural networks. In International Conference on Learning Representa- fions.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient uniform discretization for neural networks. arXiv preprint 4rXiv:1909.13144 2019.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint 11Xv:2102.05426. 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Zechun Liu, Kwang- Twang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nonuniform-to uniform quantization: Towards accurate quantization via generalized straight-through estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp-
4942 4952, 2022.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emnie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free quantization aware training for large language models. arXiv preprint 2xXiv:2305.77888 2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Forty-first International Conference Machine Learning.
Stuart Lloyd. Least squares quantization pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Shuming Ma. Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint a 202. 1, 2024.
13

---

Published as a conference paper at ICLR 2025
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models. arXiv preprint wwXiv:1609007844 2016.
Mistral AI Team. Mistral large 2, July 26 2024. URL intps::/mistrallaaaaaan
mistraa--arge-
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort, Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning. pp. 7197-7206. PMLR, 2020.
Sangyun Oh, Hyeonuk Sim, Jounghyun Kim, and Jongeun Lee. Non-uniform step size quantization for accurate post-training quantization. In European Conference Computer Vision, pp. 658- 673. Springer, 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint wrXiv:1606.00033 2016.
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Bycongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix mul- tiplication based on luts for efficient inference in large-scale generative language models. arXiv preprint X2vv:2266.09557, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(14 167, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint /606.066000005500 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9)999-106 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024.
http::///ooeeeeeeeeeeeee net/ wttp://wwww..ttttttmttmmocce....
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp- 31094-30 31116. PMLR, 2023.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-18109 18109, 2020.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models medicine. Nature medicine, 1900-9990 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wrXiv:2307.09288. 2023b.
14

---

Published as a conference paper at ICLR 2025
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipf:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint 04X1i:24002.43396, 2024.
Turboderp-org.
Exilamav2: fast inference library for running ilms locally modern consumer- class gpus.
com htroo//tuuoorpppoooo org/ex.////.ooooo amav2 2024. Accessed:
2025-02-28.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang. Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. Advances Neural Information Processing Systems, 0166657-66382, 2025.
Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: pruning weights that cancel one another in neural networks. In International Conference on Machine Learning, pp. 25668 25683.. PMLR, 2022.
Edouard Yvinec, Arnaud Dapogny, and Kevin Bailly. Nupes: Non-uniform post-training quantiza- tion via power exponent search. arXiv preprint arXiv:2308.0550 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv: 1905,00990 2019.
Haochen Zhang, Junze Yin, Guanchu Wang, Zirui Liu, Tianyi Zhang, Anshumali Shrivastava, Lin Yang, and Vladimir Braverman. 13s: Importance sampling subspace selection for low-rank opti- mization in Ilm pretraining. arXiv preprint 202.:2200..05790 2025a.
Tianyi Zhang, Junda Su, Aditya Desai, Oscar Wu, Zhaozhuo Xu, and Anshumali Shrivastava. Sketch to adapt: Fine-tunable sketches for efficient Ilm adaptation, 2025b. URL https://arxiv.
06364
Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is bit per chan- nel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304-3331, 2025c.
Tianyi Zhang, Jonah Yi, Bowen Yao, Zhaozhuo Xu, and Anshumali Shrivastava. Nomad-attention: Efficient Ilm inference on cpus through multiply-add-fiee attention. Advances in Neural Informa- tion Processing Systems, 37: 10111-2212222500 2025d.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. In International Conference on Machine Learning, pp. 61 621--11,4 PMLR, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4662 2023.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXPLANATIONS ON QUANTIZATION GRID
Affine Liama-3-88 MLP-Down Weights
Non-uniform Grid
Frequency (og scale)
10a
10:
107
-0.02
100
0.07
04
.00
0
0.02
04
Weight Values
Weight
Figure 3: Comparison of affine (left) and non-uniform (right) 2-bit quantization grids applied to the weights in the first MLP--own layer of Llama-3-8B. The affine grid uses evenly spaced quantization grid points between the minimum and maximum weights. In contrast, the non-uniform grid allows grid points be placed flexibly, their positions stored in look-up table. This enables finer quantization in dense regions and coarser quantization in sparse regions, better aligning with the weight distribution and reducing quantization error.
In the context of quantization, a grid is predefined set of values representing the possible quantized outputs for full-precision parameters. During quantization, each full-precision parameter mapped to its nearest grid point on the quantization grid. For example, in 2-bit quantization scheme with grid points { (-1.0, 0..33, 0.33, 1.0), floating-Pol -oiit weight of 0.25 would be assigned to 0.33, the closest grid point.
Affine Quantization Grid An affine quantization grid distributes points uniformly across the range of the weights being quantized. The dynamic range of the weights, defined as [W mins W Waxx] determines the spacing of the grid points. For example, if [W W max] = [- 1.0, 1.0] in 2-bit quantization setting, the grid points would be evenly spaced at 1.00 0..3, 0.33, 1.0. This uniform
distribution is computationally simple and widely used in practice. but may lead to suboptimal precision when the weight distribution is non-uniform, as many grid points may underutilized.
Non-uniform Quantization Grid Non-uniform grids allocate grid points more flexibly, allowing denser spacing in bigh-probability regions of the weight distribution and sparser spacing in low- probability regions. This approach minimizes quantization error by adapting the grid the data distribution. Non-uniform grids typically store the grid points in a look-up table, enabling flexible placement that better represents the original data. Figure BegureBilluster iillustrates an example affine grid and non-uniform grid applied to the weights of Llama-3-8B.
Grouped Quantization The quantization grid for a set of weights is determined by the range [Wiinn W within the group. Smaller group sizes allow for narrower dynamic range, leading to finer granularity in the quantization grid and higher precision. Grouping contiguous weights into
blocks is a common practice in quantization literature (Lin al. 2024 Frantar et al. 2022 and ensures balance between memory efficiency and precision.
B CONTROLLING RANGE SHRINKAGE
7 enumerate candidate pairs (S, 2) scasing factors and zero-points- determine the optimal loss- oss-error-awarc affine quantization grid. This process involves iteratively refining S and Z by reducing the maximum value max w) and increasing the minimum value min(w). However, excessive shrinking the range may result in poor representation of extreme values, leading to model quality degradation.
In Equation
16

---

Published as a conference paper at ICLR 2025
Algorithm 2 LeanQuant-Exact for Millon Millon-parametter Networks
Input: row R in the weight matrix, sample input matrix X, bit width b, hyperparameter p Output: Quantized row
1: w <- 0,
2: H (2XXT)-1
3: if uusing non-uniform grid then
4: G +- arg min (diag(H-1))-)) qquant,.. (w, 9) w/²
D E.
0-|0]=2>
5: else if using affine grid then
6: S,Z +- arg min (diag(H")) ()- /f S. - -w/² (8,2)ES
> E.
7: end if
8: for j <- ,,....c.d do
9:
if using non uniform grid then
+- arg min, (quant (w; -9-
10:
2H,
11: w; quant (w;, 9)
else if using affine grid then
+- min, (guant (ree,, S.2) w2)
13:
14:
w: + quant S
15:
end if
H
16:
W W
Wi w:)
H
H
end for 19: return W
To control the extent of range reduction, we introduce the parameter which determines the degree of shrinkage. Lower bit widths require more aggressive shrinking due to the limited number of grid points. We set for b-bit quantization as follows:
0.27 if = 4,
0.37
if =3=3,
(8)
0.47
if 2
C LEANQUANT-EXACT
The pseudocode of LeanQuant-Eii Exact for accurately quantizing million-parameen networks is pre- sented in Algorithm
C.I BERT EXPERIMENTS WITH LEANQUANT-EXACE
Method
Bits
BERT-3
BERT
FP32
32
84.66
88.53
OBQ
4.03
84.40
87.96
LeanQuant... -Exact
4.13
84.58
88.49
OBQ
3.03
83.47
84.72
L -Exact
3.06
84,20
86.21
Table 6: F1 scores SQuAD of BERT models quantized using OBQ and -eanQuunt,,--xace LeanQuant,,,Exxal outperforms OBQ in maintaining model quality.
We compare the performance of BERT models (Devlin et al.) 2018), quantized with OBQ Frantar & Alistarh 2022) and LeanQuant,,, Exact, on the SQuAD dataset (Rajpurkar et al.] 2016). We quantize the T2-layer BERT-base (Devlin et al. 2018) and the 3-layer BERT-3 variant from Kurtic
17

---

Published as a conference paper at ICLR 2025
Table 7: Perplexity evaluations of Llama models under different quantization methods and bit widths. The results of GPTQ, AWQ, OmniQuant are from Shao et al. (2024), and the results of SqueezeLLM are from Kim et al. (2023). The official Squeeze code does not support 2-bit quantization, and we report the available results from Kim et al. (023).
WikiText-2
C4
Grid
Method
1-7B
1-13B
2-7B
2-13B
2-708
1-7B
2-7B
2-13B
2-70B
Avg.
FP16
16
5.58
5.09
4.88
3.31
6.97
5.52
5.697
GPTQ
6.13
5.13
7.43
7.37
5.67
6.008
Affine
AWQ
4,00
6.08
5.34
6.15
5.12
7.52
6.86
7.68
6.74
OmnilQuant
4,00
5,86
5.21
5.74
5,02
3.47
7.34
6.76
7.35
6.65
5.65
5.905
LeanQuant.g
4,00
5.92
5.25
5.73
5,08
3.49
7.30
6.76
7.25
6.63
5.63
5,904
Nom-umin Nom-umiform
SqueezeLLM
4.04-4.05
4,99
5.58
5.818
LeanQuant
4,04-4,05
5.81
5.19
5.64
4,99
3,42
7.21
6.70
7.13
6.57
5.58
5.824
GPTQ
3,00
8.06
6.76
8.37
6.44
4.82
9.49
8.16
9.81
8.02
6.57
7.650
AWQ
3,00
11.88
7.45
24.00
10.45
13.26
9.13
23.85
13.07
Affine
OmniQuaant
LeanQuunk.gr
3.00
6.62
5.76
6.61
5.66
3.91
7.98
7.19
8.27
7.23
5.90
6.513
Non-umiform
SqueczeLLM
3.02
6.32
5.60
6.18
5.36
3.77
7.75
7.08
7.72
6.97
5.83
6.258
LeanQuuniey
3.02
6.34
5.60
6.19
5,40
3.80
7.74
7.05
7.73
6.98
5.83
6.266
GPTQ
2,00
1.155
6.8E4
3.8E4
5.6E4
2,0E4
689. 13
2.5E3
NaN
323. $23.12
48.82
NaN
Affine
OmniQuant
17.21
26.76
12.28
26.395
LeanQuuns.g
2,00
18.53
14.42
25.69
24.43
7.92
19.99
16.53
27.11
20.92
10.84
18.638
Non-uniform
SqueezelIIMM
2011
-NAA
61.25
LeanQuanter
2.01
15.65
9.64
15.51
10.06
6.35
17.62
10.93
17.07
11.83
7.96
12.262
al. 2022 to 3 and bits. OBQ and LeanQuant- are calibrated using 1024 samples from the training set, and the F1 score is reported on the test set.
D ROBUSTNESS OF LEANQUANT GRIDS DURING QUANTIZATION
LeanQuant prevents drastic increase to the task loss by learning the quantization grid for better preservation of the precision of outlier inverse diagonals. However, since the not-yet-quantized weights will shift during the iterative quantization process and the quantization grid is fixed be- forehand, one potential problem arises: the quantization grid may no longer be well-aligned with the outliers after certain iterations. Fortunately, this is not a problem in practice. The loss-error- awareness property of LeanQuant grids prevents high-norm weight perturbations 6, (Equation 3 from ocurring, hence the weights do not shift by much during the iterations. Furthermore, new inverse-diagonal outliers will arise during the iterative quantization process. In OBQ, the inverse Hessian updated after each iteration follows,
H
where H is the inverse Hessian with its i-th row and column removed. The remaining inverse diagonals only decrease in magnitude towards zero after each column and row removal.
E EXPERIMENT DETAILS
Baseline Reproduction We the quantized models provided by the official repository where possible. We obtained quantized L. L.aMM-7B, I.LMMA-33,, Llama-2-7B, Llama-2-13B from the OmniQuant repository, and LLaMA-7B, LLaMA-13B, Llama-2-7B, Llama-2-13B, Mistral-7B from the SqueezeLLM repository. We obtained the community- -driven GPTQ-quantized version of The other quantized models are reproduced using
Llama-3.1-405B-Ime -0SS---tttttctt from HuggingFace ||] the official codebases and recommended hyperparameters. For OmniQuant, we set the training
4558 Instrncttttttt wwtwco//ohhoooooooooo wttp.ccc.ooguuuuuuunooo..o.o quagiiiggguunn///ueea /wettttttnttttaaeeaee ammamm-3
18

---

Published as a conference paper at ICLR 2025
Liama Irsstruct an MT: Bercch
Win Rate:t44 4399
LeanQuartar Quant
bi 125)
4.3
61
36
Win RadeRBleddcoom 4399
LeanQuantar
bi size 125)
56
90
34
LuanQ SquasreLLM
Win Ratect3 2399
33
96
29
(4b8)
Feemer Win
Ti+
Furner Lose
Evaluation of quantized Llama-3-88 3-BB- IB-Instruct on MT-Bench using OpenAl GPT-4o judge. The win rates reported exclude ties.
Figure 4:
Table 8: shro accuracy comparison between LeanQuante and rotation-based quantization method QuaRot.
ARC
LAMBADA
MMLU
Method
Bits
Avg.
Easy
Chg
Sid
OpenAl
STEM
Humon.
Social
Other
QuaRot-RTN
4,00
65 65.92
46.93
65.75
71.92
49.67
51.24
65.75
64.24
60.18
QuaRat-GPTQ
51.60
64.04
LeanQuant.g
4.00
76.30
50.51
68.27
75.90
51.51
53.43
70.69
68.43
64.38
QuaRot-RIN
4.00
74.54
47.44
66.19
74.17
39.74
41.74
53.04
51.95
56.10
Llama 13B
QuaRot GPTQ
4.00
76.35
49.23
69,82
76.48
41.29
47.01
58.99
$7.84
59.63
LeanQuant
4.00
76.43
49.06
69.90
76.48
42.53
47.86
60.03
58.93
60.15
epochs 20, enable Learnable Weight Clipping (LWC), LWC learning rate of 1e-2. For SqueezeLLM, there is no tunable parameters. For GPTQ, we turn activation ordering (quantizing columns in order decreasing activation size) for more accurate model.
Perplexity Evaluations et al. 2022) sequences from the test set of the Wiki'Tex22 and C4 datasets (Merity et al., 2016]] Raffel et al. 2020) are concatenated into 128 sequences of length 2048 tokens for perplexity testing.
follow the perplexity evaluation procedure described by (Frantar
Accuracy Evaluations We use Im-evaluation-hhrres. (Gao et al. 2023) for evaluating zero-shot accuracy tasks. The task names evaluate lambada, air_are, winogrande, piqa, hellaswag, mmlu.
F PERPLEXITY EVALUATIONS
The perplexity evaluation results on WikiText2 (Merity al. 2016) and C4 (Raffel al. 2020) for quantized models are presented in Table
G LLM-AS-A-JUDGE
LLM as a Judge The evaluation results on MT-Bench using GPT-4o (2024-05-13) as judge are presented in Figure 4 We pitch 3-bit and 4-bit, with group size of 128, LeanQuant,r against Om- niQuant, and 4-bit LeanQuant,,, against SqueezeLLM. LeanQuant achieves higher win rate than the baselines.
H MORE ACCURACY RESULTS
The zero-shot accuracy results on benchmarks for quantized LLaMA-7B, LLaMA-13B. Llama-2- 7B (Touvron et al. 2023a b) are presented in Table 1O We also compare affine LeanQuant with the rotation-based quantization algorithm QuaRot (Ashkboos et al. 2025), with results presented in Table Furthermore, we compare affine, group-wise quantization using LeanQuant,gr OmniQuant, and AWQ in Table
19

---

Published as a conference paper at ICLR 2025
Table 9: Zero-shot accuracy of affine, group-wise quantized models using LeanQuant Omni- Quant, and AWQ.
ARC
LAMBADA
MMLU
Method
Group Size
Bits
Avg.
Easy
STEM
Human.
Social
Llama-2-7B
OmniQuant
75.21
37.79
46.77
AWQ
128
4.25
75.17
43,26
67.40
72.70
34.89
37.79
46,34
45,93
52.94
LeanQuanleyr
128
4.25
76.26
43.00
67.75
74.44
35.65
39.02
47.94
49.79
54.23
Llama-2-138
OmniQuant
128
4.25
78.24
47.61
69.75
75.99
42.06
47.74
59.83
58.61
59.98
AWQ
128
4.25
78.91
46.50
70.17
76.19
41.39
46.70
59.38
$5,94
59.40
LeanQuant.g
128
4.25
79.00
47.18
70.75
77.95
41.96
47.86
59.54
58.97
60.40
1aamm-3.1-70B-tnsttree
AWQ
128
4.25
86.62
62.12
72.09
75.70
74.88
80.60
87.07
83.71
77.85
LeunQuant.gr
128
4.25
86.49
61.69
72.70
76.07
75.61
80.79
87.55
83.71
78.08
I QUANTIZATION COST AND OVERHEAD
The time cost of LeanQuant for different models and configurations are presented in Table T3 A comparison of GPU memory consumption for different quantization algorithms on different-sized LLMs is presented in Table
J INFERENCE EFFICIENCY OF QUANTIZED MODELS
Table 12 presents the inference efficiency of 4-bit quantized Llama-3-8B during the decoding and
prefill phases. For non-uniform LeanQuant models, we have developed dedicated CUDA kernel for efficient inference, and we compare its efficiency against the Squeezel.L.M kernel in Table T2
For affine LeanQuant models, we leverage the ex xxllamav2 kernels (Turboderp-org 2024
The inference efficiency in Table 12 evaluated on NVIDIA A A100-40GB GPU, For decoding, report tokens per second per batch while generating 4096 tokens. For the prefill phase, we measure time to first token using a 4096-token prompt.
K ABLATION STUDY
Sensitivity to Hyperparameter p Ablative experiments on the effects of the hyperparameter on the quality of LeanQuant models are presented in Table[14] In the case of p 0, the inverse Hessian diagonals are ignored as the weights for clustering, and the centroids are learned based on the density of weights. worth noting that p results sub-optimal model quality compared to higher values of which means that the loss-error cess-error-wwaad awareness property of the quantization grid is critical for maintaining model quality.
Grid Point Initialization Ablative experiments comparing k k-means++ initialization with our pro- posed uniformly spaced grid initialization are presented in Table [3]
L Loss ERROR COMPARISON
A comparison of the sum of loss errors between GPTQ and LeanQuant (affine and non-uniform) presented in Figure 5
M MORE RELATED WORKS
Quantization for Large Language Models Quantization reduces the precision LLM parame- ters achieve model compression and enable memory-efficient inference. Calibration-free
20

---

Published as a conference paper at ICLR 2025
Table 10: Zero-shot accuracy of more quantized L.MSs benchmarks.
ARC
LAMBADA
MMLU
Method
Bits
HellaS
PIQA
WinoGi
Avg.
Easy
Chg
Std
OpenAl
STEM
Hunlian
Social
Other
LLaMA-7B
FP16
16
75.29
41,81
67.77
73.49
28,20
32.03
31.65
36.53
56.92
78.67
70.09
53.86
GPTQ
4.00
73.61
39.51
65.61
71.98
24.29
28.12
25,80
31.16
55.61
77.80
70.40
49.03
OmniQuant
4.00
74.49
39.68
65.38
71.96
30.32
30.92
33.38
35.79
55.82
78.45
68.67
53.17
LeanQuanteg
4,00
74.03
41.64
63.75
70.56
31.27
33.13
32.66
37.79
56.31
78.40
69,30
53.53
Affine
GPTQ
3.00
66.67
35.84
49,89
58.37
25.94
27.46
23,46
24.98
51.76
75.35
64.72
43.78
OmniQuunt
3.00
72.22
38.48
59.67
69.20
27.18
27.65
25. 64.
29.61
52. 22.99
76.55
67.17
49.67
LeanQuantar
3.00
73.70
40.78
65.28
72.66
27.85
31.37
30.61
35.31
56.23
78.45
70.09
52.94
GPTQ
2.00
26.35
22.01
0.00
0.00
23.69
25 08
24.28
24.01
25.69
53.70
50.99
24.95
QmniQuant
2,00
$1.05
22.70
11,80
23. 13
26.51
26.04
24.37
23.69
34.71
64.36
54,30
32.97
LeanQuant.r
2.00
55.98
27.22
47.45
24.45
22.85
25.46
37.19
68.12
60.77
39.40
L.
74.92
40.61
71.99
29.12
29.
56.55
78,29
69.38
Non-umffom
LeunQuant
4.05
75.17
41.55
69.01
74.33
27.37
N0.50
29.41
35.44
56.09
77.91
70.17
53.36
Squeezel LM
3.02
71.46
37,88
61.71
71.05
23.66
26. 99
26. 16
30.54
54.89
77.86
68,59
50.07
Non-
LeanQuant
3.02
71.76
36.35
63.77
71.71
26.64
29.05
29.80
32.70
53.80
77.09
69.53
51.11
Squeezel I
LeanQuant
2.01
50.38
24.40
31,67
41.30
21.79
24.19
21.74
24.36
37.49
65.67
58.33
36.48
LLaMA-13B
FP16
16
77.40
46.42
71.12
76.19
36.41
41.55
48,49
48.54
59.92
79.16
72.69
59.81
GPTQ
4.00
77.06
45.56
69.12
75.28
34.44
39.15
45,95
46.73
588.99
78.56
72.53
56.63
Omniquunt
4.00
75.97
45.22
68.25
75.59
35.30
40.21
48.20
47.25
59.11
78.94
72.61
58.79
LeanQuant
4.00
76.39
46.42
76.27
35.52
39.45
46.18
47.22
58.82
78.94
72.30
58,91
Affine
GPTQ
3.00
70.92
39.93
$7.29
64.82
29.37
31.94
33.18
35.34
54.05
76.99
68.43
49.13
OmniQuant
3.00
75.42
42.83
60,80
71.34
29.56
34.24
36.24
41.17
57.27
77.97
69.61
54.22
LeanQ
3.00
75.84
43.34
67.49
74.85
33.37
36.56
41.92
44.74
56.75
77.97
70.48
56.66
GPTQ
2.00
27.10
21.93
0.00
25.76
49.72
OmniQuant
2.00
$9.51
29.52
17.85
23.35
22.52
24.12
22.81
24.46
42.01
67.25
$6.12
35.41
LeanQuant
2.00
61.45
29.27
44.63
50.82
28.73
26.01
27.20
27.26
39.08
71.27
68.82
42.87
SquezzeLLM
4.04
76.56
46.25
69.53
75.22
32.98
19
42.18
44.00
59.29
78.62
71.82
70.
Non-uniforn
LeanQuantes
4.04
76.39
45.05
71.55
76.48
34.76
38.77
46.12
47.18
59,29
78.78
73,24
58.87
SquezzeLLM
3.02
75,46
43.77
65.07
72.75
30.45
34.24
37.18
40.46
57.32
78.29
71.35
55.12
LeanQuamt,,,
3.02
75.17
43.00
70.41
77.29
33.81
38.32
43.16
45.06
57.35
78.29
71.35
57.56
Sq
2.01
-N/A-
LeanQuanta.
2.01
65.66
32.42
54,49
66.93
23.44
25.50
23.98
28.42
45,66
72,80
66.14
45.95
Llama-2-133
2-13B
FP16
16
79.50
48.46
70.35
76.73
42.28
47.89
61.16
59.38
60.05
79.05
72.22
63.37
GPTQ
4.00
78.32
45.48
68.33
75.35
40.28
46.08
56.48
54.65
58.92
78.45
71.82
59.59
OmniQuunt
4.00
77.69
47.10
68.74
75.57
41.39
46.10
$7.39
55.87
59.48
79.00
70.32
61.70
LeanQuant. or
4.00
79.42
47.27
75.90
42.21
59.90
57.93
59.07
71.82
62.57
Afhne
GPTQ
3.00
72.85
39.85
67.20
34.86
38.85
47.97
46.48
54.61
76.28
70.32
53.62
OmniQuant
3.00
76.60
43.34
60.70
70.54
38.60
42.59
53.23
51.82
$7.42
77.97
69.14
58.36 38.36
LeanQuant_ of
3.00
77.31
44.54
75.88
37.93
43,80
53.07
$2.62
56.36
76.99
70.72
59.76
GPTQ
2.00
20.12
0.00
22.84
23.53
23.98
25.97
47.75
24.
OmniQuant
2.00
48.19
24.66
10.21
20. 14
21.34
24.21
21.77
23.85
40.16
63.00
52.33
31.81
LeanQuant.g
2.00
50.88
24.32
32,70
39.57
21.50
24.38
21.90
24.40
38.01
67.19
56.91
36.52
Sq
4.04
78.91
47.70
70.00
76.23
42.72
47.89
60.19
58.32
59.74
78.73
72.77
63.02
Non-un inn-uniform
LeanQuant.,
4.04
78.91
47.56
71.12
77.43
43.51
47.44
59.54
58.83
59.58
78.62
72.05
63.15
Squeezel I.M
3.02
77.27
43.17
66.37
73.80
38.22
44.63
55,18
53.11
58.74
77.86
69.42
59 80
LeanQuant.,
3.02
77.19
44,20
71.14
78.59
40.72
45.46
56.87
55.10
56.38
77.75
70.09
61.23
Squeezel L.M
2.01
N/A
LeanQuant,,
62.46
30.20
47,00
61.09
25,28
27.74
27.56
28.87
42.20
62.04
44.03
quantization approaches, such as LLM int8 DDettmers et al. 2022a). NormalFloat (Dettmers et al. 2024]. and Student Float (Dotzel et al. calibration data. In contrast, methods like GPTQ (Frantar et al. 2022). AWQ (Lin et al. 2024). Om-
2024). perform zero zero-shot quantization without requiring
niQuant (Shao et al. 2024), SpQR (Dettmers et al.), SqueezeLLM (Kim et al. 2023). QUIP (Chee) et al. 2024). AQLM (Egiazarian et al. 2024). and QUIP# (Tseng et 2024) leverage calibration to improve quantization quality by adapting to input data distributions. Some methods (Xiao et al.)
2023 extend quantization to both model weights and intermediate activations. Some approaches combine quantization-awae aware training to push the limits of quantization; for example, LLM-QAT (Liu et al. 2023) fine-tunes quantized models to recover model quality, BitNet (Ma et al. 2024) explores ternary-valued LLMs, while OneBit (Xu et al. 2025) demonstrates the feasibility of 1-bit quantization for L.MSs.
21

---

Published as a conference paper at ICLR 2025
Table 11: GPU memory consumption of quantization algorithms different-sizedd LLMs. "OOM" means out of memory.
Llama-3-8B Llama-3-70B Mistral-Large (123B) Llama-3.1-405B
OmniQuant
25.3 GB
OOM
OOM
OOM
SqueezeLLM
0OM
OOM
0OM
0OM
GPTQ
7.9 GB
17.1 GB
32.8 GB
0OM
LeanQuant
7.9 GB
17.2 GB
33.0 GB
65.4 GB
Table 12: Inference efficiency of 4-bit quantized Llama-3-8B in the decoding and prefill phases.
Decoding efficiency is measured in tokens per second per batch for generating 4096 tokens, while prefill efficiency is evaluated by time to first token for 4096-token prompt. All results obtained on an NVIDIA A100-40GB GPU.
Decoding (tokens/s batch)
Prefill (time first token)
Batch Size Batch Size Batch Size
Batch Size
Batch Size
Batch Size
Sq
26.09
19.68
7.22
153.26s
333.36s
0OM
LeanQuant 23.73
18.28
9.02
0.36s
1.34s
5.24s
LeanQuantar 38.30
28.78
10.20
0.33s
1.23s
4.98s
Efficient LLMs Beyond quantization, various techniques have been proposed enhance LLM effi ciency. KV cache compression methods such as KIVI (Liu et al.) and CQ (Zhang et al. 2025c) reduce memory overhead by compressing key-value cache during LLM decoding. Pruning ap- proaches, such as SparseGPT (Frantar & Alistarh (023). remove model parameters in a structured or un-structured manner to create sparse, efficient models. Model sketching (Zhang et al. 2025b)
enables efficient fine-tuning by compressing LLMs and make them directly fine-tunable. Hardware- aware optimizations, including FlashAttention (Dao al.] (2022) and NoMAD-Attention (Zhang et al. 2025d), improve memory and compute efficiency for modern accelerators. Optimizer-state compression techniques (Zhao et al., 2024. ZZhang al. 2025a) reduce memory usage during pre- training and fine-tuning.
Uniform and Non-uniform Quantization Quantization techniques can be broadly categorized into uniform (affine) and non-uniform methods. Uniform quantization (Krishnamoorthi 2018 Frantar ct al. 2022) divides the range of values into equal-sized intervals, which is hardware-efficient but often fails to accommodate the non-uniform distribution of the weights of deep neural networks. Non-uniform quantization improves model compression by allocating precision dynamically based data distribution. Additive Powers-of-Two Quantization (Li et al.) introduces efficient uniform discretization scheme that leverages power-of-twe representations.
comuniform-to-uniffe
quantization (Liu al. (022) bridges the gap between non-uniform and uniform quantization using a generalized straight- through estimator for training. NUPES (Yvinec et al. (023)) formulates non- uniform post-training quantization as a power exponent search problem. Mr. Mr.BiQ (Jeon et al. 2022)
focuses on reducing reconstruction error through post-training non-uniform quantization, improving model performance. Methods such as non-uniform step size quantization (Oh et al., 2022) refine
quantization granularity to enhance accuracy. while learning-based approaches (Gongyo al. 2024) adaptively determine step sizes optimize neural network quantization.
22

---

Published as a conference paper at ICLR 2025
Table 13: Total time taken by LeanQuant for quantizing different-sized LLMs, using a single L40s-
48GB GPU, an AMD 7RYC 48-Core CPU, and 370GB of RAM. Llama-3. 1-405B is quantized using 2 Quadro RTX 8000 GPUs, an AMD EPYC 7742 64-Core CPU, and 48TBB of RAM.
Model
Grid
Group Size
Bits
Time
Affine
4.00
14 mins
Llama-2- 2-7B
Affine
128
4.25
15 mins
Non-uniform
4.05
35 mins
Affine
4.00
16 mins
Llama-3-8B
Affine
128
4.25
20 mins
Non-uniform
4.05
mins
Affine
4.00
178 mins
1lama-2-70B
Non-uniform
4.04
mins
Mistral- Laniral-Larg--Isttuee--4444 (123B)
Affine
4.00
252 mins
Llama-3. 1ama--.11405B
Affine
128
4.25
1241 mins
Table 14: The perplexity of LeanQuant models on WikiText2 and C4, using different values of
WikiTex12 3-bit
C4
Grid
Hyperparameter
4-bit
2-bit
4-bit
3-bit
2-bit
p
12.13
29.92
5,991.18
16.73
22.71
5,998 .6
Non-uniform
p
5.39
5.98
25.09
7.89
8.50
20.27
p
5.37
5.92
22.32
7.88
8.48
19.81
=
5.38
5.96
25.61
7.88
8.47
21.65
Mistral-7B
= 0
14.52
80.54
230.66
16.94
69.04
243.65
Affine
p =
5.52
8.58
55.50
8.03
16.84
41.99
= 3
5,51
6,36
18.33
8,03
8.80
20.20
p =
6.31
8.02
8.86
20.47
p =
5.69
6.76
NaN
7.15
8.23
62.00
Non-uniform
p
5.65
6.30
17.16
7.13
7.83
19.14
p = 3
5.64
6.25
17.84
7.13
7.80
19.55
p
5.64
6.28
15.82
7.14
7.83
18.89
Llama-2-7B
p
5.84
8.19
93.01
7.30
9.54
85.62
2
5.77
7.33
27.82
7.27
8.83
28.86
Affine
p =
6.80
25.97
7.26
8.32
27.57
p 4
5.75
6.69
26.82
7.25
8.29
28.14
Table 15: Ablative experiments grid point initialization.
Llama-2-7B
2-7B
Llama-3-8B
Mistral-7B
Grid Init.
4-bit
3-bit
2-bit
4-bit
3-bit
2-bit
4-bit
3-bit
2-bit
K-means++
5.64
6.25
17.84
6.59
8.31
46.31
5.37
5.92
22.32
WikiText2
Uniformly Spaced (ours)
5.66
6.20
17.53
7.88
41.78
5.40
5.88
19.06
K-meanst+
7.13
7.80
19.55
10.17
12.53
39.86
7.88
8.48
19.81
C4
Uniformly Spaced (ours)
7.14
7.72
18.75
10.20
12.16
36.00
7.91
8.42
17.85
23

---

Published as a conference paper at ICLR 2025
GPTQ
LoanQuainy
LeanQuant,
Liama 2-78
Liama 2:
Liama 2-78 (2.6))
100000
330000
15000
2000000
200000
2000000
Liama-2-138
Laam-22-38
Liama-2-138
100000
2000000
Sum of Loss Ervere
50000
300000
50000000
LLaMA-78
200000
330000
5000000
(2-51)
1000000
4300000
700000
3000000
230000
Amseiion Aery
Inlecdon vague
Atteniann Query
Anautto Out
MaB--aal Sola
Marttonkey
Antentonn Query
Antedpen Cut
Astatton Oar
18 Sala
Cowns
Down
Figure 5: Comparison of loss errors of each layer for GPTQ and LeanQuant (affine and non- uniform) during iterative quantization.
24


Reviews:
[{"review_id": "mt45uY3fy6", "reviewer": "Reviewer_zNzV", "summary": "This paper presents a method for optimizing the quantization grid in a layerwise loss-aware manner to reduce low-bit weight quantization error. Unlike previous optimization-based approaches like GPTQ, they derive expressions for the optimal settings for the quantization grid rather than the optimal weight update to apply at each step. The approach is generalizable to both non-uniform and uniform quantization grids. This method is able to preferentially preserve more sensitive values, while maintaining computational efficiency by leveraging the layerwise optimization objective. This grid-tuning approach can also be integrated with previous weight update approaches like GPTQ to attain higher accuracy. They demonstrate the accuracy benefits of their approach, and also highlight the computational and memory efficiency of applying their quantization method relative to prior works.", "questions": "- Can you provide any intuition for why the uniform grid point spacing method helps improve accuracy for k-means than k-means++ style grid points (i.e. a visualization of where the initial grid points end up being placed in both cases, versus the “target” grid points)?\n- For the efficiency results in Table 9, is the K-means non-uniform quantization grid search parallelizable on the GPU, or is it being run on the CPU?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- They present an algorithm to tune the quantization grid using the layerwise loss-aware objective\n- They present both non-uniform and uniform methods (the non-uniform method is flexible and leverages clustering, whereas the non-uniform method performs a constrained search over potential scale factors)\n- They provide accelerated GPU kernels to solve the objective for their affine quantization approach\n- They provide detailed analysis of their method against prior work in both uniform and non-uniform quantization regimes\n- They provide multiple ablations demonstrating how their method reduces the loss error, as well as the impacts of K-means initialization", "weaknesses": "- Although the method of determining weight sensitivity using the layerwise loss is distinct, the approach of performing K-Means clustering in equation (6) to derive non-uniform datatypes is the same as prior work (eg. the sensitivity-weighted clustering approach to derive non-uniform datatypes in SqueezeLLM)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhw5oo9T11", "reviewer": "Reviewer_v8KG", "summary": "The paper proposes LeanQuant, an accurate, versatile, and scalable quantization approach. LeanQuant is built upon the iterative loss-error-based quantization framework and identify one of the biggest limitations of such methods: the quantization grid introduces high quantization errors due to the existence of outliers in the inverse Hessian diagonals.", "questions": "1. Whether it can be compared or ablated with other importance metrics such as the \"Hessian diagonal\" in SqueezeLLM, these works are similar in idea. Please see the weakness part.\n\n2. There is a lack of literature survey on uniform and non-uniform quantization in related works.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is simple and effective, compatible with iterative loss-error-based quantization approaches such as GPTQ. The calibration process is relatively lightweight, allowing for the quantization of models up to 405B. The experimental design of this paper considers the latest model series and different model sizes, which is comprehensive.", "weaknesses": "LeanQuant is built on the core idea of using \"inverse Hessian diagonals\" metric to quantify the importance of weights. Similar ideas are also introduced in works such as SqueezeLLM [1], which exploits the \"Hessian diagonal\" metric directly derived from the quantization loss function to identify important weights. Compared to the metric in SqueezeLLM, I think the derivation of the metric in LeanQuant is relatively heuristic. Although LeanQuant beats SqueezeLLM in the final performance, it would be better to directly compare the different importance metrics and show the strengths of the proposed \"inverse Hessian diagonals\" metric since SqueezeLLM does not leverage GPTQ to minimize the quantization error.\n\n[1] Kim, Sehoon, et al. \"Squeezellm: Dense-and-sparse quantization.\" arXiv preprint arXiv:2306.07629 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nepn06gtYA", "reviewer": "Reviewer_VgEF", "summary": "This paper proposes Loss-error-aware Network Quantization (LeanQuant), a method designed to learn loss-error-aware quantization grids to reduce quantization error. LeanQuant determines optimal quantization parameters, such as scaling factor and zero-point, by iteratively narrowing the weight range in discrete steps. To accelerate grid learning, custom CUDA kernels were developed for efficient grid searching. Consequently, the paper demonstrates that LeanQuant can achieve superior linguistic capabilities with quantized LLMs compared to previous methods.", "questions": "1.\tCould you provide a step-by-step explanation of how LeanQuant is adapted for non-uniform quantization by highlighting key differences from the affine quantization process?\n2.\tTo ensure consistency between the promised and delivered comparisons, I suggest the authors either include a summary of the AWQ results in the main text or explicitly state why they were omitted.\n3.\tWhy did you choose row-wise quantization for the comparison in Table 1? Additionally, do you have results for group-wise quantization? Providing more analysis with various quantization methods could address potential concerns about the fairness of the comparison and offer insights into LeanQuant's performance across different quantization schemes.\n4.\tI recommend including runtime comparisons with GPTQ, AWQ, and OmniQuant in Table 5 to provide a more comprehensive view of LeanQuant's efficiency relative to other existing methods\n5.\tI recommend including peak GPU memory usage for AWQ and GPTQ in Table 4, or explaining why this information was omitted. This would offer a more complete comparison of resource requirements across methods.\n6.\tHow are the quantization grids calibrated for non-uniform quantization? Since each grid is independent for non-uniform quantization, I assume the calibration process must differ from the loss-error-aware affine grid described in Section 3.2.2.\n7.\tWhat would happen if group-wise quantization were applied to the affine quantization method presented in Table 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tLeanQuant calibrates quantization grids, and it is a novel approach in the post-training quantization of LLMs.\n2.\tThe development of fused CUDA kernels efficiently supports the calibration process, so it makes the complex grid learning procedure more accessible for broad application due to its enhanced efficiency.", "weaknesses": "Overall, several details and evaluation results are missing, which makes it difficult to fully understand and be convinced of the effectiveness of LeanQuant.\n\n1.\tA detailed explanation of how LeanQuant can be extended to non-uniform quantization is missing.\n2.\tAlthough Section 4 (Experiments) states that the proposed method will be compared with AWQ, GPTQ, and OmniQuant, the evaluation results for AWQ are not presented in the main text (some results for AWQ are included in the appendix only).\n3.\tWhile group-wise quantization is widely used in LLM quantization, Table 1 appears to evaluate affine quantization only with row-wise quantization. (Note that AWQ is specialized for group-wise quantization, and layer-wise quantization with AWQ sometimes results in slightly worse linguistic capabilities compared to GPTQ.)\n4.\tThe runtime for LeanQuant is presented with and without the custom kernel in Table 5, but no comparison is made with previous methods.\n5.\tPeak GPU memory usage for AWQ and GPTQ is not included in Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "prrTZw9GOL", "reviewer": "Reviewer_uLwi", "summary": "The paper introduces a new quantization method, LeanQuant, designed to improve the efficiency and accuracy of large language models (LLMs) without requiring specialized hardware adaptations.\n\nKey Contributions:\n1. LeanQuant innovates on post-training quantization (PTQ) by introducing a loss-error-aware quantization grid. This grid preserves model quality better than traditional min-max grids for outliers in the inverse Hessian diagonals, which otherwise lead to significant accuracy loss.\n2. The method generalizes to affine and non-uniform quantization types, making it adaptable to existing frameworks and more compatible with standard inference kernels.\n3. LeanQuant demonstrates scalability with successful quantization of models up to 405 billion parameters, achieving high accuracy using limited computational resources.\n4. By designing a fused GPU kernel, LeanQuant achieves substantial speedups in grid learning, making quantization efficient and suitable for a range of model sizes.", "questions": "The author’s motivation is well-founded, and the experiments are thorough. However, addressing the issues I raised in the weaknesses would greatly strengthen the paper.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of this paper lies in its innovative approach to addressing both accuracy and scalability in quantizing large language models, all while maintaining low hardware requirements. LeanQuant introduces a loss-error-aware quantization grid,  tackling the common issue in traditional quantization methods where accuracy is lost due to outliers in inverse Hessian diagonal elements. This approach adapts the quantization grid to preserve the precision of key weights, improving model quality, especially in low-bit quantization.\n\nAdditionally, LeanQuant’s multi-format compatibility enables it to integrate seamlessly with popular deep learning frameworks, eliminating the need for specialized inference kernels and greatly enhancing deployment flexibility.\n\nOn the scalability side, LeanQuant incorporates a fused GPU kernel design, accelerating the quantization process and allowing it to efficiently handle extremely large models (up to 405 billion parameters) with limited GPU resources. This efficiency showcases LeanQuant’s practicality under constrained computing resources, highlighting its potential for low-cost, high-efficiency AI applications. As a result, the paper presents a robust solution for large-scale model quantization, excelling in quantization quality, resource efficiency, and deployment adaptability.", "weaknesses": "1.  Quantization isn't solely about reducing memory usage or data transfer—it’s fundamentally about reducing the computational workload, which in turn decreases the size of intermediate data and the need for data movement. If the focus is only on reducing model size without addressing the computational load, it could result in diminished acceleration benefits. This is because, even if memory requirements decrease, hardware may still need to perform the same level of computation, which undermines the intended efficiency.\n2. As you pointed out, while LeanQuant can quantize models down to 3-bit or 2-bit, modern GPUs often support 4-bit or 8-bit multiplications. Thus, quantizing to lower bit widths (like 3 or 2 bits) might not yield computational efficiency if, during execution, the kernel still calls for a 4-bit multiplier. This forced conversion undermines the benefits of lower-bit quantization since it fails to leverage the intended reduction in computational and resource demands.\n3. Without considering the storage format of hardware, 3-bit or 2-bit weights may still be stored on the GPU in higher bit-widths (e.g., 4-bit, 8-bit, or even 16-bit), resulting in excess storage and data transfer requirements. This dilutes the benefits of quantization. Designing quantization methods with hardware-oriented data formats is essential to ensure that low-bit quantized data can be efficiently processed and stored by hardware.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NyeXUEblue", "reviewer": "Reviewer_evsw", "summary": "The paper proposes LeanQuant, a novel post-training quantization (PTQ) method that addresses the limitations of existing methods in reducing memory requirements and inference costs of large language models (LLMs). LeanQuant identifies a critical issue in prior PTQ methods, where the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals. To overcome this, LeanQuant learns loss-error-aware grids instead of using non-adaptive min-max affine grids, enabling accurate and versatile quantization. The method uses an iterative loss-error-based quantization framework, which involves computing the loss-error for each layer, learning a loss-error-aware grid for each layer, and quantizing the model using the learned grids. LeanQuant is evaluated on several large language models, including Llama-3.1 405B, one of the largest open-source LLMs, and achieves high accuracy, comparing favorably to recent competitive baselines. The method is also shown to be scalable, achieving accurate quantization of Llama-3.1 405B using two Quadro RTX 8000-48GB GPUs in 21 hours. LeanQuant's ability to learn loss-error-aware grids enables it to generalize to a wide range of quantization types, including affine and non-uniform quantization, making it a promising solution for reducing the memory requirements and inference costs of LLMs. Overall, LeanQuant provides a novel and effective approach to PTQ, enabling the efficient deployment of large language models in real-world applications.", "questions": "The authors emphasize that range settings are important for quantized network accuracy, and GPTQ uses min-max quantization. In fact, GPTQ employs grid search (instead of SGD) to find the best range setting that minimizes the MSE error (||W-Wq||) https://github.com/IST-DASLab/gptq/blob/main/quant.py#L78-L95 Can authors compare with GPTQ with this MSE range settings? Can the author also explain the conceptual difference between loss-error-aware affine grid and GPTQ + MSE grid search?\n \nRegarding the non-uniform loss-error-aware grid, since the weight value changes during iterative GPTQ quantization, it might affect the k-means centroid as well. Did the authors consider this effect? Did the authors measure how far the weight - \\delta deviates from the original weight?\n \nThe authors provide two methods: non-uniform quantization and affine quantization. Can the authors briefly explain in which cases each quantization scheme is recommended?\n \nCan authors compare the results with the most recent LLM quantization works, such as QuIP[1], QuIP#[2], QuaRot[3] and SpinQuant[4]?\n\n[1] QuIP: 2-Bit Quantization of Large Language Models With Guarantees\n\n[2] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks\n\n[3] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs\n\n[4] SpinQuant: LLM Quantization with Learned Rotations", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Paper Strengths:\n\n**Compared to existing iterative loss-error-based quantization frameworks**: LeanQuant addresses the critical limitation of min-max affine quantization grids, which fail to preserve model quality due to outliers in inverse Hessian diagonals, by proposing learning loss-error-aware grids.\n\n**Compared to prior methods using non-adaptive min-max affine grids**: LeanQuant's approach produces quantized models that are more accurate and generalizes to a wider range of quantization types, including affine and non-uniform quantization.\n\n**Compared to other methods in terms of scalability**: LeanQuant is scalable, achieving very accurate quantization of Llama-3.1 405B, one of the largest open-source LLMs to date, using two Quadro RTX 8000-48GB GPUs in 21 hours, which is an improvement over other methods.", "weaknesses": "Paper Weaknesses:\n\nThis paper is limited in novelty. The authors proposed two methods: (1) non-uniform loss-error-aware grid and (2) loss-error-aware affine grid. The first method, the non-uniform loss-error-aware grid, is GPTQ combined with k-means, which is similar to GPTVQ [1]. The loss-error-aware affine grid is equivalent to GPTQ with grid range search, which is implemented in the original GPTQ codebase. \nAdditionally, they did not compare their work with the most recent state-of-the-art (SoTA) papers.\n\n[1] GPTVQ: The Blessing of Dimensionality for LLM Quantization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lMUGKHh9DP", "reviewer": "Reviewer_K8An", "summary": "Authors propose “Loss-error-aware Network Quantization” a post-training quantization method for reducing memory requirements in addition to latency of LLMs. The method is proposed as an accurate, versatile, and scalable alternative to traditional methods which are either specific hardware and software platforms or require high resources making them unscalable for the larger models. \nThe method is based on an iterative loss-error-based quantization framework, overcoming limitation in previous methods: the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals, instead, authors use loss-error-aware quantization grids, accounting for these outliers.", "questions": "Please address all the weakness", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper has written “BACKGROUND” sections well, giving an amazing context of quantization. \n2.\tResults validate the superiority of the proposed quantized method.\n3.\tThe claim of scalability and accuracy is satisfactorily answered, \n4.\tAlgorithm 1 seems to help make sense of the procedure visually. \n5.\tThe ablation study is written well, which helps make the paper a lot more sense, especially Q1, and Q2.", "weaknesses": "1)\t**The paper assumes the readers are well-versed in quantization**. Two elementary concepts of this work: *quantization grid* and affine / non-uniform grids, are not clearly defined inside the paper.   \na.\t**What exactly is grid?** [Line 110] “representing the full-precision floating-point parameters with a limited set of grid points on the quantization grid” What does this mean? An example or visualization would be really helpful here.   \nb.\t**[Line 116] “grid points are evenly spaced over the dynamic range of a group of weights”** What does this mean?  \nc.\tA simpler discussion/visualization on uniform and non-uniform grids is requested.   \nd.\tEquation 6, is first solved to establish a quantization grid. It's confusing to know what the solution to equation 6 means if the reader is unfamiliar with the quantization grid.   \ne.\t[Line 117] Group of weights? What’s that? Quantization needs some form of grouping of weights, why? Works don’t seem to establish it.    \n\n\n2)\t**Page 6 needs a simpler rewrite if possible**.   \na.\tThe works seem to be indicating some problems with the grouping of weights using KNN, but it's confusing to understand the problem. \nb.\t[Line 285] equation doesn’t have an equation number\nc.\tIt's not clear what part is already established and what parts are being proposed by the work. Equation 6: where did this equation come from or is derived in some previous work?, similarly Equation 7, and the equation on [Line 285], are authors proposing these equations. There doesn’t seem to be any explanation/derivation for these, just of out blue written as a solution. If they are already established solutions, citing works will help. \nd.\t“Efficient Fused GPU Kernel for Grid Learning” gives the 50x efficiency. Can this multi-threading technique be applied to existing work and make them more efficient as well? \n\n3)\tThe solution is based on the presence of outliers with abnormally high values in empirical distributions of inverse Hessian diagonals, (figure 1). This can be dataset-specific or model-specific. Show the presence of outliers in other datasets as well as models as well.\n\n4)\t**Contradictory results?** Table 10 shows p values as not so important for quantization. P > 0 is good enough for solving highly accurate quantization? Why does Table 10 doesn’t have p=1? Also, p >1 indicates the “extra” weightage of “outliers” in quantization, the first separation from existing work. \n[Line 259] : “The min-max affine grid employed by previous iterative loss-error-based quantization methods do not account for the existence of inverse-diagonal outliers, which may cause significant model quality degradation”\nDoes this mean, that just by the presence of outliers p=1, has more or less the same performance as P > 1, just accounting for outliers creates better grid formation? \nResults for other techniques (SqueezeLLM, GPTQ, OmniQuant) with p=1 or >1 (equation 6), outlier accounted grids can help validate the importance of this outlier driven gird.\n\n5)\tMinor issue: \na.\tTable 2 & 3: please mention of vanilla FP16 accuracy. \nb.\tTable 2 & 3: Also non-uniform LeanQuant accuracy. \nc.\tTable 4: GPTQ Memory values missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)