LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

## Abstract

Abstract Large language models (LLMs) have shown immense potential across various domains, but their high memory requirements and inference costs remain critical challenges for deployment. Post-training quantization (PTQ) has emerged as a promising technique to reduce memory requirements and decoding latency. However, recent accurate quantization methods often depend on specialized computations or custom data formats to achieve better model quality, which limits their compatibility with popular frameworks, as they require dedicated inference kernels tailored to specific hardware and software platforms, hindering wider adoption. Furthermore, many competitive methods have high resource requirements and computational overhead, making it challenging to scale them to hundreds of billions of parameters. In response to these challenges, we propose LeanQuant (Loss-error-aware Network Quantization), a novel quantization method that is accurate, versatile, and scalable. In the existing popular iterative loss-error-based quantization framework, we identify a critical limitation in prior methods: the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals. To overcome this fundamental issue, we propose learning loss-error-aware grids, instead of using non-adaptive min-max affine grids. Our approach not only produces quantized models that are more accurate but also generalizes to a wider range of quantization types, including affine and non-uniform quantization, enhancing compatibility with more frameworks. Extensive empirical evaluations on recent LLMs demonstrate that LeanQuant is highly accurate , comparing favorably against recent competitive baselines in model quality, and scalable , achieving very accurate quantization of Llama-3.1 405B, one of the largest open-source LLMs to date, using two Quadro RTX 8000-48GB GPUs in 21 hours.

## 1 Introduction

Large language models (LLMs) have demonstrated impressive reasoning (Wei et al., ) and problem solving abilities (Kojima et al., ), and have shown the potential to bring transformative changes to various fields such as law (Kaddour et al., ), education (Kasneci et al., ), and medicine (Thirunavukarasu et al., ). However, deploying LLMs in a cost-effective manner presents significant challenges due to their substantial memory and computational demands (Chen et al., ), which hinders the accessibility and democratization of artificial intelligence (AI) (Kaddour et al., ).

Post-training quantization (PTQ) (Krishnamoorthi, ) is a technique for reducing the memory requirement for model inference by reducing the precision of floating-point weights of a pre-trained model and storing them in a compact low-bit-width format. PTQ offers the additional benefit of reducing the decoding latency of LLMs by reducing memory reads, since LLM inference is often bottlenecked by memory bandwidth (Kim et al., ). Although quantization causes a certain amount of precision loss in the parameters, the model quality can be reasonably preserved even in lower bit widths (Frantar et al., ; Chee et al., ). For many tasks, a quantized model is preferred over a full model due to its better size-accuracy trade-off (Dettmers & Zettlemoyer, ). As open-source foundational models continue to scale up in size (Dubey et al., ), accurate and efficient quantization becomes essential for making AI accessible to a wider audience. For instance, serving Llama-3.1 405B (Dubey et al., ) using its original 16-bit weights requires a cluster of 2 nodes, each with 8$\times$80GB GPUs, while the 4-bit quantized version can be deployed on a single nodes of 8$\times$48GB GPUs, eliminating the overhead of inter-node communication.

Challenges of Deploying Quantized Models One of the biggest challenges of successful deployment of quantized models is implementing optimized kernels for quantized GEMM (general matrix multiply) that are tailored to various hardware platforms and software frameworks. In order to accelerate inference of quantized models, fused kernels, which fuse dequantization and matrix multiplication in the same subroutine, have to be implemented and tuned for the specific hardware accelerator. These kernels require specialized designs and tunings for different hardware accelerators to be fully optimized (Park et al., ).

Recent quantization algorithms have chosen to employ specialized computations or custom data formats to reduce the impact of quantization on model quality, but they require more sophisticated kernel designs for efficient inference. For example, AQLM (Egiazarian et al., ) and QUIP# (Tseng et al., ) perform dequantization through look-ups from multi-dimensional or multi-bit codebooks, and Dotzel et al. () proposed new data types such as Student Float to reduce quantization errors. While these approaches demonstrate promising results, their reliance on specialized operations and data formats can hinder their widespread adoption due to the need for optimized inference kernels for each hardware platform and software framework. For example, llama.cpp (Gerganov, ), a popular LLM inference engine that supports mobile devices, only supports affine and non-uniform quantization formats. Consequently, instead of focusing on developing better quantization methods with specialized operations, it may be more worthwhile to investigate improving the accuracy of existing widely adopted quantization formats, such as affine integer quantization and non-uniform quantization, which are supported by popular deep learning libraries (Paszke et al., ) and deployment frameworks (Kwon et al., ).

Scalability Challenges of Accurate Quantization To improve the quality of quantized models, existing approaches often incur higher computational overhead and require more hardware resources. As foundational models scale up in size (Hoffmann et al., ), these quantization approaches may struggle to scale to very large models such as Llama-3.1 405B (405 billion parameters) (Dubey et al., ). For instance, LLM-QAT (Liu et al., ) uses 100K samples of training data and hundreds of GPU-hours to recover the performance of a quantized LLaMA-13B model (Touvron et al., ). For AQLM (Egiazarian et al., ), the time needed for quantizing a 7B to 70B LLM ranges from 1 to 14 days of an A100-80GB GPU. For SqueezeLLM (Kim et al., ), due to its use of the gradients of model parameters, quantizing a 70B LLM requires at least 240GB of total GPU memory, or 8$\times$32GB GPUs. As these accurate quantization approaches demand significant hardware resources and long optimization time, it is crucial to develop accurate methods that are efficient in terms of resource usage and time cost, to ensure the accessibility of increasingly large foundational models.

Our Proposal In this work, we propose LeanQuant, an accurate, versatile, and scalable quantization approach. We build upon the iterative loss-error-based quantization framework (Frantar & Alistarh, ; Frantar et al., ) and identify one of the biggest limitations of such methods: the min-max affine quantization grid introduces high loss errors due to the existence of outliers in the inverse Hessian diagonals. We introduce techniques for learning loss-error-aware quantization grids, which mitigate this issue and greatly improve the accuracy and quality of quantized models. We empirically demonstrate that LeanQuant compares favorably against competitive baselines in the 4/3/2-bit regions. Our approach is versatile, able to generalize to multiple commonly used quantization formats, such as affine and non-uniform quantization, allowing our quantized models to be directly compatible with existing highly optimized inference kernels (Frantar et al., ; Park et al., ) for maximum accessibility. Furthermore, our method is scalable and efficient. By designing and implementing a fused GPU kernel for LeanQuant grid learning, we achieve the accurate quantization of LLMs up to 123B in size using a single L40s-48GB GPU in 4 hours, and Llama-3.1 405B using 2 Quadro RTX 8000-48GB GPUs in 21 hours.

## 2 Background

In this section, we introduce the relevant background for our proposal including quantization grids and iterative loss-error-based quantization.

### 2.1 Quantization Grid

Quantization achieves model compression by representing the full-precision floating-point parameters with a limited set of grid points on the quantization grid. The number of points on the grid depends on the bit width: a $b$-bit code represents $2^{b}$ distinct values, so 2-bit quantization yields 4 grid points. The placement of these points significantly affects the precision of the original parameters, and imprecise placement can degrade model quality. To address this, different types of quantization grids, such as affine, non-uniform, and normal, have been introduced, which we survey as follows.

Affine Grid In an affine quantization grid (Krishnamoorthi, ), the grid points are evenly spaced over the dynamic range of a group of weights. The weights of the network are divided into groups of fixed size, e.g. every 128 contiguous parameters as a group. For min-max asymmetric affine quantization, the scaling factor $S$ and the zero-point $Z$ are recorded for each group of weights (the zero-point is omitted in the symmetric affine case). Then, the $i$-th weight $w_{i}$ in a group $\mathbf{w}$ is quantized to $b$-bit as follows,

$$ $w_{i}^{\mathbf{int}}=\mathrm{clip}(\lfloor\frac{w_{i}}{S}\rceil+Z,0,2^{b}-1), \textrm{where }S=\frac{\max(\mathbf{w})-\min(\mathbf{w})}{2^{b}-1}\textrm{ and }Z=-\lfloor\frac{\min(\mathbf{w})}{S}\rceil$ $$

$$ $\mathrm{quant}_{\textit{aff}}(w_{i},S,Z)=(w_{i}^{\mathbf{int}}-Z)S$ $$

where $\lfloor\cdot\rceil$ is the rounding operator, $\mathrm{clip}(\cdot)$ constrains the weight value within the range of $b$-bit integer, $w_{i}^{\mathbf{int}}$ is the compact integer representation of $w_{i}$, and $\mathrm{quant}_{\textit{aff}}(w_{i},S,Z)$ is the value of $w_{i}$ after being quantized to the nearest grid point.

Non-uniform Grid The grid points on a non-uniform grid are placed in a non-equidistant manner (Li et al., ). The motivation behind non-uniform quantization is to allow for finer precision in regions where model parameters are more concentrated or sensitive. Each row in a weight matrix has a distinct set of grid points $\mathcal{G}$, where $\lvert\mathcal{G}\rvert=2^{b}$ for $b$-bit non-uniform quantization. The weight $w_{i}$ is quantized to the nearest grid point in $\mathcal{G}$ as follows,

$$ $\mathrm{quant}_{\textit{nu}}(w_{i},\mathcal{G})=\operatorname*{arg\,min}_{g\in \mathcal{G}}\lvert g-w_{i}\rvert$ $$

Other Grid Types Previous works have observed that LLM parameters tend to be distributed similarly as Normal or Student T’s Distribution, hence they propose new grid types, such as NormalFloat (Dettmers et al., ) and Student Float (Dotzel et al., ), which place grid lines at the quantiles of these distributions. Our proposed method can also be extended to work with these quantization formats.

### 2.2 Iterative Loss-error-based Quantization

Iterative loss-error-based quantization (Frantar & Alistarh, ) is a promising framework for quantizing deep neural networks to low bit widths while maintaining strong performance on downstream tasks. In particular, Optimal Brain Quantization (OBQ) (Frantar & Alistarh, ), which is based on the seminal works by LeCun et al. () and Hassibi et al. (), aims to minimize the impact of weight perturbations introduced by parameter quantization on the network’s task loss.

Let $\mathcal{L}(\mathbf{w}_{\mathcal{N}})$ be the task loss of a network $\mathcal{N}$ evaluated at the weights $\mathbf{w}_{\mathcal{N}}$ (flattened to a vector). Then, the OBQ objective is to minimize the loss error $\epsilon$, which is defined as

$$ $\begin{split}\epsilon&=\mathcal{L}(\mathbf{w}_{\mathcal{N}}+\bm{\delta}_{ \mathcal{N}})-\mathcal{L}(\mathbf{w}_{\mathcal{N}})\end{split}$ $$

where $\bm{\delta}_{\mathcal{N}}$ is the weight perturbation introduced by quantization. The loss error $\epsilon$ can be approximated with a Taylor series (LeCun et al., ) as

$$ $\epsilon=\underbrace{\big{(}\frac{\partial{\mathcal{L}}}{\partial{\mathbf{w}_{ \mathcal{N}}}}\big{)}^{\top}\bm{\delta}_{\mathcal{N}}}_{\text{negligible}}+ \frac{1}{2}\bm{\delta}_{\mathcal{N}}^{\top}\frac{\partial^{2}{\mathcal{L}}}{ \partial{\mathbf{w}^{2}_{\mathcal{N}}}}\bm{\delta}_{\mathcal{N}}+\underbrace{O \big{(}\lVert\bm{\delta}_{\mathcal{N}}\rVert^{3}\big{)}}_{\text{negligible}}$ $$

where the first term is omitted due to $\frac{\partial{\mathcal{L}}}{\partial{\mathbf{w}_{\mathcal{N}}}}\approx\mathbf
{0}$ in a converged network, and the third and higher terms can be ignored due to small norms. Computing the exact Hessian $\mathbf{H}=\frac{\partial^{2}{\mathcal{L}}}{\partial{\mathbf{w}^{2}_{\mathcal{
N}}}}$ in a deep network is difficult, hence OBQ leverages an approximation of loss error proposed by Nagel et al. (),

$$ $\mathbb{E}(\epsilon)\approx\sum_{\mathbf{W}\in\mathcal{N}}\big{\lVert}\mathbf{ W}\mathbf{X}-\hat{\mathbf{W}}\mathbf{X}\big{\rVert}_{F}^{2}$ $$

where $\mathbf{W},\hat{\mathbf{W}},\mathbf{X}$ are the weight matrix, quantized weight matrix, and the input matrix to a linear layer in the network $\mathcal{N}$. As a result, the OBQ objective can be decomposed into layer-wise independent convex problems,

$$ $\operatorname*{arg\,min}_{\hat{\mathbf{W}}}\lVert\mathbf{W}\mathbf{X}-\hat{ \mathbf{W}}\mathbf{X}\rVert_{F}^{2}$ (1) $$

which can be further decomposed into row-wise independent problems, since Equation 1 can be written as a sum of squares over the rows of $\mathbf{W}$.

OBQ employs an iterative quantization approach, in which a single weight in a row $\mathbf{w}$ is quantized in each step, and then the remaining not-yet-quantized weights in the same row are updated to compensate for the introduced error. Given the constraint that the parameter $w_{i}$, indexed by $i$ in row $\mathbf{w}$, is being quantized, the optimal weight perturbation $\bm{\delta}$ to the remaining weights can be solved with the following Lagrangian,

$$ $L(\bm{\delta},\lambda)=\frac{1}{2}\bm{\delta}^{\top}\mathbf{H}\bm{\delta}+ \lambda\Big{(}\mathbf{e}_{i}^{\top}\bm{\delta}-\big{(}\mathrm{quant}(w_{i})-w_ {i}\big{)}\Big{)}$ (2) $$

where the Hessian $\mathbf{H}=2\mathbf{X}\mathbf{X}^{\top}$ (from Equation 1) is computed on a small sample of input data and $e_{i}$ is the $i$-th standard basis vector. Solving Equation 2 yields the optimal weight perturbation $\bm{\delta}_{i}$ and the resulting loss error $\epsilon_{i}$ after quantizing $w_{i}$,

$$ $\bm{\delta}_{i}=\frac{\mathrm{quant}(w_{i})-w_{i}}{\mathbf{H}^{-1}_{i,i}} \mathbf{H}^{-1}_{:,i},\;\;\epsilon_{i}=\frac{1}{2}\frac{\big{(}\mathrm{quant}( w_{i})-w_{i}\big{)}^{2}}{\mathbf{H}^{-1}_{i,i}}$ (3) $$

where $\mathbf{H}^{-1}_{i,i}$ and $\mathbf{H}^{-1}_{:,i}$ denotes the $i$-th diagonal entry and the $i$-th column of the inverse Hessian, respectively.

The loss error $\epsilon_{i}$ reflects the negative impact of quantizing parameter $w_{i}$ on the model quality, and it is always a non-negative value. OBQ utilizes the loss error $\epsilon_{i}$ as a heuristic for greedy optimization. Specifically, in each iteration, OBQ computes $\epsilon$ for all weights in a row and greedily selects the $i$-th parameter with the minimum $\epsilon_{i}$ to quantize. Then, $w_{i}$ is round to the nearest value on the quantization grid, and the remaining weights are updated via $\mathbf{w}\leftarrow\mathbf{w}-\bm{\delta}_{i}$, and the updated inverse Hessian for the remaining weights, with the $i$-th row and column removed from $\mathbf{H}$, is computed as

$$ ${\mathbf{H}^{-1}_{-i,-i}=\big{(}\mathbf{H}^{-1}-\frac{\mathbf{H}^{-1}_{:,i} \mathbf{H}^{-1}_{i,:}}{\mathbf{H}^{-1}_{i,i}}\big{)}_{-i,-i}}$ (4) $$

This iterative process continues until all weights are quantized.

Scaling to Billion-Parameter LLMs Using Cholesky and Dampening OBQ produces accurate post-training quantized models for million-parameter networks, but fails to scale to billion-parameter LLMs due to two primary reasons: the inefficient time complexity and the accumulation of numerical inaccuracies during updates. To improve its computational efficiency, Frantar et al. () propose to quantize the weights in a fixed non-greedy order for all rows, and keep the weight updates within a block of $B$ columns at a time. To prevent model quality collapse resulted from the accumulation of numerical inaccuracies by repeated weight updates, Frantar et al. () propose to apply a mild dampening (1% of the average diagonals) to the diagonal entries of the Hessian $\mathbf{H}$ and leverage a Cholesky decomposition
of the inverse Hessian $\mathbf{H}^{-1}$ in place of the update in Equation 4. The resulting algorithm is GPTQ, which is able to efficiently quantizes billion-parameter LLMs.

## 3 Methodology

Figure: Figure 1: (Left) The empirical distributions of inverse Hessian diagonals, computed on 262K tokens from the C4 dataset for the Llama-3-8B model, contain outliers that can cause high loss errors. (Right) Our proposed loss-error-aware non-uniform and affine grids better preserves the quantized precision of outliers, leading to more accurate quantized models.
Refer to caption: x1.png

In this section, we introduce our proposed approach Loss-error-aware network Quantization (LeanQuant), for accurately and efficiently quantizing LLMs.

### 3.1 Revisiting the Loss Error

To motivate our proposed approach, we first revisit the loss error $\epsilon_{i}$ in Equation 3, which approximates the (detrimental) increase in the network’s task loss, introduced by quantizing weight $w_{i}$. This error $\epsilon_{i}$ has been used as a heuristic in multiple previous works (LeCun et al., ; Hassibi et al., ; Singh & Alistarh, ; Frantar & Alistarh, ) for choosing the next best weight $i$ to prune or quantize. It has been shown to be a highly informative metric for measuring the impact of quantization.

By examining Equation 3, one finds that the loss error $\epsilon_{i}$ is proportional to the square of weight quantization error and inversely proportional to the diagonal entry of the inverse Hessian, i.e.,

$$ $\epsilon_{i}\propto\big{(}\mathrm{quant}(w_{i})-w_{i}\big{)}^{2}\;\text{ and } \;\epsilon_{i}\propto\frac{1}{\mathbf{H}^{-1}_{i,i}}$ (5) $$

Hence, we further examine the empirical distribution of $\frac{1}{\mathrm{diag}(\mathbf{H}^{-1})}$, to which the loss error row vector $\bm{\epsilon}$ is proportional. We obtain the empirical distributions on layers of Llama-3-8B (Dubey et al., ) with 128 sequences of length 2048 tokens from the C4 dataset (Raffel et al., ), and compute the inverse Hessian as $\mathbf{H}^{-1}=(2\mathbf{X}\mathbf{X}^{\top})^{-1}$ where $\mathbf{X}$ is the layer input matrix. As shown in Figure 1, The majority of the inverse diagonals are concentrated in low-magnitude regions, with a few outliers having high magnitudes. Quantizing the weights corresponding to these outliers can lead to high loss errors if these weights are not well-aligned with the quantization grid points. Preserving the quantized precision of the weights corresponding to these inverse-diagonal outliers is especially important because the loss error increases quadratically with their quantization error (Equation 5). Iterative loss-error-based quantization approaches (OBQ, GPTQ, etc.) employ min-max affine quantization grid, which is suboptimal for preserving the quantized precision of the inverse-diagonal outliers, leading to high loss errors and model quality degradation. Our idea is to learn quantization grids that minimize the loss error $\epsilon$.

### 3.2 Loss-Error-Aware Network Quantization

The min-max affine grid employed by previous iterative loss-error-based quantization methods do not account for the existence of inverse-diagonal outliers, which may cause significant model quality degradation. We propose loss-error-aware quantization grids, which preserve the quantized precision of weights corresponding to the inverse diagonal outliers, to produce more accurate models. We propose techniques to learn loss-error-aware grids for different quantization formats, including non-uniform and affine, and design fused GPU kernels to achieve efficient and scalable quantization.

#### 3.2.1 Non-Uniform Loss-Error-Aware Grid

For non-uniform quantization, we perform clustering with the model parameters, weighted by their corresponding exponentiated inverse Hessian diagonals, to obtain a set of grid points that are loss-error-aware. Concretely, we learn the set of grid points $\mathcal{G}$ for $b$-bit quantization through the following objective,

$$ $\operatorname*{arg\,min}_{\mathcal{G}:\left|\mathcal{G}\right|=2^{b}}\sum_{i}( \mathbf{H}^{-1}_{i,i})^{-p}\left|\mathrm{quant}_{\textit{nu}}(w_{i},\mathcal{G })-w_{i}\right|^{2}$ (6) $$

where $p$ is a hyperparameter that indicates the strength of preserving the quantized precision of outliers. We leverage k-means (Lloyd, ) to optimize the objective. After the quantization grid $\mathcal{G}$ has been established, the weights are iteratively quantized to the nearest grid point in $\mathcal{G}$.

Grid Initialization The quality of clustering results is highly dependent on initialization (Arthur et al., ), since Lloyd’s Algorithm (Lloyd, ) converges to a locally optimal solution. Hence in lower bit width regions, the initialization of grid points for clustering can greatly influence the quantized model quality. K-means++ (Arthur et al., ) leverages iterative sampling to select initial centers that are likely far from each other to obtain more optimal results. However, we found that it still tends to produce sub-optimal results in lower bit width regions (3 and 2-bit). We propose a lightweight and robust grid point initialization method: uniformly spaced grid initialization, which uses uniformly spaced points $\mathcal{G}_{\textrm{init}}$ over the dynamic range of the weights as initial centers, where

$$ $\mathcal{G}_{\textrm{init}}=\Big{\{}\min(\mathbf{w})+\frac{\max(\mathbf{w})- \min(\mathbf{w})}{2^{b}-1}t\,\Big{|}\,t\in\{0,\dots,2^{b}-1\}\Big{\}}$ $$

We perform ablative experiments to study the effectiveness of uniformly spaced grid initialization in Table 11 in the Appendix.

#### 3.2.2 Loss-error-aware Affine Grid

We propose a different strategy for learning affine loss-error-aware grid than non-uniform grid. The challenge with learning affine grid is that the grid points are uniformly placed over an interval, so the clustering strategy does not apply. Searching for the optimal scaling factor $S$ and zero-point $Z$ over the real numbers can be done using gradient descent, but it is potentially time and memory consuming and prone to finding local minimas. Hence, we adopt the strategy of searching for the optimal $S$ and $Z$ in a constrained space $\mathbb{S}$, and formulate the problem of learning affine loss-error-aware grid as

$$ $\operatorname*{arg\,min}_{(S,Z)\in\mathbb{S}}\sum_{i}(\mathbf{H}^{-1}_{i,i})^{ -p}\Big{\lvert}\mathrm{quant}_{\textit{aff}}(w_{i},S,Z)-w_{i}\Big{\rvert}^{2}, \textrm{where }\\ \leavevmode\resizebox{436.8687pt}{}{$\mathbb{S}=\Bigg{\{}\bigg{(}\underbrace{ \frac{\big{(}\max(\mathbf{w})-t_{\mathrm{max}}\frac{R}{T}\big{)}-\big{(}\min( \mathbf{w})+t_{\mathrm{min}}\frac{R}{T}\big{)}}{2^{b}-1}}_{\textrm{scaling factor }S},\underbrace{-\big{\lfloor}\frac{\min(\mathbf{w})+t_{\mathrm{min}} \frac{R}{T}}{S}\big{\rceil}}_{\textrm{zero-point }Z}\bigg{)}\bigg{|}t_{\mathrm {min}},t_{\mathrm{max}}\in\{0,\dots,\frac{T}{2}-1\}\Bigg{\}}$}$ (7) $$

where $R=\max(\mathbf{w})-\min(\mathbf{w})$ is the dynamic range of the weights, and $T$ is the number of partitions over $R$. To obtain the $S$ and $Z$ that minimize the loss errors, we enumerate candidates for $S$ and $Z$ by shrinking the min-max range of the weights in discrete steps. The value of $T$ determines the granularity of the search, which we set to 2048.

Efficient Fused GPU Kernel for Grid Learning The learning process in Equation 7 is costly due to the enumeration of $(\frac{T}{2})^{2}$ pairs of scaling factor and zero-point. Simply looping over all enumerations is time-consuming. Hence, we design and implement a fused kernel that leverages the parallel processing power of GPUs for efficient grid searching. We designate each thread block to be responsible for processing a single group of weights, and each thread in the thread block to be responsible for enumerating all combinations of a single $t_{\mathrm{min}}$ and all $t_{\mathrm{max}}$. The minimum loss for each $t_{\mathrm{min}}$ is accumulated by each thread, and they are aggregated at the end on the thread block level to determine the optimal scaling factor and zero-point for the current weight group. This mode of processing enables the parallel computation of the optimal scaling factor and zero-point for all weight groups, accelerating the end-to-end quantization process by more than 50$\times$. An efficiency analysis of our fused kernel is presented in Section 4.3.

#### 3.2.3 LeanQuant

Our proposed loss-error-aware quantization grid can integrate with any iterative loss-error-based quantization approaches to improve quantized model quality. A comparison between min-max affine quantization grid with loss-error-aware non-uniform and affine grid learned for a layer of Llama-3-8B (Dubey et al., ) is presented in Figure 1. We propose LeanQuant, which integrates loss-error-aware grid with GPTQ (Frantar et al., ), and present it in Algorithm 1. Additionally, for quantizing million-parameter models more accurately, we propose LeanQuant-Exact, which integrates loss-error-aware grid with OBQ (Frantar & Alistarh, ), and present it in Algorithm 2 in the Appendix. We further denote the underlying grid type used by LeanQuant with subscripts, such as LeanQuantaff and LeanQuantnu for affine and non-uniform grid, respectively.

The parameter $p$ balances the strength of precision preservation between inverse-diagonal outliers and non-outliers. Higher values of $p$ prioritizes the precision preservation of outliers more, while $p=0$ prioritizes the preservation of all weights equally. We found the quantized model quality to be not very sensitive to values of $p$, and use $p=4$ for all models. A sensitivity analysis for $p$ is given in Section 4.3.

Figure: Algorithm 1 LeanQuant for LLM quantization

## 4 Experiments

**Table 1: Zero-shot accuracy of quantized LLMs on benchmarks. The results of more models can be found in Table 8 of the Appendix. ^†2-bit quantization is unsupported by the SqueezeLLM codebase.**
|  | Method | Bits | ARC | LAMBADA | MMLU | HellaS | PIQA | WinoG | Avg. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Easy | Chg | Std | OpenAI | STEM | Human. | Social | Other |  |  |  |  |  |  |
| Llama-3-8B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | BF16 | 16 | 80.30 | 50.17 | 68.85 | 75.82 | 53.82 | 54.88 | 73.29 | 70.42 | 60.11 | 79.71 | 73.56 | 67.36 |
|  | GPTQ | 4.00 | 74.83 | 44.11 | 63.42 | 70.75 | 47.29 | 52.28 | 66.04 | 64.89 | 57.98 | 77.26 | 71.82 | 61.58 |
| Affine | OmniQuant | 4.00 | 76.89 | 47.35 | 61.05 | 69.16 | 49.38 | 49.05 | 66.62 | 64.40 | 58.25 | 78.84 | 71.98 | 63.00 |
| LeanQuantaff | 4.00 | 76.60 | 46.93 | 66.89 | 74.07 | 51.89 | 52.96 | 70.04 | 68.43 | 58.47 | 77.91 | 72.77 | 65.18 |  |
| GPTQ | 3.00 | 50.84 | 24.32 | 24.16 | 38.89 | 26.23 | 29.16 | 34.38 | 30.00 | 45.07 | 64.64 | 60.69 | 37.75 |  |
| OmniQuant | 3.00 | 60.90 | 30.12 | 21.08 | 27.63 | 26.32 | 27.80 | 29.51 | 29.90 | 46.98 | 68.17 | 59.98 | 38.95 |  |
| LeanQuantaff | 3.00 | 69.44 | 35.75 | 46.81 | 65.42 | 42.59 | 44.78 | 58.17 | 56.97 | 52.72 | 74.86 | 69.93 | 56.13 |  |
| GPTQ | 2.00 | 25.46 | 22.53 | 0.00 | 0.00 | 21.06 | 23.95 | 21.16 | 23.78 | 25.66 | 52.77 | 51.54 | 24.25 |  |
| OmniQuant | 2.00 | 26.81 | 21.67 | 0.00 | 0.00 | 21.34 | 24.21 | 21.71 | 23.98 | 25.90 | 53.75 | 47.43 | 24.26 |  |
|  | LeanQuantaff | 2.00 | 35.06 | 18.26 | 11.33 | 14.71 | 21.31 | 24.17 | 21.71 | 24.01 | 31.43 | 59.30 | 51.85 | 28.47 |
| Non-uniform | SqueezeLLM | 4.05 | 79.59 | 49.32 | 66.18 | 73.24 | 51.13 | 53.32 | 70.78 | 68.59 | 59.10 | 79.33 | 73.80 | 65.85 |
| LeanQuantnu | 4.05 | 79.50 | 49.15 | 67.36 | 74.95 | 52.17 | 53.16 | 71.40 | 68.75 | 59.19 | 78.89 | 74.11 | 66.24 |  |
| SqueezeLLM | 3.02 | 73.19 | 43.52 | 58.22 | 66.58 | 43.61 | 46.57 | 61.91 | 60.03 | 56.17 | 77.64 | 69.22 | 59.70 |  |
| LeanQuantnu | 3.02 | 77.74 | 47.01 | 63.32 | 72.17 | 48.84 | 49.05 | 65.45 | 62.79 | 56.42 | 78.24 | 71.67 | 62.97 |  |
| SqueezeLLM^† | 2.01 | - N/A - |  |  |  |  |  |  |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 58.21 | 26.62 | 31.22 | 39.16 | 25.98 | 25.48 | 27.01 | 26.65 | 40.78 | 68.01 | 60.38 | 39.05 |  |
| Llama-2-7B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | FP16 | 16 | 76.26 | 43.43 | 68.33 | 73.88 | 34.38 | 39.79 | 47.32 | 47.12 | 57.10 | 78.07 | 68.98 | 57.70 |
|  | GPTQ | 4.00 | 74.16 | 40.78 | 65.38 | 71.94 | 32.67 | 36.92 | 42.61 | 42.61 | 55.99 | 77.48 | 68.32 | 53.47 |
| Affine | OmniQuant | 4.00 | 74.12 | 40.70 | 64.10 | 70.62 | 28.80 | 32.18 | 34.71 | 35.79 | 55.37 | 76.93 | 68.67 | 52.91 |
| LeanQuantaff | 4.00 | 75.00 | 41.21 | 65.03 | 72.02 | 34.82 | 36.94 | 46.77 | 44.54 | 55.32 | 77.15 | 68.75 | 56.14 |  |
| GPTQ | 3.00 | 66.29 | 34.22 | 46.46 | 58.18 | 28.20 | 26.99 | 32.11 | 29.90 | 49.05 | 73.23 | 62.83 | 44.12 |  |
| OmniQuant | 3.00 | 70.12 | 37.29 | 53.27 | 66.66 | 29.05 | 31.05 | 30.61 | 30.38 | 52.58 | 74.05 | 66.46 | 49.23 |  |
| LeanQuantaff | 3.00 | 71.84 | 38.99 | 59.13 | 69.05 | 33.56 | 32.96 | 41.11 | 40.10 | 52.19 | 75.57 | 66.69 | 52.84 |  |
| GPTQ | 2.00 | 25.97 | 21.67 | 0.00 | 0.00 | 21.31 | 23.25 | 21.11 | 23.01 | 25.76 | 51.74 | 48.78 | 23.66 |  |
| OmniQuant | 2.00 | 37.42 | 21.76 | 1.28 | 3.24 | 21.47 | 24.14 | 21.74 | 23.91 | 29.59 | 57.18 | 51.93 | 26.70 |  |
|  | LeanQuantaff | 2.00 | 41.08 | 20.99 | 16.98 | 21.93 | 21.25 | 24.06 | 21.77 | 23.88 | 31.94 | 61.64 | 56.51 | 31.09 |
| Non-uniform | SqueezeLLM | 4.05 | 75.59 | 41.98 | 67.81 | 72.79 | 34.32 | 38.94 | 45.40 | 44.96 | 56.80 | 77.48 | 68.43 | 56.77 |
| LeanQuantnu | 4.05 | 75.97 | 42.66 | 68.14 | 74.25 | 34.35 | 39.06 | 46.05 | 46.51 | 56.03 | 77.86 | 69.38 | 57.30 |  |
| SqueezeLLM | 3.02 | 73.06 | 40.27 | 61.96 | 70.11 | 33.75 | 35.22 | 43.35 | 43.16 | 54.15 | 76.50 | 67.88 | 54.49 |  |
| LeanQuantnu | 3.02 | 73.74 | 40.19 | 66.12 | 73.16 | 32.25 | 35.54 | 43.40 | 43.39 | 53.24 | 76.44 | 68.35 | 55.07 |  |
| SqueezeLLM^† | 2.01 | - N/A - |  |  |  |  |  |  |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 51.81 | 23.98 | 28.68 | 38.21 | 22.26 | 23.89 | 22.49 | 24.01 | 35.88 | 66.38 | 58.17 | 35.98 |  |
| Mistral-7B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | BF16 | 16 | 80.77 | 50.09 | 69.38 | 75.63 | 50.46 | 53.48 | 69.35 | 68.01 | 61.26 | 80.58 | 73.88 | 66.62 |
|  | GPTQ | 4.00 | 79.00 | 46.25 | 66.99 | 73.67 | 46.24 | 50.82 | 66.20 | 64.66 | 59.36 | 79.65 | 72.93 | 62.68 |
| Affine | OmniQuant | 4.00 | 78.49 | 46.25 | 63.28 | 71.20 | 45.96 | 51.35 | 65.68 | 64.76 | 60.19 | 79.87 | 71.90 | 63.54 |
| LeanQuantaff | 4.00 | 79.71 | 48.04 | 68.33 | 75.70 | 47.42 | 51.84 | 68.05 | 66.43 | 59.65 | 80.41 | 73.48 | 65.37 |  |
| GPTQ | 3.00 | 70.54 | 38.65 | 52.63 | 62.10 | 36.31 | 38.89 | 49.20 | 47.86 | 54.76 | 77.58 | 67.96 | 52.60 |  |
| OmniQuant | 3.00 | 70.54 | 35.07 | 35.49 | 46.54 | 33.71 | 32.88 | 40.23 | 37.85 | 52.35 | 75.19 | 63.93 | 47.62 |  |
| LeanQuantaff | 3.00 | 76.94 | 44.62 | 65.63 | 74.60 | 44.18 | 45.59 | 61.20 | 59.03 | 56.36 | 78.89 | 72.30 | 61.76 |  |
| GPTQ | 2.00 | 26.73 | 22.27 | 0.00 | 0.00 | 23.31 | 24.46 | 23.86 | 23.42 | 25.35 | 51.52 | 49.72 | 24.39 |  |
| OmniQuant | 2.00 | 27.06 | 21.67 | 0.00 | 0.00 | 21.25 | 24.29 | 21.71 | 23.98 | 25.89 | 51.25 | 51.54 | 24.42 |  |
|  | LeanQuantaff | 2.00 | 57.91 | 27.22 | 36.91 | 49.00 | 24.23 | 24.91 | 24.60 | 27.29 | 40.27 | 69.15 | 60.46 | 40.18 |
| Non-uniform | SqueezeLLM | 4.05 | 79.73 | 49.06 | 68.28 | 74.93 | 48.81 | 52.73 | 68.87 | 66.98 | 59.80 | 80.25 | 73.56 | 65.73 |
| LeanQuantnu | 4.05 | 79.80 | 48.89 | 69.03 | 76.03 | 48.84 | 52.86 | 68.87 | 66.69 | 60.19 | 80.14 | 74.59 | 65.99 |  |
| SqueezeLLM | 3.02 | 77.54 | 45.93 | 64.06 | 71.43 | 43.96 | 47.93 | 62.69 | 59.16 | 58.76 | 79.43 | 71.98 | 62.08 |  |
| LeanQuantnu | 3.02 | 77.74 | 45.99 | 67.59 | 76.07 | 44.24 | 47.97 | 62.14 | 62.47 | 57.28 | 79.27 | 72.22 | 63.00 |  |
| SqueezeLLM^† | 2.01 | - N/A - |  |  |  |  |  |  |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 63.47 | 30.55 | 41.01 | 54.61 | 31.34 | 29.97 | 32.14 | 33.96 | 42.29 | 71.38 | 64.01 | 44.97 |  |

In this section, we perform extensive experiments to validate the effectiveness and scalability of our proposed LeanQuant for quantizing LLMs against competitive baselines. We first introduce the baselines, models, evaluation metrics and datasets, and hardware used for the experiments. We then describe the experimental results and findings, and analyze the efficiency and scalability of our proposal. Finally, we perform ablative experiments to validate each component of proposed approach.

Baselines We compare LeanQuantaff against competitive affine quantization approaches AWQ (Lin et al., ), GPTQ (Frantar et al., ), and OmniQuant (Shao et al., ), and LeanQuantnu against the existing state-of-the-art non-uniform method SqueezeLLM (Kim et al., ). For the baselines, we use the quantized models provided by their official repository where possible, and quantize the unavailable models using their official codebase and recommended hyperparameters. More details on baseline reproduction and evaluation methods can be found in Section C of the Appendix. For all LeanQuant models, we use a small calibration set of 128 sequences of 2048 tokens from the C4 dataset (Raffel et al., ) for computing the Hessian $\mathbf{H}$, and set $p=4$.

Models We consider the following recent, popular LLMs for quantization: Llama 1/2/3 series models (Touvron et al., ; ; Dubey et al., ), Mistral-7B-v0.1 (Jiang et al., ), Mistral-Large-Instruct-2407 (123B) (Mistral AI Team, ), and Llama-3.1-405B-Instruct (Dubey et al., ).

Evaluation Metrics and Datasets We evaluate quantized LLMs using the perplexity metric on the datasets WikiText2 (Merity et al., ) and C4 (Raffel et al., ), and zero-shot accuracy on the benchmarks ARC (Clark et al., ), LAMBADA (Paperno et al., ), MMLU (Hendrycks et al., ), HellaSwag (Zellers et al., ), PIQA (Bisk et al., ), and WinoGrande (Sakaguchi et al., ). We also quantize and evaluate the instruction-following Llama-3-8B-Instruct using OpenAI GPT-4o (2024-05-13) as a judge on the MT-Bench (Zheng et al., ), and the results are presented in Section E in the Appendix.

Testbed Hardware LeanQuant models are quantized using a machine quipped with an L40s-48GB GPU, an AMD EPYC 7R13 48-Core CPU, and 370GB of RAM. To fit Llama-3.1-405B-Instruct in RAM, which is around 800GB in size, we use a machine equipped with 2 Quadro RTX 8000 GPUs, an AMD EPYC 7742 64-Core CPU, and 1.48TB of RAM.

**Table 2: Zero-shot accuracy comparison of the quantized 123B Mistral-Large-Instruct-2407 model.**
| Model | Method | Bits | Arc | LAMBADA | MMLU | Avg. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Easy | Chg. | Std. | OpenAI | STEM | Human. | Social | Other |  |  |  |  |
| Mistral-Large-Instruct-2407 | GPTQ | 4.00 | 84.60 | 63.99 | 74.38 | 80.52 | 76.31 | 77.23 | 89.31 | 85.23 | 78.95 |
| LeanQuantaff | 4.00 | 85.14 | 63.99 | 74.99 | 81.14 | 76.56 | 77.32 | 89.21 | 85.87 | 79.28 |  |

**Table 3: Zero-shot accuracy comparison of the quantized Llama-3.1-405B-Instruct model.**
| Model | Method | Group Size | Bits | Arc-E | Arc-C | LAMBADA-Std | PIQA | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3.1-405B-Instruct | GPTQ | 128 | 4.25 | 88.21 | 65.10 | 76.96 | 82.75 | 78.26 |
| LeanQuantaff | 128 | 4.25 | 88.34 | 65.70 | 77.86 | 83.03 | 78.73 |  |

### 4.1 Main Results

Accuracy and Perplexity The zero-shot accuracy of quantized models on benchmarks are presented in Table 1, as well as in Table 8 in the Appendix, and the perplexity results are shown in Table 7 in the Appendix. At the same bit width, LeanQuant achieves significantly better (lower) perplexity than GPTQ and AWQ, and performs on par with OmniQuant and SqueezeLLM. However, perplexity may not be a representative metric for evaluating the accuracy of quantized models. In terms of zero-shot accuracy on various benchmarks, LeanQuantaff mostly outperforms GPTQ and OmniQuant, and LeanQuantnu similarly performs better than SqueezeLLM in most cases. We highlight that LeanQuantaff improves the average zero-shot accuracy on 11 tasks over OmniQuant by 17.18% for 3-bit Llama-3-8B, and by 14.14% for 3-bit Mistral-7B. Compared to GPTQ, LeanQuantaff improves the average zero-shot accuracy by 18.38% for 3-bit Llama-3-8B, and by 9.16% for 3-bit Mistral-7B.

Effectiveness on Very Large LLMs We quantize the 123B Mistral-Large-Instruct-2407 and the 405B Llama-3.1 model using LeanQuantaff and GPTQ, and present their zero-shot accuracy in Table 2 and 3, respectively. OmniQuant and SqueezeLLM fail to quantize to these models due to GPU out-of-memory errors. For Llama-3.1 405B, we use a smaller set of evaluation tasks due to the high inference costs. LeanQuantaff models mostly outperforms GPTQ in zero-shot accuracy. We employ row-wise quantization for Mistral-Large and group-wise quantization (with size 128) for Llama-3.1. This showcases that our method is effective for both row-wise affine quantization and group-wise affine quantization.

### 4.2 Memory and Time Efficiency

We report the maximum GPU memory consumption of LeanQuant and the baselines during quantization on models of different sizes in Table 5. LeanQuant is significantly more memory efficient than OmniQuant and SqueezeLLM: it successfully scales to 123B Mistral-Large using a single 48GB GPU, and to 405B Llama-3.1 models using two 48GB GPUs, while OmniQuant fails to quantize Llama-3-70B and SqueezeLLM fails to quantize Llama-3-8B on a single 48GB GPU. The time cost of LeanQuant for different sized models are reported in Table 9 in the Appendix. LeanQuant can quantize 7B/8B models in less than an hour, the 123B model in 4.2 hours, and the 405B model in 20.7 hours.

### 4.3 Ablation Study

Q1: Does LeanQuant effectively reduce the loss error $\epsilon$ compared to other iterative loss-error-based methods? Yes, LeanQuant effectively reduces loss errors $\epsilon$ compared to GPTQ, as shown in Figure 2, as well as in Figure 4 in the Appendix. The sum of loss errors are computed as Equation 3. Moreover, non-uniform LeanQuant generally achieves lower loss errors than affine LeanQuant, due to more degrees of freedom in the grid point placements, which also explains why LeanQuantnu achieves higher accuracy than LeanQuantaff on benchmarks in Table 1.

Q2: Is LeanQuant sensitive to the hyperparameter $p$? No, we found LeanQuant to be not very sensitive to $p$. A sensitivity analysis on the hyperparameter $p$ is given in Table 10 in the Appendix. LeanQuant works well with $p$ values of 3 or 4.

Q3: Is uniformly spaced grid initialization beneficial for model quality? Yes, uniformly spaced grid initialization consistently outperforms k-means++ (Arthur et al., ) initialization on different models in 3-bit and 2-bit regions, as shown in Table 11 in the Appendix.

Q4: Does the fused GPU kernel for LeanQuantaff accelerate quantization? Yes, our fused kernel for learning affine grids accelerate the end-to-end quantization process by more than 50$\times$, as shown in Table 5, which enables LeanQuant to be scaled to very large models.

Figure: Figure 2: Comparison of loss errors $\epsilon$, summed over each layer, for GPTQ and LeanQuant (affine and non-uniform) during iterative quantization.
Refer to caption: x2.png

**Table 4: Peak GPU memory consumption of different algorithms during 4-bit quantization. “OOM” indicates out of memory on a single 48GB GPU, except for Llama-3.1-405B where we use 2 48GB GPUs.**
| Model | OmniQuant | SqueezeLLM | LeanQuant |
| --- | --- | --- | --- |
| Llama-3-8B | 25.3 GB | OOM | 7.9 GB |
| Llama-3-70B | OOM | OOM | 17.2 GB |
| Mistral-Large (123B) | OOM | OOM | 33.0 GB |
| Llama-3.1-405B | OOM | OOM | 65.4 GB |

## 5 Related Works

Iterative Loss-error-based Compression Optimal Brain Damage (LeCun et al., ) introduced a saliency-score-based iterative pruning algorithm for neural networks, and Optimal Brain Surgeon (Hassibi & Stork, ; Hassibi et al., ) extended it to apply a weight update to compensate for the error introduced in each iteration. These methods inspired a number of works on model pruning (Guo et al., ; Singh & Alistarh, ; Yu et al., ) and weight quantization (Li et al., ; Frantar & Alistarh, ; Frantar et al., ).

Efficient LLM Inference LLM inference is computationally and memory demanding, and existing works accelerate inference and reduce memory requirements through post-training weight quantization (Dettmers et al., ; Lin et al., ; Frantar et al., ; Chee et al., ; Kim et al., ; Shao et al., ; Egiazarian et al., ; Tseng et al., ), pruning (Frantar & Alistarh, ; Ashkboos et al., ), weight-activation quantization (Xiao et al., ), offloading Sheng et al. (), etc.

## 6 Conclusion

In this work, we propose LeanQuant, an accurate, versatile, and scalable quantization method for LLMs. Motivated by the finding that the min-max affine grid causes large errors in the network’s task loss in iterative loss-error-based methods, we propose to learn loss-error-aware grids to enable more accurate quantized models, and design fused kernels for efficient and scalable quantization. Our method generalizes to multiple quantization formats to enable greater accessibility. Extensive empirical evaluations reveal that our quantized models compares favorably against competitive baselines in accuracy, and can scale to Llama-3.1 405B, one of the largest open-source LLM to date.

## Appendix

## Appendix A LeanQuant-Exact

The pseudocode of LeanQuant-Exact for accurately quantizing million-parameter networks is presented in Algorithm 2.

Figure: Algorithm 2 LeanQuant-Exact for Millon-parameter Networks

### A.1 BERT Experiments with LeanQuant-Exact

We compare the performance of BERT models (Devlin et al., ), quantized with OBQ (Frantar & Alistarh, ) and LeanQuantnu-Exact, on the SQuAD dataset (Rajpurkar et al., ). We quantize the 12-layer BERT-base (Devlin et al., ) and the 3-layer BERT-3 variant from Kurtic et al. () to 3 and 4 bits. OBQ and LeanQuant-Exact are calibrated using 1024 samples from the training set, and the F1 score is reported on the test set.

**Table 6: F1 scores on SQuAD of BERT models quantized using OBQ and LeanQuantnu-Exact. LeanQuantnu-Exact outperforms OBQ in maintaining model quality.**
| Method | Bits | BERT-3 | BERT |
| --- | --- | --- | --- |
| FP32 | 32 | 84.66 | 88.53 |
| OBQ | 4.03 | 84.40 | 87.96 |
| LeanQuantnu-Exact | 4.13 | 84.58 | 88.49 |
| OBQ | 3.03 | 83.47 | 84.72 |
| LeanQuantnu-Exact | 3.06 | 84.20 | 86.21 |

## Appendix B Discussion on Error Accumulation During Iterative Quantization

LeanQuant prevents drastic increase to the task loss by learning the quantization grid for better preservation of the precision of outlier inverse diagonals. However, since the not-yet-quantized weights will shift during the iterative quantization process and the quantization grid is fixed beforehand, one potential problem arises: the quantization grid is no longer well-aligned with the outliers after certain iterations. Fortunately, this is not a problem in practice. The loss-error-awareness property of LeanQuant grids prevents high-norm weight perturbations $\bm{\delta}_{i}$ (Equation 3) from ocurring, hence the weights do not shift by much during the iterations. Furthermore, no new inverse-diagonal outliers will arise during the iterative quantization process, since the remaining inverse diagonals only decrease in magnitude towards zero after each column and row removal (Equation 4).

## Appendix C Experiment Details

##### Baseline Reproduction

We use the quantized models provided by the official repository where possible. We obtained quantized LLaMA-7B, LLaMA-13B, Llama-2-7B, Llama-2-13B from the OmniQuant repository, and LLaMA-7B, LLaMA-13B, Llama-2-7B, Llama-2-13B, Mistral-7B from the SqueezeLLM repository. We obtained the community-driven GPTQ-quantized version of Llama-3.1-405B-Instruct from HuggingFace (^1^11[https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-GPTQ-INT4](https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-GPTQ-INT4)). The other quantized models are reproduced using the official codebases and recommended hyperparameters. For OmniQuant, we set the training epochs to 20, enable Learnable Weight Clipping (LWC), set an LWC learning rate of 1e-2. For SqueezeLLM, there is no tunable parameters. For GPTQ, we turn on activation ordering (quantizing columns in order of decreasing activation size) for more accurate model.

##### Perplexity Evaluations

We follow the perplexity evaluation procedure described by (Frantar et al., ): sequences from the test set of the WikiText2 and C4 datasets (Merity et al., ; Raffel et al., ) are concatenated into 128 sequences of length 2048 tokens for perplexity testing.

##### Accuracy Evaluations

We use lm-evaluation-harness (Gao et al., ) for evaluating zero-shot accuracy on tasks. The task names we evaluate are lambada, ai2_arc, winogrande, piqa, hellaswag, mmlu.

## Appendix D Perplexity Evaluations

The perplexity evaluation results on WikiText2 (Merity et al., ) and C4 (Raffel et al., ) for quantized models are presented in Table 7.

## Appendix E LLM-as-a-Judge

LLM as a Judge The evaluation results on MT-Bench using GPT-4o (2024-05-13) as a judge are presented in Figure 3. We pitch 3-bit and 4-bit, with group size of 128, LeanQuantaff against OmniQuant, and 4-bit LeanQuantnu against SqueezeLLM. LeanQuant achieves higher win rate than the baselines.

Figure: Figure 3: Evaluation of quantized Llama-3-8B-Instruct on MT-Bench using OpenAI GPT-4o as a judge. The win rates reported exclude ties.
Refer to caption: x3.png

## Appendix F Accuracy Results on More Models

The zero-shot accuracy results on benchmarks for quantized LLaMA-7B, LLaMA-13B, Llama-2-7B (Touvron et al., ; ) are presented in Table 8.

**Table 7: Perplexity evaluations of Llama models under different quantization methods and bit widths. The results of GPTQ, AWQ, OmniQuant are from Shao et al. (), and the results of SqueezeLLM are from Kim et al. (). ^† The official SqueezeLLM code does not support 2-bit quantization, and we report the available results from Kim et al. ().**
|  |  |  | WikiText-2 | C4 |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Grid | Method | Bits | 1-7B | 1-13B | 2-7B | 2-13B | 2-70B | 1-7B | 1-13B | 2-7B | 2-13B | 2-70B | Avg. |
|  | FP16 | 16 | 5.58 | 5.09 | 5.47 | 4.88 | 3.31 | 7.08 | 6.61 | 6.97 | 6.46 | 5.52 | 5.697 |
| Affine | GPTQ | 4.00 | 6.13 | 5.40 | 5.83 | 5.13 | 3.58 | 7.43 | 6.84 | 7.37 | 6.70 | 5.67 | 6.008 |
| AWQ | 4.00 | 6.08 | 5.34 | 6.15 | 5.12 | - | 7.52 | 6.86 | 7.68 | 6.74 | - | - |  |
| OmniQuant | 4.00 | 5.86 | 5.21 | 5.74 | 5.02 | 3.47 | 7.34 | 6.76 | 7.35 | 6.65 | 5.65 | 5.905 |  |
| LeanQuantaff | 4.00 | 5.92 | 5.25 | 5.73 | 5.08 | 3.49 | 7.30 | 6.76 | 7.25 | 6.63 | 5.63 | 5.904 |  |
| Non-uniform | SqueezeLLM | 4.04-4.05 | 5.79 | 5.18 | 5.62 | 4.99 | 3.41 | 7.21 | 6.71 | 7.12 | 6.57 | 5.58 | 5.818 |
| LeanQuantnu | 4.04-4.05 | 5.81 | 5.19 | 5.64 | 4.99 | 3.42 | 7.21 | 6.70 | 7.13 | 6.57 | 5.58 | 5.824 |  |
| Affine | GPTQ | 3.00 | 8.06 | 6.76 | 8.37 | 6.44 | 4.82 | 9.49 | 8.16 | 9.81 | 8.02 | 6.57 | 7.650 |
| AWQ | 3.00 | 11.88 | 7.45 | 24.00 | 10.45 | - | 13.26 | 9.13 | 23.85 | 13.07 | - | - |  |
| OmniQuant | 3.00 | 6.49 | 5.68 | 6.58 | 5.58 | 3.92 | 8.19 | 7.32 | 8.65 | 7.44 | 6.06 | 6.591 |  |
| LeanQuantaff | 3.00 | 6.62 | 5.76 | 6.61 | 5.66 | 3.91 | 7.98 | 7.19 | 8.27 | 7.23 | 5.90 | 6.513 |  |
| Non-uniform | SqueezeLLM | 3.02 | 6.32 | 5.60 | 6.18 | 5.36 | 3.77 | 7.75 | 7.08 | 7.72 | 6.97 | 5.83 | 6.258 |
| LeanQuantnu | 3.02 | 6.34 | 5.60 | 6.19 | 5.40 | 3.80 | 7.74 | 7.05 | 7.73 | 6.98 | 5.83 | 6.266 |  |
| Affine | GPTQ | 2.00 | 1.1E5 | 6.8E4 | 3.8E4 | 5.6E4 | 2.0E4 | 689.13 | 2.5E3 | NaN | 323.12 | 48.82 | NaN |
| OmniQuant | 2.00 | 15.47 | 13.21 | 37.37 | 17.21 | 7.81 | 24.89 | 18.31 | 90.64 | 26.76 | 12.28 | 26.395 |  |
| LeanQuantaff | 2.00 | 18.53 | 14.42 | 25.69 | 24.43 | 7.92 | 19.99 | 16.53 | 27.11 | 20.92 | 10.84 | 18.638 |  |
| Non-uniform | SqueezeLLM^† | 2.01 | - N/A - | 61.25 | 10.86 | - N/A - | N/A |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 15.65 | 9.64 | 15.51 | 10.06 | 6.35 | 17.62 | 10.93 | 17.07 | 11.83 | 7.96 | 12.262 |  |

**Table 8: Zero-shot accuracy of more quantized LLMs on benchmarks.**
|  | Method | Bits | ARC | LAMBADA | MMLU | HellaS | PIQA | WinoG | Avg. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Easy | Chg | Std | OpenAI | STEM | Human. | Social | Other |  |  |  |  |  |  |
| LLaMA-7B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | FP16 | 16 | 75.29 | 41.81 | 67.77 | 73.49 | 28.20 | 32.03 | 31.65 | 36.53 | 56.92 | 78.67 | 70.09 | 53.86 |
|  | GPTQ | 4.00 | 73.61 | 39.51 | 65.61 | 71.98 | 24.29 | 28.12 | 25.80 | 31.16 | 55.61 | 77.80 | 70.40 | 49.03 |
| Affine | OmniQuant | 4.00 | 74.49 | 39.68 | 65.38 | 71.96 | 30.32 | 30.92 | 33.38 | 35.79 | 55.82 | 78.45 | 68.67 | 53.17 |
| LeanQuantaff | 4.00 | 74.03 | 41.64 | 63.75 | 70.56 | 31.27 | 33.13 | 32.66 | 37.79 | 56.31 | 78.40 | 69.30 | 53.53 |  |
| GPTQ | 3.00 | 66.67 | 35.84 | 49.89 | 58.37 | 25.94 | 27.46 | 23.46 | 24.98 | 51.76 | 75.35 | 64.72 | 43.78 |  |
| OmniQuant | 3.00 | 72.22 | 38.48 | 59.67 | 69.20 | 27.18 | 27.65 | 25.64 | 29.61 | 52.99 | 76.55 | 67.17 | 49.67 |  |
| LeanQuantaff | 3.00 | 73.70 | 40.78 | 65.28 | 72.66 | 27.85 | 31.37 | 30.61 | 35.31 | 56.23 | 78.45 | 70.09 | 52.94 |  |
| GPTQ | 2.00 | 26.35 | 22.01 | 0.00 | 0.00 | 23.69 | 25.08 | 24.28 | 24.01 | 25.69 | 53.70 | 50.99 | 24.95 |  |
| OmniQuant | 2.00 | 51.05 | 22.70 | 11.80 | 23.13 | 26.51 | 26.04 | 24.37 | 23.69 | 34.71 | 64.36 | 54.30 | 32.97 |  |
|  | LeanQuantaff | 2.00 | 55.98 | 27.22 | 38.56 | 47.45 | 24.45 | 25.31 | 22.85 | 25.46 | 37.19 | 68.12 | 60.77 | 39.40 |
| Non-uniform | SqueezeLLM | 4.05 | 76.56 | 46.25 | 69.53 | 75.22 | 32.98 | 37.19 | 42.18 | 44.00 | 59.29 | 78.62 | 71.82 | 57.60 |
| LeanQuantnu | 4.05 | 76.39 | 45.05 | 71.55 | 76.48 | 34.76 | 38.77 | 46.12 | 47.18 | 59.29 | 78.78 | 73.24 | 58.87 |  |
| SqueezeLLM | 3.02 | 75.46 | 43.77 | 65.07 | 72.75 | 30.45 | 34.24 | 37.18 | 40.46 | 57.32 | 78.29 | 71.35 | 55.12 |  |
| LeanQuantnu | 3.02 | 75.17 | 43.00 | 70.41 | 77.29 | 33.81 | 38.32 | 43.16 | 45.06 | 57.35 | 78.29 | 71.35 | 57.56 |  |
| SqueezeLLM | 2.01 | - N/A - |  |  |  |  |  |  |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 50.38 | 24.40 | 31.67 | 41.30 | 21.79 | 24.19 | 21.74 | 24.36 | 37.49 | 65.67 | 58.33 | 36.48 |  |
| LLaMA-13B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | FP16 | 16 | 77.40 | 46.42 | 71.12 | 76.19 | 36.41 | 41.55 | 48.49 | 48.54 | 59.92 | 79.16 | 72.69 | 59.81 |
|  | GPTQ | 4.00 | 77.06 | 45.56 | 69.12 | 75.28 | 34.44 | 39.15 | 45.95 | 46.73 | 58.99 | 78.56 | 72.53 | 56.63 |
| Affine | OmniQuant | 4.00 | 75.97 | 45.22 | 68.25 | 75.59 | 35.30 | 40.21 | 48.20 | 47.25 | 59.11 | 78.94 | 72.61 | 58.79 |
| LeanQuantaff | 4.00 | 76.39 | 46.42 | 70.48 | 76.27 | 35.52 | 39.45 | 46.18 | 47.22 | 58.82 | 78.94 | 72.30 | 58.91 |  |
| GPTQ | 3.00 | 70.92 | 39.93 | 57.29 | 64.82 | 29.37 | 31.94 | 33.18 | 35.34 | 54.05 | 76.99 | 68.43 | 49.13 |  |
| OmniQuant | 3.00 | 75.42 | 42.83 | 60.80 | 71.34 | 29.56 | 34.24 | 36.24 | 41.17 | 57.27 | 77.97 | 69.61 | 54.22 |  |
| LeanQuantaff | 3.00 | 75.84 | 43.34 | 67.49 | 74.85 | 33.37 | 36.56 | 41.92 | 44.74 | 56.75 | 77.97 | 70.48 | 56.66 |  |
| GPTQ | 2.00 | 27.10 | 21.93 | 0.02 | 0.00 | 23.37 | 25.50 | 23.56 | 24.49 | 25.76 | 53.16 | 49.72 | 24.75 |  |
| OmniQuant | 2.00 | 59.51 | 29.52 | 17.85 | 23.35 | 22.52 | 24.12 | 22.81 | 24.46 | 42.01 | 67.25 | 56.12 | 35.41 |  |
|  | LeanQuantaff | 2.00 | 61.45 | 29.27 | 44.63 | 50.82 | 28.73 | 26.01 | 27.20 | 27.26 | 39.08 | 71.27 | 65.82 | 42.87 |
| Non-uniform | SqueezeLLM | 4.04 | 76.56 | 46.25 | 69.53 | 75.22 | 32.98 | 37.19 | 42.18 | 44.00 | 59.29 | 78.62 | 71.82 | 57.60 |
| LeanQuantnu | 4.04 | 76.39 | 45.05 | 71.55 | 76.48 | 34.76 | 38.77 | 46.12 | 47.18 | 59.29 | 78.78 | 73.24 | 58.87 |  |
| SqueezeLLM | 3.02 | 75.46 | 43.77 | 65.07 | 72.75 | 30.45 | 34.24 | 37.18 | 40.46 | 57.32 | 78.29 | 71.35 | 55.12 |  |
| LeanQuantnu | 3.02 | 75.17 | 43.00 | 70.41 | 77.29 | 33.81 | 38.32 | 43.16 | 45.06 | 57.35 | 78.29 | 71.35 | 57.56 |  |
| SqueezeLLM | 2.01 | - N/A - |  |  |  |  |  |  |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 65.66 | 32.42 | 54.49 | 66.93 | 23.44 | 25.50 | 23.98 | 28.42 | 45.66 | 72.80 | 66.14 | 45.95 |  |
| Llama-2-13B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | FP16 | 16 | 79.50 | 48.46 | 70.35 | 76.73 | 42.28 | 47.89 | 61.16 | 59.38 | 60.06 | 79.05 | 72.22 | 63.37 |
|  | GPTQ | 4.00 | 78.32 | 45.48 | 68.33 | 75.35 | 40.28 | 46.08 | 56.48 | 54.65 | 58.92 | 78.45 | 71.82 | 59.59 |
| Affine | OmniQuant | 4.00 | 77.69 | 47.10 | 68.74 | 75.57 | 41.39 | 46.10 | 57.39 | 55.87 | 59.48 | 79.00 | 70.32 | 61.70 |
| LeanQuantaff | 4.00 | 79.42 | 47.27 | 69.16 | 75.90 | 42.21 | 47.31 | 59.90 | 57.93 | 59.07 | 78.24 | 71.82 | 62.57 |  |
| GPTQ | 3.00 | 72.85 | 39.85 | 59.77 | 67.20 | 34.86 | 38.85 | 47.97 | 46.48 | 54.61 | 76.28 | 70.32 | 53.62 |  |
| OmniQuant | 3.00 | 76.60 | 43.34 | 60.70 | 70.54 | 38.60 | 42.59 | 53.23 | 51.82 | 57.42 | 77.97 | 69.14 | 58.36 |  |
| LeanQuantaff | 3.00 | 77.31 | 44.54 | 68.15 | 75.88 | 37.93 | 43.80 | 53.07 | 52.62 | 56.36 | 76.99 | 70.72 | 59.76 |  |
| GPTQ | 2.00 | 25.84 | 20.22 | 0.00 | 0.00 | 22.84 | 25.59 | 23.53 | 23.98 | 25.97 | 52.07 | 47.75 | 24.19 |  |
| OmniQuant | 2.00 | 48.19 | 24.66 | 10.21 | 20.14 | 21.34 | 24.21 | 21.77 | 23.85 | 40.16 | 63.00 | 52.33 | 31.81 |  |
|  | LeanQuantaff | 2.00 | 50.88 | 24.32 | 32.70 | 39.57 | 21.50 | 24.38 | 21.90 | 24.40 | 38.01 | 67.19 | 56.91 | 36.52 |
| Non-uniform | SqueezeLLM | 4.04 | 78.91 | 47.70 | 70.00 | 76.23 | 42.72 | 47.89 | 60.19 | 58.32 | 59.74 | 78.73 | 72.77 | 63.02 |
| LeanQuantnu | 4.04 | 78.91 | 47.56 | 71.12 | 77.43 | 43.51 | 47.44 | 59.54 | 58.83 | 59.58 | 78.62 | 72.06 | 63.15 |  |
| SqueezeLLM | 3.02 | 77.27 | 43.17 | 66.37 | 73.80 | 38.22 | 44.63 | 55.18 | 53.11 | 58.74 | 77.86 | 69.46 | 59.80 |  |
| LeanQuantnu | 3.02 | 77.19 | 44.20 | 71.14 | 78.59 | 40.72 | 45.46 | 56.87 | 55.10 | 56.38 | 77.75 | 70.09 | 61.23 |  |
| SqueezeLLM | 2.01 | - N/A - |  |  |  |  |  |  |  |  |  |  |  |  |
| LeanQuantnu | 2.01 | 62.46 | 30.20 | 47.00 | 61.09 | 25.28 | 27.74 | 27.56 | 28.87 | 42.20 | 69.91 | 62.04 | 44.03 |  |

## Appendix G Quantization Time Cost

The time cost of LeanQuant for different models and configurations are presented in Table 9.

**Table 9: Total time taken by LeanQuant for quantizing different-sized LLMs, using a single L40s-48GB GPU, an AMD EPYC 7R13 48-Core CPU, and 370GB of RAM. Llama-3.1-405B is quantized using 2 Quadro RTX 8000 GPUs, an AMD EPYC 7742 64-Core CPU, and 1.48TB of RAM.**
| Model | Grid | Group Size | Bits | Time |
| --- | --- | --- | --- | --- |
| Llama-2-7B | Affine | - | 4.00 | 14 mins |
| Affine | 128 | 4.25 | 15 mins |  |
| Non-uniform | - | 4.05 | 35 mins |  |
| Llama-3-8B | Affine | - | 4.00 | 16 mins |
| Affine | 128 | 4.25 | 20 mins |  |
| Non-uniform | - | 4.05 | 37 mins |  |
| Llama-2-70B | Affine | - | 4.00 | 178 mins |
| Non-uniform | - | 4.04 | 335 mins |  |
| Mistral-Large-Instruct-2407 (123B) | Affine | - | 4.00 | 252 mins |
| Llama-3.1-405B | Affine | 128 | 4.25 | 1241 mins |

## Appendix H Ablation Study

Sensitivity to Hyperparameter $p$ Ablative experiments on the effects of the hyperparameter $p$ on the quality of LeanQuant models are presented in Table 10. In the case of $p=0$, the inverse Hessian diagonals are ignored as the weights for clustering, and the centroids are learned based on the density of weights. It is worth noting that $p=0$ results in sub-optimal model quality compared to higher values of $p$, which means that the loss-error-awareness property of the quantization grid is critical for maintaining model quality.

Grid Point Initialization Ablative experiments comparing k-means++ initialization with our proposed uniformly spaced grid initialization are presented in Table 11.

**Table 10: The perplexity of LeanQuant models on WikiText2 and C4, using different values of $p$.**
|  |  |  | WikiText2 | C4 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Grid | Hyperparameter | 4-bit | 3-bit | 2-bit | 4-bit | 3-bit | 2-bit |
|  |  | $p=0$ | 12.13 | 29.92 | 5,991.18 | 16.73 | 22.71 | 5,998.96 |
|  |  | $p=2$ | 5.39 | 5.98 | 25.09 | 7.89 | 8.50 | 20.27 |
|  |  | $p=3$ | 5.37 | 5.92 | 22.32 | 7.88 | 8.48 | 19.81 |
|  | Non-uniform | $p=4$ | 5.38 | 5.96 | 25.61 | 7.88 | 8.47 | 21.65 |
|  |  | $p=0$ | 14.52 | 80.54 | 230.66 | 16.94 | 69.04 | 243.65 |
|  |  | $p=2$ | 5.52 | 8.58 | 55.50 | 8.03 | 16.84 | 41.99 |
|  |  | $p=3$ | 5.51 | 6.36 | 18.33 | 8.03 | 8.80 | 20.20 |
| Mistral-7B | Affine | $p=4$ | 5.51 | 6.31 | 18.00 | 8.02 | 8.86 | 20.47 |
|  |  | $p=0$ | 5.69 | 6.76 | NaN | 7.15 | 8.23 | 62.00 |
|  |  | $p=2$ | 5.65 | 6.30 | 17.16 | 7.13 | 7.83 | 19.14 |
|  |  | $p=3$ | 5.64 | 6.25 | 17.84 | 7.13 | 7.80 | 19.55 |
|  | Non-uniform | $p=4$ | 5.64 | 6.28 | 15.82 | 7.14 | 7.83 | 18.89 |
|  |  | $p=0$ | 5.84 | 8.19 | 93.01 | 7.30 | 9.54 | 85.62 |
|  |  | $p=2$ | 5.77 | 7.33 | 27.82 | 7.27 | 8.83 | 28.86 |
|  |  | $p=3$ | 5.75 | 6.80 | 25.97 | 7.26 | 8.32 | 27.57 |
| Llama-2-7B | Affine | $p=4$ | 5.75 | 6.69 | 26.82 | 7.25 | 8.29 | 28.14 |

## Appendix I Loss Error Comparison

A comparison of the sum of loss errors $\epsilon$ between GPTQ and LeanQuant (affine and non-uniform) is presented in Figure 4.

Figure: Figure 4: Comparison of loss errors $\epsilon$ of each layer for GPTQ and LeanQuant (affine and non-uniform) during iterative quantization.
Refer to caption: x4.png

**Table 11: Ablative experiments on grid point initialization.**
|  |  | Llama-2-7B | Llama-3-8B | Mistral-7B |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Grid Init. | 4-bit | 3-bit | 2-bit | 4-bit | 3-bit | 2-bit | 4-bit | 3-bit | 2-bit |
| WikiText2 | K-means++ | 5.64 | 6.25 | 17.84 | 6.59 | 8.31 | 46.31 | 5.37 | 5.92 | 22.32 |
| Uniformly Spaced (ours) | 5.66 | 6.20 | 17.53 | 6.59 | 7.88 | 41.78 | 5.40 | 5.88 | 19.06 |  |
| C4 | K-means++ | 7.13 | 7.80 | 19.55 | 10.17 | 12.53 | 39.86 | 7.88 | 8.48 | 19.81 |
| Uniformly Spaced (ours) | 7.14 | 7.72 | 18.75 | 10.20 | 12.16 | 36.00 | 7.91 | 8.42 | 17.85 |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can you provide any intuition for why the uniform grid point spacing method helps improve accuracy for k-means than k-means++ style grid points (i.e. a visualization of where the initial grid points end up being placed in both cases, versus the “target” grid points)?
- For the efficiency results in Table 9, is the K-means non-uniform quantization grid search parallelizable on the GPU, or is it being run on the CPU?
soundness: 4
strengths: - They present an algorithm to tune the quantization grid using the layerwise loss-aware objective
- They present both non-uniform and uniform methods (the non-uniform method is flexible and leverages clustering, whereas the non-uniform method performs a constrained search over potential scale factors)
- They provide accelerated GPU kernels to solve the objective for their affine quantization approach
- They provide detailed analysis of their method against prior work in both uniform and non-uniform quantization regimes
- They provide multiple ablations demonstrating how their method reduces the loss error, as well as the impacts of K-means initialization
summary: This paper presents a method for optimizing the quantization grid in a layerwise loss-aware manner to reduce low-bit weight quantization error. Unlike previous optimization-based approaches like GPTQ, they derive expressions for the optimal settings for the quantization grid rather than the optimal weight update to apply at each step. The approach is generalizable to both non-uniform and uniform quantization grids. This method is able to preferentially preserve more sensitive values, while maintaining computational efficiency by leveraging the layerwise optimization objective. This grid-tuning approach can also be integrated with previous weight update approaches like GPTQ to attain higher accuracy. They demonstrate the accuracy benefits of their approach, and also highlight the computational and memory efficiency of applying their quantization method relative to prior works.
weaknesses: - Although the method of determining weight sensitivity using the layerwise loss is distinct, the approach of performing K-Means clustering in equation (6) to derive non-uniform datatypes is the same as prior work (eg. the sensitivity-weighted clustering approach to derive non-uniform datatypes in SqueezeLLM)

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Whether it can be compared or ablated with other importance metrics such as the "Hessian diagonal" in SqueezeLLM, these works are similar in idea. Please see the weakness part.

2. There is a lack of literature survey on uniform and non-uniform quantization in related works.
soundness: 3
strengths: The idea is simple and effective, compatible with iterative loss-error-based quantization approaches such as GPTQ. The calibration process is relatively lightweight, allowing for the quantization of models up to 405B. The experimental design of this paper considers the latest model series and different model sizes, which is comprehensive.
summary: The paper proposes LeanQuant, an accurate, versatile, and scalable quantization approach. LeanQuant is built upon the iterative loss-error-based quantization framework and identify one of the biggest limitations of such methods: the quantization grid introduces high quantization errors due to the existence of outliers in the inverse Hessian diagonals.
weaknesses: LeanQuant is built on the core idea of using "inverse Hessian diagonals" metric to quantify the importance of weights. Similar ideas are also introduced in works such as SqueezeLLM [1], which exploits the "Hessian diagonal" metric directly derived from the quantization loss function to identify important weights. Compared to the metric in SqueezeLLM, I think the derivation of the metric in LeanQuant is relatively heuristic. Although LeanQuant beats SqueezeLLM in the final performance, it would be better to directly compare the different importance metrics and show the strengths of the proposed "inverse Hessian diagonals" metric since SqueezeLLM does not leverage GPTQ to minimize the quantization error.

[1] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization." arXiv preprint arXiv:2306.07629 (2023).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.	Could you provide a step-by-step explanation of how LeanQuant is adapted for non-uniform quantization by highlighting key differences from the affine quantization process?
2.	To ensure consistency between the promised and delivered comparisons, I suggest the authors either include a summary of the AWQ results in the main text or explicitly state why they were omitted.
3.	Why did you choose row-wise quantization for the comparison in Table 1? Additionally, do you have results for group-wise quantization? Providing more analysis with various quantization methods could address potential concerns about the fairness of the comparison and offer insights into LeanQuant's performance across different quantization schemes.
4.	I recommend including runtime comparisons with GPTQ, AWQ, and OmniQuant in Table 5 to provide a more comprehensive view of LeanQuant's efficiency relative to other existing methods
5.	I recommend including peak GPU memory usage for AWQ and GPTQ in Table 4, or explaining why this information was omitted. This would offer a more complete comparison of resource requirements across methods.
6.	How are the quantization grids calibrated for non-uniform quantization? Since each grid is independent for non-uniform quantization, I assume the calibration process must differ from the loss-error-aware affine grid described in Section 3.2.2.
7.	What would happen if group-wise quantization were applied to the affine quantization method presented in Table 1?
soundness: 3
strengths: 1.	LeanQuant calibrates quantization grids, and it is a novel approach in the post-training quantization of LLMs.
2.	The development of fused CUDA kernels efficiently supports the calibration process, so it makes the complex grid learning procedure more accessible for broad application due to its enhanced efficiency.
summary: This paper proposes Loss-error-aware Network Quantization (LeanQuant), a method designed to learn loss-error-aware quantization grids to reduce quantization error. LeanQuant determines optimal quantization parameters, such as scaling factor and zero-point, by iteratively narrowing the weight range in discrete steps. To accelerate grid learning, custom CUDA kernels were developed for efficient grid searching. Consequently, the paper demonstrates that LeanQuant can achieve superior linguistic capabilities with quantized LLMs compared to previous methods.
weaknesses: Overall, several details and evaluation results are missing, which makes it difficult to fully understand and be convinced of the effectiveness of LeanQuant.

1.	A detailed explanation of how LeanQuant can be extended to non-uniform quantization is missing.
2.	Although Section 4 (Experiments) states that the proposed method will be compared with AWQ, GPTQ, and OmniQuant, the evaluation results for AWQ are not presented in the main text (some results for AWQ are included in the appendix only).
3.	While group-wise quantization is widely used in LLM quantization, Table 1 appears to evaluate affine quantization only with row-wise quantization. (Note that AWQ is specialized for group-wise quantization, and layer-wise quantization with AWQ sometimes results in slightly worse linguistic capabilities compared to GPTQ.)
4.	The runtime for LeanQuant is presented with and without the custom kernel in Table 5, but no comparison is made with previous methods.
5.	Peak GPU memory usage for AWQ and GPTQ is not included in Table 4.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: The author’s motivation is well-founded, and the experiments are thorough. However, addressing the issues I raised in the weaknesses would greatly strengthen the paper.
soundness: 2
strengths: The strength of this paper lies in its innovative approach to addressing both accuracy and scalability in quantizing large language models, all while maintaining low hardware requirements. LeanQuant introduces a loss-error-aware quantization grid,  tackling the common issue in traditional quantization methods where accuracy is lost due to outliers in inverse Hessian diagonal elements. This approach adapts the quantization grid to preserve the precision of key weights, improving model quality, especially in low-bit quantization.

Additionally, LeanQuant’s multi-format compatibility enables it to integrate seamlessly with popular deep learning frameworks, eliminating the need for specialized inference kernels and greatly enhancing deployment flexibility.

On the scalability side, LeanQuant incorporates a fused GPU kernel design, accelerating the quantization process and allowing it to efficiently handle extremely large models (up to 405 billion parameters) with limited GPU resources. This efficiency showcases LeanQuant’s practicality under constrained computing resources, highlighting its potential for low-cost, high-efficiency AI applications. As a result, the paper presents a robust solution for large-scale model quantization, excelling in quantization quality, resource efficiency, and deployment adaptability.
summary: The paper introduces a new quantization method, LeanQuant, designed to improve the efficiency and accuracy of large language models (LLMs) without requiring specialized hardware adaptations.

Key Contributions:
1. LeanQuant innovates on post-training quantization (PTQ) by introducing a loss-error-aware quantization grid. This grid preserves model quality better than traditional min-max grids for outliers in the inverse Hessian diagonals, which otherwise lead to significant accuracy loss.
2. The method generalizes to affine and non-uniform quantization types, making it adaptable to existing frameworks and more compatible with standard inference kernels.
3. LeanQuant demonstrates scalability with successful quantization of models up to 405 billion parameters, achieving high accuracy using limited computational resources.
4. By designing a fused GPU kernel, LeanQuant achieves substantial speedups in grid learning, making quantization efficient and suitable for a range of model sizes.
weaknesses: 1.  Quantization isn't solely about reducing memory usage or data transfer—it’s fundamentally about reducing the computational workload, which in turn decreases the size of intermediate data and the need for data movement. If the focus is only on reducing model size without addressing the computational load, it could result in diminished acceleration benefits. This is because, even if memory requirements decrease, hardware may still need to perform the same level of computation, which undermines the intended efficiency.
2. As you pointed out, while LeanQuant can quantize models down to 3-bit or 2-bit, modern GPUs often support 4-bit or 8-bit multiplications. Thus, quantizing to lower bit widths (like 3 or 2 bits) might not yield computational efficiency if, during execution, the kernel still calls for a 4-bit multiplier. This forced conversion undermines the benefits of lower-bit quantization since it fails to leverage the intended reduction in computational and resource demands.
3. Without considering the storage format of hardware, 3-bit or 2-bit weights may still be stored on the GPU in higher bit-widths (e.g., 4-bit, 8-bit, or even 16-bit), resulting in excess storage and data transfer requirements. This dilutes the benefits of quantization. Designing quantization methods with hardware-oriented data formats is essential to ensure that low-bit quantized data can be efficiently processed and stored by hardware.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors emphasize that range settings are important for quantized network accuracy, and GPTQ uses min-max quantization. In fact, GPTQ employs grid search (instead of SGD) to find the best range setting that minimizes the MSE error (||W-Wq||) https://github.com/IST-DASLab/gptq/blob/main/quant.py#L78-L95 Can authors compare with GPTQ with this MSE range settings? Can the author also explain the conceptual difference between loss-error-aware affine grid and GPTQ + MSE grid search?
 
Regarding the non-uniform loss-error-aware grid, since the weight value changes during iterative GPTQ quantization, it might affect the k-means centroid as well. Did the authors consider this effect? Did the authors measure how far the weight - \delta deviates from the original weight?
 
The authors provide two methods: non-uniform quantization and affine quantization. Can the authors briefly explain in which cases each quantization scheme is recommended?
 
Can authors compare the results with the most recent LLM quantization works, such as QuIP[1], QuIP#[2], QuaRot[3] and SpinQuant[4]?

[1] QuIP: 2-Bit Quantization of Large Language Models With Guarantees

[2] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

[3] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

[4] SpinQuant: LLM Quantization with Learned Rotations
soundness: 2
strengths: Paper Strengths:

**Compared to existing iterative loss-error-based quantization frameworks**: LeanQuant addresses the critical limitation of min-max affine quantization grids, which fail to preserve model quality due to outliers in inverse Hessian diagonals, by proposing learning loss-error-aware grids.

**Compared to prior methods using non-adaptive min-max affine grids**: LeanQuant's approach produces quantized models that are more accurate and generalizes to a wider range of quantization types, including affine and non-uniform quantization.

**Compared to other methods in terms of scalability**: LeanQuant is scalable, achieving very accurate quantization of Llama-3.1 405B, one of the largest open-source LLMs to date, using two Quadro RTX 8000-48GB GPUs in 21 hours, which is an improvement over other methods.
summary: The paper proposes LeanQuant, a novel post-training quantization (PTQ) method that addresses the limitations of existing methods in reducing memory requirements and inference costs of large language models (LLMs). LeanQuant identifies a critical issue in prior PTQ methods, where the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals. To overcome this, LeanQuant learns loss-error-aware grids instead of using non-adaptive min-max affine grids, enabling accurate and versatile quantization. The method uses an iterative loss-error-based quantization framework, which involves computing the loss-error for each layer, learning a loss-error-aware grid for each layer, and quantizing the model using the learned grids. LeanQuant is evaluated on several large language models, including Llama-3.1 405B, one of the largest open-source LLMs, and achieves high accuracy, comparing favorably to recent competitive baselines. The method is also shown to be scalable, achieving accurate quantization of Llama-3.1 405B using two Quadro RTX 8000-48GB GPUs in 21 hours. LeanQuant's ability to learn loss-error-aware grids enables it to generalize to a wide range of quantization types, including affine and non-uniform quantization, making it a promising solution for reducing the memory requirements and inference costs of LLMs. Overall, LeanQuant provides a novel and effective approach to PTQ, enabling the efficient deployment of large language models in real-world applications.
weaknesses: Paper Weaknesses:

This paper is limited in novelty. The authors proposed two methods: (1) non-uniform loss-error-aware grid and (2) loss-error-aware affine grid. The first method, the non-uniform loss-error-aware grid, is GPTQ combined with k-means, which is similar to GPTVQ [1]. The loss-error-aware affine grid is equivalent to GPTQ with grid range search, which is implemented in the original GPTQ codebase. 
Additionally, they did not compare their work with the most recent state-of-the-art (SoTA) papers.

[1] GPTVQ: The Blessing of Dimensionality for LLM Quantization

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Please address all the weakness
soundness: 3
strengths: 1.	The paper has written “BACKGROUND” sections well, giving an amazing context of quantization. 
2.	Results validate the superiority of the proposed quantized method.
3.	The claim of scalability and accuracy is satisfactorily answered, 
4.	Algorithm 1 seems to help make sense of the procedure visually. 
5.	The ablation study is written well, which helps make the paper a lot more sense, especially Q1, and Q2.
summary: Authors propose “Loss-error-aware Network Quantization” a post-training quantization method for reducing memory requirements in addition to latency of LLMs. The method is proposed as an accurate, versatile, and scalable alternative to traditional methods which are either specific hardware and software platforms or require high resources making them unscalable for the larger models. 
The method is based on an iterative loss-error-based quantization framework, overcoming limitation in previous methods: the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals, instead, authors use loss-error-aware quantization grids, accounting for these outliers.
weaknesses: 1)	**The paper assumes the readers are well-versed in quantization**. Two elementary concepts of this work: *quantization grid* and affine / non-uniform grids, are not clearly defined inside the paper.   
a.	**What exactly is grid?** [Line 110] “representing the full-precision floating-point parameters with a limited set of grid points on the quantization grid” What does this mean? An example or visualization would be really helpful here.   
b.	**[Line 116] “grid points are evenly spaced over the dynamic range of a group of weights”** What does this mean?  
c.	A simpler discussion/visualization on uniform and non-uniform grids is requested.   
d.	Equation 6, is first solved to establish a quantization grid. It's confusing to know what the solution to equation 6 means if the reader is unfamiliar with the quantization grid.   
e.	[Line 117] Group of weights? What’s that? Quantization needs some form of grouping of weights, why? Works don’t seem to establish it.    


2)	**Page 6 needs a simpler rewrite if possible**.   
a.	The works seem to be indicating some problems with the grouping of weights using KNN, but it's confusing to understand the problem. 
b.	[Line 285] equation doesn’t have an equation number
c.	It's not clear what part is already established and what parts are being proposed by the work. Equation 6: where did this equation come from or is derived in some previous work?, similarly Equation 7, and the equation on [Line 285], are authors proposing these equations. There doesn’t seem to be any explanation/derivation for these, just of out blue written as a solution. If they are already established solutions, citing works will help. 
d.	“Efficient Fused GPU Kernel for Grid Learning” gives the 50x efficiency. Can this multi-threading technique be applied to existing work and make them more efficient as well? 

3)	The solution is based on the presence of outliers with abnormally high values in empirical distributions of inverse Hessian diagonals, (figure 1). This can be dataset-specific or model-specific. Show the presence of outliers in other datasets as well as models as well.

4)	**Contradictory results?** Table 10 shows p values as not so important for quantization. P > 0 is good enough for solving highly accurate quantization? Why does Table 10 doesn’t have p=1? Also, p >1 indicates the “extra” weightage of “outliers” in quantization, the first separation from existing work. 
[Line 259] : “The min-max affine grid employed by previous iterative loss-error-based quantization methods do not account for the existence of inverse-diagonal outliers, which may cause significant model quality degradation”
Does this mean, that just by the presence of outliers p=1, has more or less the same performance as P > 1, just accounting for outliers creates better grid formation? 
Results for other techniques (SqueezeLLM, GPTQ, OmniQuant) with p=1 or >1 (equation 6), outlier accounted grids can help validate the importance of this outlier driven gird.

5)	Minor issue: 
a.	Table 2 & 3: please mention of vanilla FP16 accuracy. 
b.	Table 2 & 3: Also non-uniform LeanQuant accuracy. 
c.	Table 4: GPTQ Memory values missing

Meta-review: This paper proposes LeanQuant, a post-training quantization (PTQ) method to mitigate the memory and inference cost challenges in LLMs. One of the main contributions of the paper is the introduction of loss-error quantization grids, which adapt to outliers in inverse Hessian diagonals. Outliers are known to significantly affect quantization quality. The paper claims to be generalizable to both affine and non-uniform quantization grids and shows compatibility with standard inference kernels (no hardware specific changes). With respect to results, the paper reports strong experimental results across different LLMs, models up to 405B params. LeanQuant outperforms SOTA methods like SqueezeLLM and GPTQ in both scalability and accuracy, esp. for low-bit quantization.

These are the main strengths of the paper: (a) the introduction of loss-error-aware quantization grids (works for both affine and non-uniform quantization) to address outlier sensitivity, improving quantization quality, esp. in low-bit representation regime. (b) showing results for up to 405B params on modest hardware setups (48GB GPUs), outperforming SOTA methods like SqueezeLLM. (c) using fused GPU kernels to accelerate grid learning, yielding speedups and enabling the quantization of large models.  (d) extensive experimentations on different models (Llama-2, Llama-3, and Mistral). 

There are multiple weaknesses about the paper as well: (a) there seem to be some similarities between LeanQuant and prior methods like GPTQ and SqueezeLLM. For instance, LeanQuant's non-uniform grid learning via k-means clustering shares conceptual overlap with SqueezeLLM. In addition, one reviewer also mentioned that the introduced affine quantization component is equivalent to GPTQ w/ grid range search. (b) some results are missing. esp. comparison with SOTA methods like QuaRot, QuIP and SpinQuant in the main text. (c) some reviewers talked about the fact that lower-bit quantization (3-bit and lower) might not be relevant with current hardware support (but I believe this is true for any low-bit quantization methods).

This is a borderline paper and most reviewers are positive about this work. One reviewer criticized the paper on writing that I don't believe should be the main reason for rejection. The other reviewer with most negative score also didn't engage in the discussion. After reading the rebuttal, I think the authors did a good job addressing the reviewer's concern. Overall, based on the reviewers' assessment and my reading, I recommend **Accept** for this paper.

Decision: Accept (Poster)