GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Authors: Pengxiang Zhao
Xiaoming Yuan
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Non-uniform Quantization
GPU-adaptive Algorithms
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) face significant deployment challenges due to their substantial resource requirements. While low-bit quantized weights can reduce memory usage and improve inference efficiency, current hardware lacks native support for mixed-precision General Matrix Multiplication (mpGEMM), resulting in inefficient dequantization-based implementations. Moreover, uniform quantization methods often fail to capture weight distributions adequately, leading to performance degradation. We propose GANQ (GPU-Adaptive Non-Uniform Quantization), a layer-wise post-training non-uniform quantization framework optimized for hardware-efficient lookup table-based mpGEMM. GANQ achieves superior quantization performance by utilizing a training-free, GPU-adaptive optimization algorithm to  efficiently reduce layer-wise quantization errors. Extensive experiments demonstrate GANQ's ability to reduce the perplexity gap from the FP16 baseline compared to state-of-the-art methods for both 3-bit and 4-bit quantization. Furthermore, when deployed on a single NVIDIA RTX 4090 GPU, GANQ's quantized models achieve up to 2.57$\times$ speedup over the baseline, advancing memory and inference efficiency in LLM deployment.

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Pengxiang Zhao'
Xiaoming Yuan'
Abstract
deployment for inference remains challenging due de- manding resource requirements. For example, the LLaMA- 3-70B (Dubey et al., 2024) model needs at least 140 GB of GPU memory in FP16, which exceeds current GPU capaci- ties. While larger LLMMs often yield better accuracy (Kaplan et al., 2020), these substantial resource demands hinder the practical deployment of LLMs, posing a barrier to their widespread adoption.
Large Language Models (LLMs) face significant deployment challenges due to their substantial resource requirements. While low-bit quantized weights can reduce memory usage and improve inference efficiency, current hardware lacks native support for mixed-precision General Matrix Mul- tiplication (mpGEMM), resulting in inefficient dequantization-basee implementations. More- over, uniform quantization methods often fail to capture weight distributions adequately, leading to performance degradation. We propose GANQ (GPU-Adaptive Non-Uniform Quantization), a layer-wise wise post-training non-uniform quantiza- tion framework optimized for hardware-efficient lookup table-based mpGEMM. GANQ achieves superior quantization performance by utilizing a training-free, GPU-adaptive optimization al- gorithm to efficiently reduce layer-wise quan- tization errors. Extensive experiments demon- strate GANQ's ability to reduce the perplexity gap from the FP16 baseline compared state-of- the-art methods for both 3-bit and 4-bit quanti- zation. Furthermore, when deployed on single NVIDIA RTX 4090 GPU, GANQ's quantized models achieve up to 2.57 speedup over the baseline, advancing memory and inference effi- ciency in L.M deployment.
Quantization is a promising solution to reduce inference costs for LLMs. For example, 4-bit weight quantization can reduce memory usage for model loading by nearly 75% compared to FP1 In general, quantization techniques are categorized into quantization-avee post-training quantization (PTQ). QAT integrates quanti- zation into the training process to achieve higher accuracy but is computationally expensive, often requiring extensive samples and significant GPU hours (Liu et al., 2024). This makes QAT impractical for large models. In contrast, PTQ is a cost-effective alternative that applies quantization after training, making the preferred choice for LLMs (Nagel et al., 2020; Yao et al., 2022; Frantar et al., 2022; Xiao et al., 2023; Dettmers et al., 2023; Kim et al., 2024; Lin al., 2024; Shao et al., 2024: Ma et al., 2024; Li et al., 2024b). Among PTQ methods, weight-only quantization, which uses low-precision weights while retaining high-precision activa- tions, has become a particularly attractive approach. By re- ducing memory traffic and alleviating memory-bound bottle- necks, weight-only quantization accelerates inference (Kim et al., 2024; Lin al., 2024). Additionally, compared to weight-activation quantization, it avoids significant accu- racy degradation by preserving the precision of activations, ensuring better model performance.
training (QAT) and
1. Introduction
Large language models (LLMs) have demonstrated impres- sive performance across various domains (Brown et al., 2020: Achiam et al., 2023; Touvron et al., 2023a;b; Dubey al., 2024; Gemini Team al., 2023; Chowdhery al., 2023; Zhang et al., 2023; Wang et al., 2023; Arefeen al.,
Despite its promise, weight-only quantization faces two key challenges. First, it shifts the core computation of LLM inference from standard General Matrix Multiplication (GEMM) to mixed mixed-precisien precision GEMM (mpGEMM), where low-precision weights (e.g.. INT4/3/2) are multiplied with high-precision activations (e.g.. FP16). Current hardware lacks native support for mpGEMM, necessitating dequanti- zation to upscale low-bit weights into supported formats (see the left part of Figure 1(a)). This additional step introduces inefficiencies, particularly in large-batch sce- narios, undermining the expected performance gains (Mo et al., 2024). Second, most existing methods rely on uni-
2024; Li et al., 2024a; Huang et al., 2024). However, their
'Department of Mathematics, The University of Hong Kong, Hong Kong SAR, China. Correspondence to: Xiaoming Yuan <xmyuan hku.hk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
W (INT4/3/2)
(NT44/3/)) T ((FP16)
0.8
Dequantization
Look up table
01
0.6
W (FP16)
X (FP16)
W (F166
(FP16)
Weight Values
0.4
0.2
0.45
-0.12
0.38
0.26
0.2
GEMM
GEMM
02
13
-0.6
-0.8
(FP16)
(FP16)
W (FP16)
k_pro)
Q_pro)
v_pno|
0_pro]
gate_pnn
up_pno)
down_pro
(a) Dequantization-hase and LUT--hased of mpGEMMM
(b) Violin plots of -LaMA-2-7B's first decoder layer weights.
Figure 1. (a) A comparison of two mpGEMM implementations: a dequantization based approach (left) versus a LUT--based method (right). (b) Violin plots showing the first decoder layer's weight distribution in the LLaMA-2-7B model, clearly illustrating their deviation from uniform distribution.
form quantization Q
0. 2N - 1] Z defined as 2(x) = clamp(L + 0,0, - 1), where [.] denotes 2
R
-
gramming problem. This model minimizes the discrepancy between the outputs of the quantized and original layers, thereby preserving accuracy. To efficiently address this com- plex model, GANQ utilizes its decomposable structure to divide the original optimization task into multiple indepen- dent one-dimensional subproblems, which can be processed in parallel using GPU acceleration to achieve substantial computational efficiency. Besides, GANQ employs an al- ternating direction optimization framework that capitalizes on the splittable structure of decision variables, effectively reducing quantization error.
rounding N is the target bit width, 8 is the scaling factor, and 2 is the zero-point (Frantar et al., 2022; Xiao et al., 2023; Dettmers et al., 2023; Lin et al., 2024; Shao et al., 2024; Ma al., 2024; Li al., 2024b). However, LLM weight distributions are often highly non-uniform (see Fig- ure 1(b)), making uniform quantization inadequate and re- sulting in suboptimal representations, particularly due to outliers. Techniques such as introducing learnable scale and zero-point parameters (Shao et al., 2024), applying affine transformations to preprocess weights (Ma al., 2024), or splitting weights into various components and quantiz- ing those that are easier to process (Dettmers et al., 2023; Li et al., 2024b), have been proposed to mitigate these is- sues. While these methods improve accuracy, they primarily address challenges within the uniform quantization frame- work rather than fundamentally enhancing the quantization method itself. Furthermore, they often increase computa- tional complexity during inference due to the extra opera- tions they require.
In addition, although GANQ designed as a base quan- tization method, it is fully compatible with current tech- niques for handling outliers, such as splitting weights into sparse components (to address outliers) and quantized com- ponents (Dettmers et al., 2023; Kim et al., 2024), thereby enabling further performance enhancements.
We evaluate GANQ extensively across various model fam- ilies and sizes on language modeling tasks, The results show that GANQ consistently outperforms previous meth- ods in quantization performance. Moreover, GANQ is highly resource-efficient and easy to implement. For in- stance, GANQ processes the LLaMA-2-7B model on a sin- gle NVIDIA RTX 4090 GPU in approximately one hour, using only 128 samples, each containing 2,048 tokens. Fur- thermore, our deployed models on the NVIDIA RTX 4090 GPU achieve up to 2.57x speedups over the FP16 base- line by leveraging LUT-based inference kernels (Kim et al., 2024). These results highlight the effectiveness of GANQ in both quantization quality and inference efficiency.
To address these issues, we propose GANQ (GPU-Adaptive Non-Uniform Quantization), layer-wise post-training non- uniform quantization framework optimized for lookup table (LUT))based mpGEMM. In LUT-based mpGEMM (see the right part of Figure (a)),, complex computations are replaced with simple table lookups, supported by several GPU kernels (Kim et al., 2024; Mo et al., 2024; Guo et al., 2024). The primary challenge then becomes how to determine effective low-bit representations for the LUTs. Existing non-uniform quantization methods often rely on heuristic-based approaches, such as manually designed map- pings (e-g., power-exponent functions (Yvinec et al., 2023)) or clustering- based methods with heuristic distance met- rics (Han et al., 2015; Xu et al., 2018; Kim et al., 2024). While these methods may achieve good results in specific cases, their heuristic nature limits generalization and theoret- ical grounding. In contrast, GANQ introduces principled optimization model for laser-wise LUT-based non-uniform quantization, formulated as a mixed-integer quadratic pro-
2. Related Work
Quantization for LLMs. Quantization reduces the bit- precision of neural networks, resulting in smaller models and faster inference. It has become a key direction for compressing LLMs given their growing size and inference costs. Current quantization methods for LLMS are broadly categorized into QAT (Liu et al., 2024) and PTQ (Nagel
2

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
al., 2020; Yao et al., 2022; Frantar al., 2022; Xiao et al.,
and Sq require both mpGEMM and sparse matrix multiplication, whereas SVDQuuntt adds extra low-rank computation branch.
2023; Dettmers al., 2023; Kim al., 2024; Lin et al.,
2024; Shao et al., 2024; Ma et al., 2024; Li et al., 2024b). QAT integrates quantization into the training process, pre- serving high performance but incurring prohibitive training costs, making it impractical for LLMs. In contrast, PTQ applies quantization to pretrained models, requiring only a small subset of data and modest computational resources, making it partieularly appealing for LLMs.. PTQ methods can be further classified into wight-only quantization and weight-activation quantization.
In this work, we propose direct solution by introducing a non-uniform quantization framework that adapts to the distribution of LLM weights. Furthermore, our method is compatible with these outlier-hnndling techniques, enabling further performance enhancements when combined.
LUT-based Inference Kernel. Low-bit quantized LLMs depend on mpGEMM for efficient inference. This oper- ation, which involves multiplying low-precision weights with higher-precisiie activations, presents a critical com- putational challenge. Current hardware lacks native sup- port for mpGEMM, compelling existing implementations to adopt dequantization-basel workarounds (Frantar et al., 2022; Lin et al., 2024; Shao et al., 2024; Ma et al., 2024; Li et al., 2024b). LUT-based methods offer a compelling al- ternative by eliminating dequantization overhead. Through efficient substitution of arithmetic operations with table lookups (Kim et al., 2024; Mo et al., 2024; Guo et al., 2024), these approaches demonstrate particular suitability for mpGEMM acceleration.
Weight-only quantization focuses on compressing model weights into low-bit formats. For example, GPTQ (Fran- tar et al., 2022) utilizes the optimal brain surgeon frame- work (Hassibi & Stork, 1992) for quantization and recon- struction. OmniQuant (Shao al., 2024) introduces learn- able parameters to determine quantization factors (e.g., scale and zero-point), while Affineuuant (Ma et al., 2024) extends this idea by incorporating a learnable matrix to preprocess weights before quantization. Weight-activation quantization compresses both weights and activations, often addressing their quantization jointly. For example, SmoothQuant (Xiao al., 2023) shifts quantization difficulty from activations to weights using manually designed scaling factors. Similarly, SVDQuant (Li et al., 2024b) applies this approach while further decomposing weights into low-rank and quantized components.
Unlike previous work focused on kernel-lvee optimizations, our research instead targets fundamental improvements in LUT-based quantization accuracy and compatibility with existing kernels.
While weight-activation quantization can offer broader com- pression, studies (Kim et al., 2024: Lin al., 2024) have shown that LLM inference, especially during generation, is heavily memory-bound, with weight access dominating activation access by orders of magnitude. Consequently, weight-only quantization is more effective for on-device de- ployment of LLMss In this work, we focus on weight-only PTQ for its efficiency and suitability for LLMs.
Non-Uniform Quantization. The non-uniform distribu- tion of weights in LLMs highlights the importance of non- uniform quantization. However, existing non-uniform quan- tization methods often rely on heuristic-based approaches, limiting their generalization and theoretical grounding. NU- PES (Yvinec et al., 2023) replaces uniform quantization with power-exponent functions and employs gradient-based optimization to learn the exponent parameter. Other meth- ods focus on identifying shared weights. thereby forming codebook, which is suitable for LUT-based mpGEMM. For example, Han et al. (2015) apply k-means cluster- ing to minimize the Fuclidean distance between weights and centroids in convolutional neural networks (CNNs), while Xu et al. (2018) extend this approach by using a loss-based metric for k-means clustering in CNNs. For LLMs, SqueezeLLM (Kim et al., 2024) adapts this idea by leveraging sensitivity- k-means clustering, where the sensitivity metric measures the extent to which the model is perturbed after quantization. To mitigate the computational expense of this calculation, SqueezeLLM approximates the required Hessian matrix using the diagonal elements of the Fisher information matrix (Fisher, 1925).
Outlier Mitigation. Due to the widely used uniform quan- tization mapping and the inherent non-uniform distribu- tion of LLM weights, key challenge is the presence of outliers. These outliers unnecessarily expand the quan- tization range (see Figure 1(b)), comprising quantization performance. Recent methods have been proposed to ad- dress this issue. For example, SpQR (Dettmers et al., 2023) and SqueezeLLM (Kim et al., 2024) retain outliers in sparse matrices while applying quantization to the re- maining weights to mitigate their impact on overall perfor- mance. AWQ (Lin et al., 2024) independently quantizes the channel-wise salient weights to improve performance, and SVDQuant (Li et al., 2024b), as mentioned, decomposes weights into low-rank and quantized components. While these methods effectively handle outliers and enhance quan- tization performance, they often introduce additional com- putational overhead during inference. For instance, SpQR
In contrast, we propose a principled optimization model for layer-wise LUT--baeed non-uniform quantization for LLMs, along with an efficient GPU-adaptive algorithms to solve it.
3

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
3. Methodology
where W. E F Iix= xm is the i-th row of W. T3 E R 12x22 is the i-th row of T, S, E {0, 1)2a xn is column-wise one- hot encoding matrix indicating the mapping of elements from Tú, and 1 denotes an all-one vector.
3.1. Optimization Model for Non-uniform Quantization
Consider a linear layer with weight matrix W E ""*x" and input activation X E R=x,, where n represents the input hidden dimension, m the output dimension, and p = b accounts for batched processing of b sequences each of length As shown in the right part of Figure ((a), LUT- based quantization aims to compress W by representing its elements using a codebook. Specifically, the elements of the i-th channel in the quantized weight matrix W are selected from the codebook T, = fts,o. :i.1....,,..7. -122~11. J. where N is the bit-width of the quantization (e.g., 3 or 4 bits). Thus, each element Wij satisfies W., E ,,
The mixed-integer structure of S; introduces significant combinatorial complexity, and the bilinear interaction be- tween S; and T, in the objective further compounds the computational challenge, rendering the problem inherently non-convex and non-smooth. These factors pose serious difficulties for off-the-shelff solvers (Gurobi Optimization, 2025; IBM, 2025). especially in large-scale settings with high-dimensional weight matrices and input activations. In response, we develop a specialized, GPU-adaptive approach tailored to navigate this complex search space while scaling to practical problem sizes.
In practice, LUT-based quantization stores two components: a low-bit query matrix Q E {0. 1.... 2 - 13 1]mx", which specifies the indices of values the codebook, and the codebook itself, T E _mxx²*, which contains the quan- tized values for each channel. For example, if Qiy = 0, then Waj = tu. Compared to the widely used basic per- channel uniform quantization (Frantar et al., 2022; Xiao et al., 2023), which requires two parameters per channel (i.e., scale and zero-point), this mechanism demands slightly more storage. However, as min(m, n >> 2 in practice, the additional storage overhead is negligible. As shown in Table 1, for typical model sizes, the storage usage of LUT- based quantization remains comparable to the basic uniform quantization, differing by less than 0.2%. Moreover, some uniform quantization methods, such as OmniQuant (Shao et al., 2024) and A AffineQuant (Ma et al., 2024), also require extra parameters.
3.2. GPU-Adaptive Non-Uniform Quantization Method
To efficiently solve the model in (2) for LUT-based non- uniform quantization, we employ an alternating direction optimization framework. This framework iteratively up- dates S, and T, by decomposing the objective into two subproblems. Each subproblem optimizes one decision vari- able while keeping the other fixed. The iterative scheme is outlined as follows:
S argmin S: { [. T 1 -[
(3)
T argmin (1 | X - T.S X 1
(4)
T.
The T,-subproblem in (4) is an unconstrained quadratic program that admits a closed-form solution. Specifically, consider the objective function of T.-ssbprooleem:
To enable effective LUT-bbasd non-uniform quantization, we formulate an optimization model aimed at minimizing the layer-wise output error::
S(T)) | - -T,
(5)
min WX - W s.t. Wij = -T.q., Vi,J, (1) Q.T
To minimize this function, we apply the first-order optimal- ity condition. Taking the gradient of fT;)) with respect to T, and setting it to zero yields:
where| || || denotes the Frobenius norm, and Q and T are
the decision variables.
VT(T,) = 2(W,X x - " (S = 0. (6)
Note that the quantized output for each row (WX),, s; de- pends only on its corresponding codebook Ti. and query vector Q Consequently, the model in (1) is inherently de- composable across the rows of W. Leveraging this property, the problem can be reformulated into m independent sub- problems, which are highly parallelizable and particularly suitable for GPU acceleration. Specifically, this paralleliza- tion is achieved by expressing computations in matrix form, which enables efficient matrix-vector and element-wise op- erations across rows. Furthermore, each subproblem can be expressed as a mixed-integer quadratic programming problem:
Solving this matrix equation gives the closed-form update for T;:
( ((.) (S)+)) (7)
where (-)t denotes the Moore-PPnnose inverse. Notably, the matrix (s,)*+1 xx (S) has dimensions 2 x 2.V which is relatively small in practice (e.g., 16 x 16 under 4-bit quantization), ensuring that the computation remains efficient. Moreover, computing (7) involves only matrix- vector multiplications, making it highly efficient for GPU acceleration. Since the solutions to all T--subproblems share the same formulation, they can be combined into a
min W.X- T,S,X s.t. 1S, =1,, Vi, S.
4

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Table 1. SSorrage requirements for full-precision (FP16), basic per-channel uniform quantization (4-hit), and per-channel L.UT-based non-
uniform quantization (4- bit) for weight matrix W E Rmnx. Percentages indicate storage usage relative to full-precision representation.
CONFIGURATION
FULL (FP16)
BASIC UNIFORM (4-BIT)
LUT-BASED (4-BIT)
Theory
2mn
0.5mm + 4m
0.5mn + 32m
m n 2048 (c.g., W, in OPT-1.3B)
100.00%
25.10%
25.78%
1096 (e.g.. W, in LLaMA-2-7B)
100.00%
25.05%
25.39%
m -8922 (e.g., W, in LLaMA-2-70B)
100.00%
25.02%
25.20%
single batch computation by stacking all W; and T, vectors row-wise and organizing S; matrices into a tensor. Then,
By combining (9) and (10), we have:
matrix operations can be used to efficiently compute the batch. This approach leverages modern GPUs' parallel pro- cessing capabilities, significantly reducing computational overhead and improving overall efficiency.
(W, ( (W; - T.)
(11)
= =(W SLL )) S,
(12)
= -
(13)
The primary challenge lies in the S,-subproblem (3), which is discrete, non-convex, and non-smooth combinatorial optimization problem. In the case of 4-bit quantization, each element S, can assume one of 16 possible values. A brute- force search over all combinations would require ((16") operations, rendering it computationally prohibitive. There- fore, developing efficient solution techniques is essential for practical applications.
Leverage the structure of L, we minimize (13) using back- substitution approach to efficiently derive sub-optimal solution to (3). Specifically, there is
W. L - S.I.
(14)
(W.L), - (T...L),)
(15)
1=
To address the S,-subproblem, we propose an efficient method that leverages the problem's 's inherent structure. The objective in (3) can be expanded as:
2
Wi,u T (St) Lugs
(16)
W W.X S 2
(8)
Following (16), we can solve for S: from the last column
(j = ni 1) to the first column (j = 0), minimizing each of the squared terms respectively. The (n 1)-th column of L has only one nonzero entry in rows 2 1, namely
= WW - TXX ) - -W,-T T.
(9)
Then, consider the Cholesky decomposition of XXT:
- Therefore, for j 1, the residual involves single term:
xx
LL
(10)
WW (St) Ln-1.n-1-
(17)
where L is a lower triangle matrix, meaning all its entries above the diagonal are zero.
Minimizing with respect to (S:) gives that it should select element from Té that satisfies
Remark 3.1. If XX" is not positive definite, which is rare but can occur cases like the fc2 layer of OPT models, we can add AI (A > 0) to guarantee positive definiteness before Cholesky decomposition. Specifically, for any non- zero vector adjusted matrix satisfies v (XX +AIvy= v| + 22| >0 O. This preconditioning is standard technique in numerical linear algebra.
idx argmin W W W
T
(18)
Then, we set (S.) = and all other elements in this column to O.
Once (Si).n-1 determined, the process moves to the (n 2)-th column. The residual becomes
Remark 3.1 describes basic strategy to ensure positive definiteness by augmenting the matrix X.X with scaled identity matrix. However, selecting an appropriate À man- ually can be cumbersome and suboptimal. In practice, we adopt an adaptive preconditioning approach that enforces di- agonal dominance, inherently ensuring positive definiteness without requiring manual hyperparameter tuning. Details are provided in Appendix A.
W W..-2 T (S.).
L, Ln
(19)
W W T (S.) (S.)...))
L 1nn-1
(20)
where (20) is constant value given (S:), ..).,..1 In the follow- ing steps, we refer W Win-- (S,) as Tn-1- Then,
5

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
back-substitttion
GPU-adaptive row-wise parallel computation
W
TO
So
0
0
Previous Residual:
0
0
Fu Wt T (S.
?
?
0
0
0
2. Determine S
Eu-jas ricler T.
W
T
S
idx argming W.y +
4/
(Siiax = 1, (S,)-idx)
Wm-1
m-1
Sm-1
Figure An illustration of the back-substitution framework for determining S., leveraging the lower triangular structure of
we solve for (S:).,n-2 by minimizing the square (19)
Algorithm GANQ: GPU-Adaptive Layer-Wins LUT-
(20):
Based Non-Unifomn Quantization
Input: WeRmxn XER*XP, initial codebook T TRR22
idx= argmin
W 2 + Ta- 1 T (21)
number of itcrations K
Output: Updated T and query matrix Q @* -1] -1] -1)mX" 0 0
2
Initialize S
tensor format
and we set (S = 1 and the rest of (S O.
Compute
X.X
Compute L = Cholesky(H)
Cholesky decomposition
This back-substitution process continues for j = n
for k <- 0 to K - do
0. At each step the element of (S,)ias.j set to 1 is determined as
Initialize
0
# previous residual vector
for + n to do
idx argmin, W
T
row-wise
Q Q++1 idx
idx argmin Waj + Lij u=j+1
Σ ru Lu..
T (22)
Update S using idx r =
# one-hot encoding
(W n. T ::.g Lygg-
update residual
end for
where Pu W... i.u T (S:)..u
WH( - ((( ((
# batch update
end for
Figure 2 illustrates the back-substitution framework for ef- ficiently determining S.. Since the solution processes for i = 0, 1, m - 1 are independent, similar to the batch solving of -us-ubbrosss described earlier, we can stack all W, and T, vectors row-wise and organize the S; matrices into tensor. This allows the back-substitution process to be performed for the entire problem using matrix operations, leveraging modern GPUs' parallel processing capabilities to enhance overall efficiency.
Return T QK
In our framework, the weight matrix W can similarly be decomposed into sparse component W sparses containing ex- tracted outliers, and a dense component W Wemse, processed through GANQ. Appendix details our outlier extraction method, which identifies outliers using a small ratio thresh- old (e.g., 0.5% of total parameters) while preserving the remaining weights for quantization. This decomposition reduces quantization range, thereby enhancing the quantiza- tion performance.
Finally, the full pseudocode of GANQ for layer-wise LUT- based non-uniform quantization presented Algorithm
3.3. Compatibility with Outdier-Hndllng Techniques
4. Experiments
GANQ provides a foundational framework for LUT-based non-uniform quantization and is inherently compatible with existing techniques for handling outliers in weight ma- trices. Among these techniques, a widely adopted ap- proach involves splitting the weight matrix into sparse matrix for outliers and a quantized matrix for the remain- ing weights For example, SpQR (Dettmers et al., 2023) and SqueezeLLM (Kim et al., 2024) extract outliers into a separate sparse matrix to mitigate their impact on the quantization process.
4.1. Settings
Quantization. We evaluate GANQ on weight-only non- uniform quantization. The default configuration employs INT4/3 per-channel weight quantization.
Models. We comprehensively evaluate GANQ on a range of models, including OPT (Zhang et al., 2022), LLaMA (Tou- vron et al., 2023a), LLaMA-2 (Touvron et al., 2023b), I.L.M-33 (Meta AI, 2024a), and LLaMA-3.2 (Meta AI,
6

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
2024b) model families. Specifically, we assess its per- formance across OPT-125M, OPT-350M, OPT-1.3B, OPT- 2.7B, OPT-6.7B, LLaMA-7B, LLaMA-2-7B, LLaMA-3-8B, LLaMA-3.2-IB. LLaMA-3.2-3B, LLaMA-3.2-1B I--ntrucct and L...MA--..-3--Issttte models.
perplexity results on the C4 and PTB datasets as well as results of quantized LLaMA-3.2 models in Appendix C). As shown, GANQ consistently outperforms baseline methods such as RTN, GPTQ, and OmniQuant across all configura- tions. For 4-bit quantization, GANQ achieves the lowest perplexity across both OPT and LLaMA models, with no- table improvements. Remarkably, on OPT-2.7B, GANQ's perplexity (12.33) even outperforms the full full-precision FP16 model (12.47). GANQ also demonstrates strong perfor- mance with 3-bit quantization, maintaining competitive per- plexity reductions across model sizes. For example, on OPT- 6. 7B, GANQ's perplexity is 11.39, compared to 15.11 for GPTQ and 13.47 for OmniQuant. These results underscore GANQ's s effectiveness in both 4-bit and 3-bit quantization, achieving substantial perplexity reductions across various model scales. The "_" in Table 2 indicates that OmniQuant cannot quantize LLaMA-3-8B on a single NVIDIA RTX 4090 GPU due to memory constraints or the unavailability of the pre-quantized model.
Evaluation. Following prior work (Frantar et al., 2022: Shao et al., 2024; Ma et al.. 2024; Kim et al., 2024), we evaluate the quantized models by reporting perplexity on language datasets, specifically using the WikiText-2 (Merity et al., 2017), C4 (Raffel et al., 2020), and PTB (Marcus et al., 1994) datasets. Consistent with established prac- tice, we use a sequence length of 2,048 across all models. Additionally, we assess accuracy on zero-shot tasks, in- cluding ARC Easy, ARC Challenge (Clark et al., 2018), WinoGrande (Sakaguchi al., 2021), BoolQ (Clark al., 2019), RTE (Wang et al., 2018), 8), HellaSwag (Zellers al., 2019), and GSM8K (Cobbe et al., 2021), facilitated by the LM Harness library (Gao et al., 2021). Long-context ca- pabilities are evaluated using LongBench (Bai et al., 2024) under its standard protocol.
The results in Table 3 show the zero-shot performance of the quantized ILLMMA-2-1 -- model across six tasks under 4-bit and 3-bit quantization. GANQ outperforms baseline meth- ods such as RTN, GPTQ, and OmniQuant in both bit-width configurations. With 4-bit quantization, GANQ achieves an average accuracy of 64.23%, which is comparable to the full- precision model (64.47%) For 3-bit quantization, GANQ maintains strong performance with an average accuracy of 62.22%, significantly surpassing other baseline methods. These results demonstrate GANQ's 's ability to preserve high task performance, even under aggressive quantization.
Baselines. For basic weight-only quantization, we com- pare GANQ with standard round-to-nearest uniform quan- tization (RTN), GPTQ (Frantar et al., 2022), and Omni- Quant (Shao et al., 2024). For weight-only quantization with outlier handling, we compare with GPTQ, OmniQuant, and AWQ (Lin et al., 2024), each using a group size of 128, as well as SqueezeLLM (Kim et al., 2024).
Setup. We implement GANQ using the PyTorch (Paszke et al., 2019) and utilize the HuggingFace Transformers li- brary (Wolf, 2019) for model and dataset management. Our implementation is publicly available'. All experiments are conducted on a single NVIDIA RTX 4090 GPU. For calibra- tion data, we follow the methodology outlined in previous works (Frantar et al., 2022; Shao et al., 2024; Kim et al., 2024). Specifically, we use 32 sequences for OPT models and 128 sequences for LLaMA models. Each sequence con- sists of 2,048 tokens, sampled from the first shard of the C4 dataset.
Table 4 presents performance comparison of 4-bit quan- tized LLaMA-3.2 IB IB-Instruct and 3B- 3B-Instruct models on two tasks: LongBench, which evaluates long-context capa- bilities, and GSM8K, which assesses reasoning-intent intensive tasks. Both tasks are evaluated in zero-shot settings with- out tfhthugt prompting. GANQ consistently out- performs baseline methods on both tasks, highlighting its robustness. In contrast, RTN exhibits extreme sensitiv- ity to model scale, collapsing on 3..MM--..-1B-IInsiii Similarly, GPTQ achieves near-zero GSMSK accuracy on LLaMA-3. 3.-.--.---nnttrrre sults reported Table 10, and encounters LongBench exe- cution errors (i.e., skip special tokens failures) de- spite identical experimental setups. Compared to RTN and GPTQ, OmniQuant leverages learnable quantization param- eters, leading to relatively stable performance. However, it still falls short of GANQ's results. These findings un- derscore GANQ's ' ability to deliver superior performance across diverse tasks.
Latency Profiling, Using Torch CUDA profiler, we mea- sure single-sequens (batch size 1) generation of 1024 to- kens on single NVIDIA RTX 4090 GPU, reporting CUDA time and peak memory usage with LUT-based inference kernels in (Kim al., 2024).
consistent with the perplexity re-
4.2. Main Results
Weight-only Quantization. The results in Table 2 present the WikiText-2 perplexity of quantized OPT, LLaMA, LLaMA-2, and LLaMA-3 models under 4-bit and 3-bit configurations across different model sizes (with additional
Weight-only Quantization with Outlier Handling. To mit- igate outlier impact, methods like RTN, GPTQ, AWQ, and OmniQuant divide per-channel distributions into smaller
'The code is available https:
GANG
7

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Table WikiText-2 perplexity (1) of quantized models under 4-bit 3-bit. GANQ outperforms state-of thee------e--tt methods.
OPT
LLaMA
Method
Bit-width
125M
350M
1.3B
2.7B
6.7B
7B
2-7B
3-8B
Full
16
27.66
22.00
14.63
12.47
10.86
5.68
5.47
6.13
RTN
37.11
25.94
48.18
16.73
12.14
6.29
6.12
8.53
GPTQ
31.08
23.99
15.60
12.88
11.45
6.95
6.08
2.4e2
OmniQuant
30.98
23.34
15.25
12.84
11.25
5.92
5.88
GANQ
28.58
23.04
14.94
12.33
10.70
5.83
5.65
6.61
RTN
1.3e3
64.56
1.3e4
1,3e4
5.8e3
25.54
5,4e2
2.2e3
GPTQ
52.48
34.47
21.60
16.95
15.11
16.65
9,46
1.4e2
OmniQuant
42.43
29.64
18.22
19.47
13.47
6.79
7.08
GANQ
35.98
29.42
17.05
14.10
6.33
6.25
8.83
Table 3. Accuracies (%, 1) of the quantized LLaMA-2-7B model on 6 zero-shot tasks under 4-bit and 3-bit quantization.
Method
Bit-width
HellaSwag
BoolQ
RTE
WinoGrande
Arc-e
Arc-c
Mean
Full
16
57.12
77.68
63.18
69.06
76.35
43.43
64.47
RTN
55,59
73.61
59.57
68.43
74.03
41.30
62.09
GPTQ
4
55,66
74.43
57.76
57.72
75.25
42.32
60,52
OmniQuant
55.66
75.69
63.90
68.19
74.33
39.85
62.94
GANQ
56.10
77.31
65.70
68.75
74.96
42.58
64.23
RTN
30.93
42.54
52.71
52.17
34.76
21.33
39.07
GPTQ
3
47.55
67.03
55.60
59.75
64.60
33.11
54.61
OmniQuant
3
52.58
72.11
57.40
64.72
68.73
36.43
58.66
GANQ
53.85
75.02
62.82
67.48
73.36
40.78
62.22
As shown in Table 5, GANQ" (indicating GANQ integrated with outlier handling) outperforms other baselines. Further- more, when retaining only 0.5% of outliers for LLaMA- 7B and LLaMA-2-7B, we observe the following results: LLaMA-7B (5.78 for 4-bit, 6.20 for 3-bit), LLaMA-2-7B (5.60 for 4-bit, 6.10 for 3-bit), which still outperform all
Table Performance comparison 4-bit quantized LLaMA-3.2 IB- Instuctrtt and 3B-Instruct models on LongBench (1) and GSMSK
(+) datasets.
IB Instructtt
I3struct
Method
LongBench GSM8K (%)
LongBench GSM8K (%)
other methods except for SqueezeLLM.
FP16
11.5
32,90
12.7
64.97
RTN
0.2
4.17
12.1
37.68
4.3. Profiling
GPTQ
11.14
OmniQuant
8.9
16.53
11.5
54.74
Table reports CUDA time, speedup, and peak memory us- age for GANQ and GANQ", measured on a single NVIDIA RTX 4090 GPU while generating 1,024 tokens with a batch size of on OPT-6.7B and LLaMA-7B. GANQ achieves up to 2.57x speedup over the FP16 baseline, with peak mem- ory usage reduced to 4.10 GB for OPT-6.7B and 3.30 GB for LLaMA-7B under 3-bit quantization. While GANQ" provides improved quantization accuracy through outlier handling, it incurs slightly higher inference latency due to additional sparse matrix operations, Therefore, the choice between GANQ and GANQ should be guided by the de- sired trade-off between accuracy and runtime efficiency.
GANQ
11.7
27.75
12.5
60.50
blocks (typically of size 128). SqueezeLLM retains small percentage of outliers (e.g., 0.5%) and a fixed number of full rows (default: 10). GANQ can integrate seamlessly with SqueezeL.LM's outlier handling mechanism. We eval- uate this integration through experiments. Due to memory constraints, OmniQuant cannot quantize LLaMA-3-8B, and SqueezeLLM is limited to models up to 2.7B. We use re- sults directly from their paper for these cases if available. Additionally, Squeezel.LM's current code does not support OPT-350M. For GANQ, we retain 0.5% outliers for all OPT models and LLaMA-3-AR Additionally, we retain 10 full rows for LLaMA-7B and LLaMA-2-7B to ensure a fair comparison with Squeezel..M.
The above results demonstrate that GANQ is fully compati- ble with existing LUT-based inference kernels (Kim et al., 2024). Moreover, GANQ stands to benefit from ongoing engineering advancements this class of kernels, such as
8

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Table WikiText-2 perplexity (1) of quantized models under 4-bit 3-bit. GANQ outperforms state-of the methods.
OPT
LLaMA
Method
Bit-width
125M
350M
1.3B
2.7B
6.7B
7B
2-7B
3-8B
Full
16
27.66
22.00
14.63
12.47
10.86
5.68
5.47
6.13
RTN (g128)
4
30.49
24.51
15,29
12.80
11.15
5.96
5.72
6.73
GPTQ (g128)
29.78
23.40
14.91
12.50
10.99
6.40
5.65
8.97
AWQ (g128)
4
29.09
1.3c4
14,93
12.70
10.96
5.78
5.60
6,53
OmniQuant (g128)
29.57
22.85
14.88
12.66
11.04
5.79
5.62
SqueezeLLM
28.51
14.83
12.60
10.92
5.77
5,57
GANQ
4
28.16
22.84
14.53
12.19
10.69
5.76
5,57
6,50
RTN (g128)
3
50.61
36.33
1.2c2
2.6c2
22.87
7,01
6.66
12.07
GPTQ (g128)
37.93
28.21
16.33
13.57
11.30
8.68
6,43
19.87
AWQ (g 128)
35.75
1.7c4
16.31
13.56
11.39
6.35
6,24
8.22
OmniQuant (g128)
3
35.61
27.65
16.16
13.28
11.23
6.30
6.23
SqueezeLLM
32.59
15.76
13.43
11.31
6.13
5.96
GANQ
3
32.35
26.84
15.52
13.11
11.13
6.08
5.93
7.46
Table CUDA time (s), speedup (f), and peak memory (GB) (4) for GANQ and GANQ" using the LUTT-based inference kernel from (Kim al., 2024) OPT-6.7B and LLaMA-7B models.
OPT-6.7B
LLaMA-7B
Method
Bit-width
CUDA time
Specdup (t)
Peak Memory (4)
CUDA time
Speedup (t)
Peak Memory (L)
Full
16
12.91
17.86
1.00
13.06
GANQ
4
2.24
4.88
8.46
2.11
4.14
GANQ"
4
10.39
1.61
5.13
9.82
1.82
4.12
GANQ
3
6.51
2.57
4.10
7.48
2.39
3.30
GANQ'
3
10.73
1.56
4.39
8.85
2.02
3.32
those proposed in (Guo et al., 2024; Mo et al., 2024), which can further enhance implementation efficiency and overall inference performance.
5. Conclusion
In this work, we introduce GANQ. GPU-adaptive non- uniform quantization framework for efficient deployment and inference of LLMs. GANQ introduces principled opti- mization model for layer-wise LUT-based quantization, for- mulated as mixed-integer quadratic programming problem, and solves efficiently using a GPU-adaptive alternating direction optimization algorithm. Extensive experiments GANQ reduces perplexity compared to state-of-the- oft-te-arttttt methods in 3-bit and 4-bit quantization set- tings, while preserving high accuracy. Additionally, GANQ achieves up to 2.57 speedup over FP16 baselines sin- gle NVIDIA RTX 4090 GPU, showcasing its substantial im- provements in both memory usage and inference efficiency. GANQ is highly resource-efficient easy to implement, and compatible with existing techniques for handling outliers, making it highly flexible solution for large-scale LLM de- ployment. These results highlight GANQ's potential to en- able practical and efficient deployment of high- nigh-performance LLMs across wide range of hardware configurations.
4.4. Quantization Cost
Among the evaluated methods, RTN, GPTQ, and AWQ are the most efficient in GPU memory usage and quan- tization time, due to their layer-wise heuristic approach. However, they trade off model performance. In contrast, OmniQuant and SqueezeLLLM require gradient information, leading to higher memory demands. OmniQuant can quan- tize 7B models on single NVIDIA RTX 4090 GPU but fails for 8B models and takes over 3 hours for 7B quan- tization. SqueezeLLM, which requires global gradients, can only quantize models up to 2.7B. GANQ leverages GPU-adaptive, parallel row-wise computation to quantize 7B models in approximately hour for K = 10. Consider- ing both model performance and quantization cost, GANQ an effective solution.
demonstrate

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Acknowledgments
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21100 10. 14168, 2021.
The work of X. Yuan was supported by the GRF RGC (No. 17309824).
Impact Statement
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T. and Alistarh, D. Spqr: A sparse-quantize representation for near near-lossless Ilm weight compression. arXiv preprint arXiv:2306.088 03078, 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.217885 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Fisher, R. A. Theory of statistical estimation. In Mathemat- ical proceedings of the Cambridge philosophical society, pp. 700-725, Cambridge University Press, 1925.
Anadkat, S., al. Gpt-4 technical report. arXiv preprint 2330::2303.08774.. 2023.
Arefeen, M. A., Debnath, B., and Chakradhar, S. Leancon- text: Cost-efficient domain- seeeainn-peeificcc question answering using llms. Natural Language Processing Journal, 7: 100065, 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17325, 2022.
Bai, Y. Lvv X., Zhang, J., Lyu, H., Tang, J., Huang, Z Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context under- standing- In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3119-3137, 2024.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version v0. 0. Sept, 10:8-9, 2021.
Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312 / 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-1901, 2020.
Guo, H., Brandon, W., Cholakov, R., Ragan-Kelley. J., Xing, E., and Kim, Y. Fast matrix multiplications for lookup table-quantized 1lss. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12419- 12433, 2024.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240)11-13,, 2023.
Gurobi Optimization, Gurobi Optimizer Reference Man- ual, 2025. URL https: / / /ww. www.bi.rrr....mo. com/ doc umentation/ Version 12.0.
Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 1:Xiv:5110.0000 100.00149, 2015.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Pupers), pp. 2924-2936, 2019.
Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. Advances in Neural Information Processing Systems, 5, 1992.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 05555, 2018.
Huang, L., Li, Z., Sima, C., Wang, W., Wang, J., Qiao, Y., and Li, H. Leveraging vision-centric multi-modal expertise for 3d object detection. Advances in Neural Information Processing Systems, 36, 2024.
10

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
IBM. IBM ILOG CPLEX Optimization Studio Documenta- fion, 2025. URL https::////.t....... /tt///wwwwr /www. b.mm com/ docs / n/icos. Version 22.1.
com/bblog/lllaaa-2222oooo n-addee-mbblle--derrreess// September 2024b.
2024--122
Mo, Z.,, Wang, L.,, J., Zeng, Z.,, Cao, S., Ma, L., Jing, N., Cao, T., Xue, J., Yang, F., et al. Lut tensor core: Lookup table enables efficient low-bit Ilm inference acceleration. 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.083611 2020.
arXiv preprint arXiv:2408.06600
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: dense- and-sparse quantization. In International Conference on Machine Learning, pp.
PMLR, 2024.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing systems, 32, 2019.
Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. Pre- trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1-39, 2024a.
Li, M., Lin, Y., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05000 2024b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-67, 2020.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device 11m compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- 106, 2021.
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. LIm-qat: Data-free quantization aware training for large language models. In Findings of the Association for Computational 2024.
Shao, W., Chen, M., Zhang. Z., Xu, P., Zhao, L.,, Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024.
Linguistics ACL 2024, pp.
Ma, Y., Li, H., Zheng, X., Ling, F., Xiao, X., Wang, R., Wen, S., Chao, F., and Ji, R. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint M.X.:23332.11777 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Mareus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies. A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analy- sis platform for natural language understanding. arXiv preprint arXiv: 1804.07461, 2018.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference Learning Representations, 2017.
Wang, Y., Ma, X., and Chen, W. Augmenting black-box llms with medical textbooks for clinical question answering. arXiv preprint S3.i::2209002233, 2023.
Meta AI. Llama-3: Meta ai's latest language model. ht tps: / ai meta. com hew//bloo blog/ / /meta -11 ammam33/ 2024a.
Wolf, T.
Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint urXiv:1910.03777 2019.
Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https:/ / //aa///.//// meta.

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han,
S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Xu, Y., Wang, Y., Zhou, A., Lin, W., and Xiong, H. Deep neural network compression with single and multiple level quantization. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2018.
Yao, Z.,, Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and Hc, Y. Zeroquant: Efficient and affordable post- training quantization for large-scale transformers. Ad- vances in Neural Information Processing Systems, 35: 27168- 21183, 2022.
Yvinec, E., Dapogny, A., and Bailly, K. Nupes: Non- uniform post-training quantization via power exponent search. arXiv preprint arXiv:2308.0560
2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: / 195500830.. 2019.
Zhang, B., Haddow, B., and Birch, A. Prompting large language model for machine translation: A case study. In International Conference Machine Learning, pp. 4 41092-41110. PMLR, 2023.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S.. Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0111 2022.
12

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
A. Ensuring Positive Definiteness via Diagonal Dominance
In some cases, such as the fc2 layer of OPT models, the matrix XX may fail to be positive definite. While this uncommon in our experiments, it is important to handle such situations before performing Cholesky decomposition in Algorithm which requires positive definite matrix. To ensure this, we enforce diagonal dominance via small adjustment to the matrix.
Specifically, a symmetric matrix A is guaranteed to be positive definite if it is diagonally dominant with strictly positive diagonal entries, meaning laæl 2 Ei |ayi for all i. To utilize this property, let us denote Σ = xx", where diagonal elements satisfy En 2 0. We compute an adaptive offset vector ó, where each component 6, is calculated as:
n
6, = max
Σ
E
2 10-8
(23)
where E,, denotes the element the i-th row and j-th column E. Subsequently, we obtain a diagonally dominant matrix by adding this adaptive diagonal offset and perform the Cholesky decomposition:
Cholesky (Σ Diaa(o)).
(24)
where Dōa generates diagonal matrix from the vector 8, and L is the resulting lower wwiar-tri Cholesky factor. This adaptive preconditioning procedure efficiently ensures numerical stability and positive definiteness, facilitating robust computation without manual parameter adjustments.
To evaluate sensitivity to preconditioning approaches (the standard fixed À illustrated in Remark 3.1 versus adaptive method), we conduct 4-bit quantization experiments on OPT-125M and report the resulting perplexity on WikiText-22 in Table 7.
Table 7. WikiText-2 perplexity (1) of 4-bit quantized OPT-125M under different preconditioning strategies.
xx
+II
Method in (23) (24)
1.0
10.0
40.0
100.0
29.14
29.04
28.98
29.05
29.09
28.58
The results show that quantization accuracy of GANQ is largely insensitive to the choice of preconditioning. The adaptive method achieve the best result, with fixed À yielding similar results. All results outperforms baselines, confirming the robustness.
B. Outlier Extraction Method
In this section, describe the method we implemented extract outliers from the matrix W. This method decomposes W into two components: sparse component W sparse. which contains the extracted outliers, and a dense component W denses which consists of the remaining non-outlier values. We then perform quantization on Wdense to enhance quantization performance.
The pseudocode shown in Algorithm 2. This decomposition is achieved through row-wise outlier extraction process based on an extraction ratio r, where 0 < r < 1 (c.g.. 0.5%). The process begins by computing a tail percentile p = 1 0.5 r. which determines the boundaries for identifying outliers each row symmetrically. For each row of the weight matrix, the algorithm sorts the values in ascending order and computes the upper and lower cutoff values corresponding to the percentiles and p. These cutoff values define the outliers, which are those values that fall either above the upper percentile or below the lower percentile. An outlier mask M is then created, where values that are identified as outliers are marked with 1, and non-outliers are marked with 0. The sparse component W sparse is obtained by multiplying the weight matrix W semmnnt-wise with the outlier mask, while the dense component Weene is obtained by subtracting the sparse component from the original matrix.
13

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Algorithm 2 Outlier Extraction and Weight Decomposition
Input: Weight matrix W E Rm outlier extraction ratio 0 < <<<
Output: Sparse component W sparse, Dense component W p
0.5 X
compute tail percentile initialize outlier mask M with zeros
M <-0
upper + In pl
W sondd - sort( ((:)
row-wise sorting # upper cutoff values
Cupper
W sorted E; upper
lower In
p}]
Clower
W sonted F lower
lower cutoff values identify outliers # mark outliers in the mask # extract outliers extract non-outliers
O +
W 2 Cupper) V (W
Clower)
O
W spurse W o M
W dense <- W W
Return W spase, dense
C. Additional Results
The results in Table 8 present the C4 perplexity of various quantized models under 4-bit and 3-bit configurations across different model sizes. As shown, GANQ outperforms baseline methods such as RTN, GPTQ, and OmniQuan across all configurations. For 4-bit quantization, GANQ achieves the lowest perplexity across both OPT and LLaMA models, with notable improvements. GANQ also demonstrates strong performance with 3-bit quantization, maintaining competitive perplexity reductions across model sizes. OPT-6.7B, GANQ's perplexity is 13.68, compared to 17.00 for GPTQ and 15.51 for OmniQuant. These results highlight GANQ's effectiveness both 4-bit and 3-bit quantization, achieving substantial perplexity reductions across wide range of model scales. The symbol "_" in Table 8 indicates that OmniQuan cannot quantize LLaMA-3-88 on single NVIDIA RTX 4090 GPU due to memory constraints, the quantized model is unavailable in their model 700.
Table 8, C4 perplexity (L) of quantized models under 4-bit and bit GANQ outperforms state-of- state-of-the-ate the-art methods.
OPT
LLaMA
Method
Bit-width
125M
350M
1,3B
2.7B
6.7B
7B
2-7B
3-8B
Full
16
26.56
22.59
16.07
14.34
12.71
7.34
9.45
RTN
4
33.89
26.21
27.49
18.83
14.37
8.12
8.16
13.35
GPTQ
4
29.08
24.64
17.00
14.99
13.18
8.83
7.87
$1.33
OmniQuant
4
28.76
23.85
16.85
14.93
13.10
7.66
7.77
GANQ
27.72
23.47
16.54
14.71
12.96
7.52
7.47
10.23
RTN
3
8.3e2
55.15
6.5e3
1.0c4
5,0e3
20.78
5.3e2
5.7e2
GPTQ
3
42.14
30.90
21.52
18.24
17.00
22.28
11.67
70.53
OmniQuant
3
36.37
28.82
19.61
19.10
15.51
8.75
9.38
GANQ
3
33.59
29.46
18.46
16.43
13.68
8.20
8.20
12.88
The results in Table 9 present the PTB perplexity of various quantized models under 4-bit and 3-bit configurations across different model sizes. As shown, GANQ outperforms baseline methods such as RTN, GPTQ, and OmniQuan across all configurations. For 4-bit quantization, GANQ achieves the lowest perplexity across OPT models with notable improvements. GANQ also demonstrates strong performance with 3-bit quantization, maintaining competitive perplexity reductions across model sizes. OPT-6.7B, GANQ's perplexity 16.91, compared to 21.63 for GPTQ and 21.52 for OmniQuant. These results highlight GANQ's effectiveness both 4-bit and 3-bit quantization, achieving substantial perplexity reductions across range of model sizes.
LLaMA-7B and LLaMA-2-7B perform similarly to the much smaller OPT-125M model in full-precision FP16 configuration on the PTB dataset. Specifically, the FP16 versions of LLaMA-7B (41.15) and LLaMA-2-7B (37.91) do not achieve significantly better perplexity than the OPT-125M model 99), highlighting the relative inefficiency of LLaMA models on this dataset. Therefore, we focus on reporting results for OPT models, which demonstrate stronger performance in this context.
14

---

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Table PTB perplexity (L) of quantized models under 4-bit and 3-bit. GANQ outperforms state-of-the setthe-ar methods.
OPT
Method
Bit-width
125M
350M
1.3B
2.7B
6.7B
Full
16
38,99
31.07
20.29
17.97
15.77
RTN
53.88
36.79
75.40
32.40
18.86
GPTQ
45.45
34.33
22.04
19.19
16.58
OmniQuant
4
42.53
33.80
21.79
19.00
16.18
GANQ
4
40.75
33.21
21.06
18.73
15.95
RTN
3
1.4e3
87.20
1.5e3
1.2c4
5.4c3
GPTQ
3
72.91
47.17
31,94
25.63
21.63
OmniQuant
3
59.56
42.65
26.87
29.82
21.52
GANQ
3
55.67
44.58
24.27
21.28
16.91
Table 10. Perplexity (1) of quantized LLaMA-3.2 models on WikiTex12 and C4, GANQ outperforms statc-of-the-art methods.
Dataset
Method
Bit-width
1B
3B
IB-Instruct
3B-Instruct
16
9.76
7.81
11.05
KTN
4
18.08
10.53
22.91
15.58
GPTQ
4
24.07
6.0e3
18.40
4.5e3
OmniQuant
12.90
8.87
12.33
WikiText-2
GANQ
4
10.78
8.35
14.36
11.99
RTN
3
2.6e3
4.8e2
7.0e3
5.7e2
GPTQ
3
1.68e2
6.3c3
1.1e2
3.3e3
OmmiQuant
3
4.3e2
14.82
34.89
19.55
GANQ
3
15.91
10.85
21.32
15.84
FP16
16
14.02
11.34
21.30
16.50
RTN
4
27.98
15.89
35.23
22.15
GPTQ
4
41.27
2.1e3
27.99
3.2e3
OmniQuant
4
18.34
13.08
25.20
18.02
GANQ
4
15.53
12.21
22.81
17.35
RTN
3
2.9e3
4.1e2
6.7e3
5.3e2
GPTQ
3
1.5e2
1.6c3
1.2c2
2.0c3
OmniQuant
3
3.9c2
19.96
41.43
24.80
GANQ
3
22.09
15.36
29.18
21.30
Table 10 presents the perplexity of quantized LLaMA-3.2 models on WikiText-2 and C4 under 4-bit and 3-bit configurations. GANQ consistently outperforms baseline methods (RTN, GPTQ, OmniQuant) across model variants and bit-widths. Our systematic optimization framework for layer-wise non-un -nifoooormm quantization allows GANQ to adapt effectively to varying weight distributions, unlike baseline methods, which demonstrate greater sensitivity and instability, especially 3-bit quantization.
15


Reviews:
[{"review_id": "HdlZ0imOKE", "reviewer": "Reviewer_WtKA", "summary": "This paper proposes a GPU-adaptive non-uniform quantization framework for LLMs. By formulating quantization as a mixed-integer optimization problem, the authors aim to achieve efficient low-bit inference on hardware that lacks native mixed-precision matrix multiplication support. Their approach relies on lookup tables (LUTs) rather than repeated dequantization. They introduce an iterative algorithm to solve for the best codebooks per layer in a way that (i) addresses outliers, and (ii) facilitates rapid table lookups. The paper reports strong perplexity results on several LLM architectures (OPT, LLaMA, etc.), along with speedups when compared to FP16 baselines.\n\n## update after rebuttal\n\nThanks to the authors for the explanations provided in response to my questions, which largely address my initial concerns. However, the paper still requires significant revisions to offer a more comprehensive discussion and comparison with other LUT-based methods. It also needs more rigorous experiments, concrete evidence of real-world efficiency, and details on efficient implementation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough and insightful feedback. We would like to address your concerns point by point below.\n\n## Clarification of References ([1, 2]) and Response to Weakness 2\nThank you for highlighting the relevance of references \"Fast Matrix Multiplications for Lookup Table-Quantized LLMs [1]\" and \"LUT Tensor Core: Lookup Table Enables Efficient Low-Bit LLM Inference Acceleration [2]\". As their titles suggest, these works primarily contribute optimized LUT-based matrix multiplication kernels and tensor cores. In contrast, our primary contribution is a novel non-uniform quantization scheme tailored for LUT-based inference. Our method produces quantized representations that can leverage and complement the kernels or tensor cores proposed in [1,2]. Indeed, enabling such compatibility was part of our motivation, which we have explicitly mentioned in the Introduction (Lines 93-95, left column), where [2] has been cited.\n\nDue to different objectives, [1] and [2] focus on end-to-end inference speedup using custom kernels, while our work emphasizes quantization accuracy and compatibility with LUT-based kernels. \n\nFor inference speed, we currently use the kernel from [3], as noted in Section 4.1. In the future, we plan to adopt newer LUT-based inference kernels like [1,2] to take advantage of their advancements. \n\nFor quantization accuracy, to the best of our knowledge, [2] does not propose a new quantization scheme. Instead, it validates the LUT tensor core by precomputing lookup tables with operator fusion. For [1], it utilizes a learned NormalFloat quantization scheme. As noted in their Section 4.2: \"*Based on our earlier experiments, we selected a group size of 64, which strikes a good balance between quality and speed.*\" This means that every 64 elements share a lookup table. While this improves quantization quality, it increases memory overhead. For example, for LLaMA-3-8B's $W_{q}$ ($4096\\times 4096$), a group size of 64 results in 64 ($4096/64=64$) lookup tables per row (each with 16 FP16 values for 4-bit quantization). In contrast, our method is initially designed to support one lookup table per row (i.e., channel-wise), which is much more memory-efficient. Below is the theoretical model compression ratios (0.5 byte for INT4, 2 byte for FP16):\n|[1]|Ours\n|-|-|\n|$\\frac{0.5\\times 4096^2+2\\times 64\\times 16 \\times 4096}{2 \\times 4096^2}=50\\\\%$|$\\frac{0.5\\times 4096^2+2\\times 1\\times 16 \\times 4096}{2 \\times 4096^2}=25.39\\\\%$\n\nBesides, as noted in Section 3.3, our method supports outlier-handling techniques by extracting outliers into a sparse matrix, balancing compression and quality. For LLaMA-3-8B, our method already outperforms [1] in 3-bit quantization with just 0.5% outlier separation. In the 4-bit setting, increasing the outlier ratio can further improve quality of GANQ, for example, with a 5% ratio, GANQ achieves a PPL of 6.25, which is comparable to [1] using a group size of 64.\n\n|WikiText PPL|3-bit\n|-|-|\n|[1] with group size 64|7.5\n|GANQ + 0.5%|7.46\n\nThese results highlight a key trade-off between quality and memory, with our method offering a much more compact representation while maintaining strong quality.\n\nWe will include these comparisons in the revision to highlight our method's unique benefits and compatibility with existing LUT-based kernels.\n\n## Response to Weakness 1\n\nThank you for raising the concern about real-time inference improvements. To clarify, we included inference benchmarks in Table 5. Following prior works [3,4], we used batch size 1 to generate 1024 tokens and reported total CUDA time (in seconds) using the LUT kernel from [3], as detailed in Section 4.1. The results show clear overall inference speedup and peak memory reduction for OPT-6.7B and LLaMA-7B, highlighting the practical benefits of our method over uniform quantization and FP16 baselines.\n\nWe acknowledge the importance of testing across different batch sizes and sequence lengths and will expand our experiments accordingly. However, we emphasize that efficiency is largely determined by the underlying LUT-based kernel implementation.\n\nAs noted in our response regarding LUT kernels [1,2], our main contribution is a novel LUT-based non-uniform quantization scheme. We plan to integrate optimized LUT kernels in future work to further improve efficiency. We appreciate the suggestion and will highlight the practical benefits more clearly. We believe that both our work and LUT kernel innovations can jointly make valuable contributions to the community.\n\n[3] Kim, Sehoon, et al. \"Squeezellm: Dense-and-sparse quantization.\"\n\n[4] Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\"\n\n## Response to Weakness 3\n\nWe fully agree that open-sourcing will enhance reproducibility and impact. We will release the complete code upon acceptance.\n\n## Typo\nThank you for pointing out the typo, we will correct it in the revised version.", "claims_and_evidence": "The main claims are that (a) non-uniform LUT-based quantization, solved through a GPU-friendly mixed-integer optimization procedure, can significantly improve perplexity at lower bitwidths versus uniform quantization baselines; (b) the proposed method is broadly compatible with outlier-handling strategies; and (c) it yields better throughput and memory savings. \n\nThese are overall supported by experiment results.", "methods_and_evaluation_criteria": "The core method is well-motivated and the criteria and benchmarks are widely used.", "theoretical_claims": "The primary theoretical discussion is the layer-wise formulation as a mixed-integer quadratic program and its subsequent decomposition. There does not appear to be any new theorems with formal proofs requiring correctness checks.", "experimental_designs_or_analyses": "The experimental design is approporate.\n\nWhile there lacks real end-to-end performance numbers (throughput/latency in tokens/second or ms/token) directly compared to baseline quantization methods. Currently, memory usage numbers are mentioned selectively; a more explicit breakdown across methods and model sizes would clarify the trade-off.", "relation_to_broader_scientific_literature": "This paper focus on LLM quantization, which is related to works like GPTQ, OmniQuant, AWQ, etc.", "essential_references_not_discussed": "- Fast Matrix Multiplications for Lookup Table-Quantized LLMs [1]: The authors should discuss how their proposed LUT-based approach differs from or improves upon the matrix multiplication kernels and quantization schemes in that work.\n- LUT Tensor Core: Lookup Table Enables Efficient Low-Bit LLM Inference Acceleration [2]: Though this work is cited, there needs a in depth discussion since this work also reports accuracy vs. speed trade-offs with LUT-based quantization. Drawing explicit comparisons would be valuable, especially for verifying whether the GPU-adaptive optimization here significantly outperforms simpler LUT-coded designs.\n\n[1] https://arxiv.org/abs/2407.10960\n\n[2] https://arxiv.org/abs/2408.06003", "other_strengths_and_weaknesses": "I summarize all strengths and weaknesses here:\n\nStrengths:\n- Well-motivated approach in bridging the hardware-software gap for efficient low-bit inference.\n- The per-layer, parallelizable formulation is elegant and presumably implementable with moderate engineering effort.\n- Strong empirical results on perplexity across multiple models.\n\nWeaknesses:\n- Limited demonstration of real-time inference improvements. In practice, adopters would want to see the latency/throughput gains across model sizes, batch sizes, and sequence lengths.\n- Comparisons with more LUT-specific or hardware-accelerated methods are not deeply explored.\n- Code availability is not explicitly offered. Releasing the code would help ensure reproducibility and allow the community to adopt the method more easily.", "other_comments_or_suggestions": "In tables, OminiQuant -> OmniQuant", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHQcHQ4kFu", "reviewer": "Reviewer_nv8K", "summary": "The paper proposes GANQ, a post-training non-uniform quantization method optimized for hardware-efficient mpGEMM for LLMs. GANQ is LUT-based weight-only quantization capable of handling outliers. The experimental results demonstrate that the proposed GANQ outperforms baselines and achieves up to 2.57 times speedup.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review and helpful suggestions. We appreciate your positive feedback and address your concerns below. \n\n**Comment:** *$m,n,p$ in line 147 and 148 are not introduced.*\n\n**Response:** $m,n,p$ denote the dimensions in a linear layer. In LLMs, $m$ is the output dimension, $n$ the input dimension, and $p$ the total number of tokens processed (batch size $\\times$ sequence length). These values vary by models and layers. For example, in LLaMA-7B:\n||m|n\n|-|-|-|\n|$W_{k,q,v,o}$|4096|4096\n|$W_{gate,up}$|11008|4096\n|$W_{down}$|4096|11008\n \nWe use 128 samples with a sequence length of 2048, so $p=128\\times 2048$. This highlights the large scale of the problem and the efficiency of our method in solving it.\n\n**Suggestion 1:** *The settings of GANQ$^\\*$ (with outlier handling) are not explained ...*\n\n**Response:** We describe GANQ$^\\*$'s outlier-handling in Section 3.3 and detail it in Algorithm 2 (Appendix A), which separates outliers based on a ratio $r$, with the rest for quantization. As noted in Lines 365-367 (right), we typically set $r=0.5\\\\%$ for a fair comparison. We will further clarify this in the revision.\n\n**Suggestion 2:** *Some baselines are missing from the profiling comparison ...*\n\n**Response:** In Table 5, we compare the efficiency of dequantization-based and LUT-based methods. Following [1], we use GPTQ as a representative baseline for dequantization-based method, as similar methods like RTN and OmniQuant can generally share the same inference kernel and offer comparable efficiency. We will clarify this in the revision.\n\nBesides, as in [1], we use GPTQ-for-LLaMa [2] as the inference kernel for GPTQ, which currently supports 4-bit acceleration only. Thus, Table 5 includes only 4-bit GPTQ results. While briefly mentioned in Section 4.3, we will clarify it further in the revision and plan to add results with GPTQ kernels that support 3-bit acceleration.\n\n**Suggestion 3:** *GANQ$^\\*$ should be compared in profiling ...*\n\n**Response:** We show the results for GANQ$^\\*$ below:\n\n||GANQ$^\\*$|CUDA Time (s)|Speedup|Peak Memory (GB)\n|-|-|-|-|-|\n|OPT-6.7B|4-bit|10.39|1.61|5.13\n||3-bit|10.73|1.56|4.39 \n|LLaMA-7B|4-bit|9.82|1.82|4.16\n||3-bit|8.85|2.02|3.32\n\nWhile GANQ$^\\*$ offers better quantization quality via outlier handling, it incurs slightly higher inference time due to separate sparse matrix operations. Thus, the choice between GANQ and GANQ$^\\*$ should depend on the desired trade-off between quality and efficiency.\n\nWe note that 3-bit quantization for OPT-6.7B in GANQ$^\\*$ leads to longer inference times than 4-bit, likely due to: (1) memory bandwidth reduction being the main speedup factor (as in our response to *Questions for Authors* below); (2) sparse outlier operations becoming a bottleneck in this case; and (3) 3-bit values misaligning with byte boundaries (INT8), causing entries to span bytes and adding indexing overhead.\n\nWe will clearly present and discuss these findings in the revision.\n\n**Suggestion 4:** *The ratio of outliers and quantized weights could be discussed ...*\n\n**Response:** As in Tables 2 and 3, GANQ is already effective without outlier handling, outperforming baselines and close to FP16 models.\n\nAs noted in Section 3.3, adding outlier handling can further improve quantization quality but increases inference time and memory, highlighting a trade-off. To illustrate this, we compare both the PPL and efficiency metrics in one table. For example, on LLaMA-7B:\n\n||PPL on WikiText|CUDA Time (s)|Speedup|Peak Memory (GB)\n|-|-|-|-|-|\n|FP16|5.68|17.86|1.00|13.06\n|4-bit|5.83|8.46|2.11|4.14\n|4-bit+0.5%|5.76|9.82|1.82|4.16\n\nThese results demonstrate a clear trade-off between quantization quality and efficiency. We will include this discussion in the appendix to highlight this balance.\n\n**Response to Questions For Authors**\n\nGANQ's speedup mainly arises from reduced memory bandwidth usage, which align with observation in other model compression methods (Figure 3 in [3]). By storing weights as 4-bit or 3-bit indices, GANQ greatly lowers memory traffic between global memory and CUDA cores compared to FP16. \n\nWhile GANQ introduces some overhead from table lookups and weight reconstruction, these are minimal. The lookup table are small (e.g., 16 entries, 32 bytes per row for 4-bit quantization), and thus reside in fast constant or shared memory, adding negligible latency. Weight reconstruction is efficiently fused with matrix multiplication in CUDA kernels, avoiding extra global memory accesses.\n\nIn summary, GANQ’s modest overhead is far outweighed by memory savings, resulting in faster overall inference, especially on bandwidth-constrained GPUs.\n\n**Response to Other Suggestions**\n\nThanks for catching the typo, we will fix it in the revision.\n\n[1] Kim, Sehoon, et al. \"Squeezellm: Dense-and-sparse quantization.\"\n\n[2] https://github.com/qwopqwop200/GPTQ-for-LLaMa\n\n[3] Lin, Ji, et al. \"Awq: Activation-aware weight quantization for on-device llm compression and acceleration.\"", "claims_and_evidence": "1. The paper proposes GANQ (GPU-Adaptive Non-Uniform Quantization), a post-training non-uniform quantization method.\n    \n2. GANQ uses GPU-adaptive optimization, improving efficiency and achieving several times speedup compared to baselines.\n    \n3. GANQ is a non-uniform quantization method capable of handling outliers, improving performance.\n    \n4. Experimental results confirm GANQ’s efficiency and performance.", "methods_and_evaluation_criteria": "The proposed method is evaluated on perplexity, CUDA time, peak memory, and quantization cost, providing a reasonable and comprehensive assessment of GANQ.", "theoretical_claims": "The theoretical proof of GANQ is well-constructed, clear and easy to follow.\n\n1. $m, n, p$ in line 147 and 148 (right column) are not introduced.", "experimental_designs_or_analyses": "The experimental designs and analyses are clear and comprehensive. However, some suggestions could improve the paper.\n\n1. The settings of GANQ\\* (with outlier handling) are not explained, such as the ratio of outliers and quantized weights.\n    \n2. Some baselines are missing from the profiling comparison (Table 5) without explanation.\n    \n3. GANQ\\* should be compared in profiling and quantization cost.\n    \n4. The ratio of outliers and quantized weights could be discussed in more detail, as different ratios affect performance and cost.", "relation_to_broader_scientific_literature": "The paper proposes a post-training non-uniform quantization method that improves efficiency and performance compared to previous works. The experimental results align with prior findings, showing that quantization can degrade perplexity while reducing memory demands.", "essential_references_not_discussed": "All essential references are discussed.", "other_strengths_and_weaknesses": "All strengths and weaknesses have been mentioned above.", "other_comments_or_suggestions": "Line 134 (right column): a repetition of the word “measures“", "questions_for_authors": "In Table 5 (CUDA time), the full model only performs matrix multiplication, while GANQ involves table lookup, weight replacement, and then matrix multiplication. What explains GANQ being faster than the full model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YCmhnC30H3", "reviewer": "Reviewer_xu7k", "summary": "The paper present GANQ, a GPU-Adaptive Non-Uniform Quantization technique specifically tailored for efficient inference of Large Language Models (LLMs). GANQ introduces a principled optimization model based on Mixed-Integer Quadratic Programming (MIQP) to achieve layer-wise quantization using a Lookup Table (LUT) approach. It leverages GPU acceleration to efficiently handle the computational complexity. Extensive evaluations demonstrate GANQ’s significant reduction in perplexity relative to state-of-the-art methods across various models and datasets, achieving a notable inference speedup (up to 2.57X on NVIDIA RTX 4090)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and recognition of GANQ's contributions. Below, we address the concerns raised.\n\n## Concern 1: Reliability of Cholesky Decomposition and Sensitivity to Preconditioning of $XX^\\top$\n\n**Response to Weakness 1**\n\nYes, our method relies on Cholesky decomposition to efficiently solve the S-subproblem. When $XX^\\top$ is ill-conditioned, we apply preconditioning to ensure the decomposition remains feasible, maintaining the effectiveness of our framework. This is a standard technique in numerical linear algebra.\n\n**Response to Questions for authors**\n\nYes, we implement an adaptive preconditioning method for $XX^\\top$ and examine its impact on quantization accuracy and efficiency below.\n\nRemark 3.1 introduces preconditioning $XX^\\top$ via $XX^\\top+\\lambda I$, with $\\lambda>0$. In practice, we adopt an adaptive method that enforces diagonal dominance, ensuring positive definiteness without manual tuning $\\lambda$.\n\nA symmetric matrix $A$ is positive definite if it is diagonally dominant with positive diagonal entries, i.e., $|a_{ii}| \\geq \\sum_{j \\neq i} |a_{ij}|$ for all $i$. Let $\\Sigma = XX^\\top$, with $\\Sigma_{ii} \\geq 0$. For each row $i$, we compute the offset vector $\\delta$ as follows:\n$$\\delta_i = \\max\\left(\\sum_{j=1}^n|\\Sigma_{i,j}|-2\\Sigma_{i,i},\\ 10^{-8}\\right),\n$$where $\\Sigma_{i,j}$ denotes the element at the $ i$-th row and $ j$-th column of $ \\Sigma $. Then, perform\n$$L=\\mathrm{Cholesky}(\\Sigma+\\mathrm{Diag}(\\delta)),\n$$where $\\mathrm{Diag}(\\delta)$ constructs a diagonal matrix from the vector $\\delta$, and $L$ is the lower-triangular Cholesky factor. \n\nTo assess sensitivity to preconditioning (fixed $\\lambda$ and adaptive) before Cholesky decomposition, we run 4-bit quantization experiments on OPT-125M and report PPL on WikiText.\n||$\\lambda=0.5$|$\\lambda=1.0$|$\\lambda=10.0$|$\\lambda=40.0$|$\\lambda=100$|Diagonally dominant\n|-|-|-|-|-|-|-|\n|PPL|29.14|29.04|28.98|29.05|29.09|28.58\n\nFor a clear comparison on quantization accuracy, we also show the baseline methods again below.\n||Full (FP16)|RTN|GPTQ|OmniQuant\n|-|-|-|-|-|\n|PPL|27.66|37.11|31.08|30.98\n\nThe results show that quantization accuracy is largely **insensitive** to the choice of preconditioning. The adaptive method achieve the best PPL (28.58), with fixed $\\lambda$ yielding similar results. All results outperforms baselines, confirming the robustness.\n\nIn terms of efficiency, our preconditioning uses simple operations (e.g., summation, diagonal adjustments) with no noticeable impact on overall efficiency.\n\n**We initially considered this an implementation detail or engineering trick and plan to include the discussion in the Appendix of the revised paper.**\n\n## Concern 2: Details on GPU-Based MIQP Solver Implementation\n\n**Response to Weakness 2**\n\n- **Details on how the MIQP is solved on GPU**\n\nSection 3.2 details how we solve the MIQP model using an alternating direction framework and leverage GPU for efficient computation.\n\nAs shown in Equation (5), the T-subproblem has a closed-form solution with matrix-vector multiplications, which are highly efficient on GPUs.\n\nAs shown in Equations (6)–(20), Figure 2, and the corresponding text (Lines 272–274, right column; Lines 295–300, left column), we illustrate how the S-subproblem is solved. Exploiting row-wise independence, we stack $W_i$ and $T_i$ vectors and organize $S_i$ matrices into a tensor, enabling parallel computation across rows on GPUs.\n\nIn summary, we group independent vectors and matrices into larger matrices and tensors, enabling operations like multiplication and summation to be efficiently accelerated on GPUs.\n\n- **The adoption of Gurobi or CPLEX**\n\nWhile Gurobi and CPLEX are powerful commercial solvers, they are designed on generic purpose for general optimization models, including MIQP. In contrast, our framework exploits the specific structure of the model under discussion, yielding greater efficiency than generic solvers that overlook such problem-specific properties.\n\nWe attempted to use Gurobi to solve Equation (2) during rebuttal, a single-row subproblem with OPT-125M dimensions, but it produced no output within three minutes. In contrast, our GPU-accelerated framework quantizes the entire model in the same time.\n\n## Concern 3: Comparison with AWQ and SqueezeLLM\n\n**Response to Weakness 3**\n\nWe compared our method with AWQ and SqueezeLLM in Table 4, as both incorporate mechanisms for handling outliers. AWQ utilizes a quantization block size of 128 (a default setting in the AWQ paper), which helps mitigate the impact of outliers. A key contribution of SqueezeLLM is isolating outliers into a separate sparse matrix while retaining 10 FP16 rows. Accordingly, we categorize these methods as outlier-aware quantization approaches and include their results in Table 4 for a fair comparison. As shown in Table 4, our method, equipped with a similar outlier-handling mechanism, achieves better performance compared to both.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "Novel approach to use MIQP to find the best layer wise quantization strategy", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "strengths\n1. GANQ clearly formulates non-uniform quantization as an MIQP problem, allowing systematic optimization rather than heuristic approaches.\n2. Demonstrates superior perplexity improvements across both 3-bit and 4-bit quantization, and outperforming established quantization methods.\n3. exploits GPU parallelism by decomposing the original optimization into highly parallelizible subproblems.\n\nweakness:\n1. The method relies heavily on Cholesky decomposition, potentially limiting its applicability if XX^T matrices are ill-conditioned.\n2. Limited details on how the MIQP is solved on GPU, there are mature toolkits for the optimization problems such as Gurobi or CPLEX, more discussions are needed.\n3. Table 2 does not include performance of AWQ and suqeezeLLM, better to have a more comprehensive experiements.", "other_comments_or_suggestions": "## update after rebuttal: I appreciate the responses from the author. I am happy to increase my score.", "questions_for_authors": "How sensitive is GANQ’s quantization accuracy and optimization efficiency to the conditioning of the Cholesky decomposition matrix XX^T? Have you explored alternative numerical methods for stability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pq1LTEIplw", "reviewer": "Reviewer_QPHs", "summary": "This paper proposes a look-up-table based non-uniform quantization algorithm for LLMs. The algorithm is based on mixed-integer quadratic programming, with a mathematical proof that the authors deduce. The results show that the proposed method has better accuracy than state of the art methods on 4-bits and 3-bits quantization. The authors also provide a CUDA implementation that shows more than 2x speedup over fp16 baseline on NVIDIA RTX 4090 GPU.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and strong support. We appreciate your recognition of our contributions and provide detailed responses below.\n\n**Comment 1:** *The context length of perplexity.*\n\nThanks for pointing this out. We use a sequence length of 2048 for all models and methods, following prior work, and will clarify this in the revision.\n\n**Suggestion 1:** *More recent models and model families/sizes:*\n\nThank you for the suggestion. To further validate GANQ, we conducted additional experiments on the latest LLaMA-3.2 models. Below are the WikiText PPL results (sequence length 2048, same setting as in our paper).\n\n||1B|3B|1B-Instruct|3B-Instruct\n|-|:-:|:-:|:-:|:-:\n|FP16|9.76|7.81|13.16|11.05\n|(4-bit / 3-bit)\n|RTN|18.08 / 2.6e3|10.53 / 4.8e2|22.91 / 7.0e3|15.58 / 5.7e2\n|GPTQ|24.07 / 1.7e2|6.0e3 / 6.3e3|18.40 / 1.1e2|4.5e3 / 3.3e3\n|OmniQuant|12.90 / 4.3e2|8.87 / 14.82|16.31 / 34.89|12.33 / 19.55\n|GANQ|**10.78** / **15.91**|**8.35** / **10.85**|**14.36** / **21.32**|**11.99** / **15.84**\n\nThe additional results confirm that GANQ consistently achieves superior and more stable quantization performance. Our systematic optimization framework for layer-wise non-uniform quantization allows GANQ to adapt effectively to varying weight distributions, unlike baseline methods, which demonstrate greater sensitivity and instability, especially at 3-bit. We will add these findings in the revision.\n\nRegarding the suggestion to evaluate more model families (e.g., DeepSeek) and larger scales (e.g., LLaMA-3 70B), we appreciate your understanding that such experiments are limited by our hardware (RTX 4090 GPU). That says, since GANQ operates row-wise on linear layers and is designed to adapt to diverse weight distributions, we expect it to generalize well to other architectures and larger models. We plan to extend our evaluations to broader model families and larger scales in the future work.\n\n**Comment 2:** *GPTQ CUDA speedup measurements*\n\nWe acknowledge that the GPTQ efficiency reported in Table 5 is lower than that in the original GPTQ paper. This discrepancy is mainly due to differences in the inference kernel, hardware, and model size, with the kernel likely being the key factor. \n\nAs noted in Section 4.1, we followed prior work [1] and used the GPTQ-for-LLaMA [2] inference kernel for GPTQ-quantized models. The reported results reflect actual observations. However, since [2] is implemented in Triton, and our experiments were conducted on an NVIDIA RTX 4090, potential incompatibilities or limited hardware support may explain the slower inference in our setup.\n\nTo further validate this, we performed an additional test using GPTQ's official CUDA kernel implementation [3] (available only for 3-bit quantization and OPT models). The results (total CUDA time in seconds) are as follows:\n\n||FP16|GPTQ (3bit)|GANQ (3bit)\n|-|-|-|-|\n|OPT-6.7B|16.784|6.543|6.519\n\nThe results are consistent. We will incorporate these findings and clarifications into the revision to resolve potential confusion clearly.\n\n**Suggestion 2:** *The derivation of the closed-form solution of Equation 5*\n\nWe will add the detailed derivation of Equation 5 in the appendix of the revision.\n\nLet\n$$\nf(\\\\mathbf{T}_i)=\\\\|\\\\mathbf{W}_i\\\\mathbf{X} \\\\mathbf{T}_i\\\\mathbf{S}_i^{k+1}\\\\mathbf{X}\\\\|^2.\n$$\nby the first-order optimality, let\n$$\n\\\\nabla f(\\\\mathbf{T}_i)=2(\\mathbf{W}_i\\\\mathbf{X}- \\\\mathbf{T}_i\\\\mathbf{S}_i^{k+1}\\\\mathbf{X})\\\\mathbf{X}^\\\\top(\\\\mathbf{S}_i^{k+1})^\\\\top=0,\n$$\nwe have\n$$\n\\\\mathbf{T}_i^{k+1}=\\\\mathbf{W}_i \\\\mathbf{XX}^\\\\top (\\\\mathbf{S}_i^{k+1})^\\\\top ((\\\\mathbf{S}_i)^{k+1}\\\\mathbf{XX}^\\\\top (\\\\mathbf{S}_i^{k+1})^\\\\top)^\\\\dagger,\n$$\nwhere $(\\\\cdot)^\\\\dagger$ denotes the Moore-Penrose inverse.\n\n**Corrections** \n\nWe agree that \"perplexity on language datasets\" is a clearer phrase and will adopt this wording in the revision.\n\n**Response to Question for Authors**\n\nAlthough both GPTQ and our proposed method (GANQ) quantize model weights in a row-wise manner, the strategies differ significantly. GPTQ uses a greedy, sequential element-wise quantization based on the Optimal Brain Surgeon framework [4], quantizing one element at a time and updating subsequent weights to minimize reconstruction error.\n\nIn contrast, our method models LUT-based non-uniform quantization as a mixed-integer quadratic program. Rather than sequentially quantizing elements, we employ an alternating optimization for an entire row with a closed-form solution for the T-subproblem and an efficient back-substitution algorithm for the S-subproblem, enabling more systematic and effective quantization than GPTQ.\n\n\n[1] Kim, Sehoon, et al. \"Squeezellm: Dense-and-sparse quantization.\"\n\n[2] https://github.com/qwopqwop200/GPTQ-for-LLaMa\n\n[3] Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\"\n\n[4] Hassibi, Babak, and David Stork. \"Second order derivatives for network pruning: Optimal brain surgeon.\"", "claims_and_evidence": "- The claims of obtaining higher accuracy compared to other methods have been supported by strong results", "methods_and_evaluation_criteria": "- Paper evaluated different model families (OPT and Llama), diferent generations (LLama1, Llama2, and Llama3), and different sizes (8B, 70B, etc.)\n   - I would suggest adding: \n        - other more recent families (e.g., Mistral or DeepSeek)\n        - other sizes of recent models (e.g., Llama 3.2 1B, Llama3 70B). Though I understand that the authors may not be able to fit large models if they only have access to a RTX 4090 GPU\n- Compared with a good number of approaches that use or don't use outlier mitigation techniques", "theoretical_claims": "- I don't claim that I have fully understood the mathematical proof.", "experimental_designs_or_analyses": "- I believe accuracy comparisons are thorough and done properly\n   - However, I am concerned about the context length of perplexity, as different papers have evaluated perplexity with different context length. So need to double check that context length is consistent in the comparisons.\n- Though the CUDA implementation of the proposed method has decent speedup, I am concerned that the measured GPTQ CUDA speedup was around 0.3x (i.e., 3 times slower than the baseline), while Table 6 of the GPTQ paper shows speedups of up to 4.53x, albeit on a different GPU, and different LLM architecture. If there is an issue in reproducing similar speedups claimed by the GPTQ paper, I suggest not to include those measurements in the table", "relation_to_broader_scientific_literature": "The paper compared with latest state of the art quantization algorithms (GPTQ, AWQ, OmniQuant, SqueezeLLM)", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Usage of mixed integer quadratic programming is probably a novelty in the quantization field. Though, I might be wrong.\n- The proposed algorithm is based on rigorous mathematical proof\n- The proposed algorithm was optimized for parallelized compute platforms and implemented on GPU\n- \"GANQ consistently outperforms baseline methods such as RTN, GPTQ, and OminiQuant across all configurations\"\n- The proposed algorithm doesn't require a lot of memory, unlike approaches like SqueezeLLM and OmniQuant, and hence can quantize larger models on smaller GPUs.", "other_comments_or_suggestions": "- Equation 5: Please provide citation to a paper or textbook that explains how the closed form solution could be obtained\n- Line 286, Column 2: \"by reporting perplexity on language generation tasks\": my understanding generation tasks don't measure perplexity. I would prefer to re-word it to \"perlexity on language datasets\"\n    - Please specify context length of perplexity evaluation", "questions_for_authors": "- Is the algorithm similar to GPTQ's algorithm, in the sense that both decide to quantize each element in a row sequentially?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "RTCxsjGnrA", "reviewer": "Reviewer_V6cW", "summary": "The paper proposes GANQ, a GPU-Adaptive Non-Uniform Quantization framework leveraging lookup table (LUT)-based mixed-precision GEMM for efficient deployment of Large Language Models (LLMs). GANQ introduces a post-training quantization optimization algorithm to effectively solve LUT-based quantization objectives, significantly reducing both quantization cost and quantization errors. Experiments show GANQ outperforms existing methods in reducing perplexity gaps at 3-bit and 4-bit quantization, achieving up to 2.57× inference speedup on an NVIDIA RTX 4090 GPU.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback. Below, we address your concerns one by one.\n\n**Response to Weakness 1**\n\nWe acknowledge the value of broader evaluation and have conducted additional experiments.\n\nBelow are WikiText PPL results for **LLaMA-3.2** models, using the same settings as in the paper.\n\n||1B|3B|1B-Instruct|3B-Instruct\n|-|:-:|:-:|:-:|:-:\n|FP16|9.76|7.81|13.16|11.05\n|(4-bit / 3-bit)\n|RTN|18.08 / 2.6e3|10.53 / 4.8e2|22.91 / 7.0e3|15.58 / 5.7e2\n|GPTQ|24.07 / 1.7e2|6.0e3 / 6.3e3|18.40 / 1.1e2|4.5e3 / 3.3e3\n|OmniQuant|12.90 / 4.3e2|8.87 / 14.82|16.31 / 34.89|12.33 / 19.55\n|GANQ|**10.78** / **15.91**|**8.35** / **10.85**|**14.36** / **21.32**|**11.99** / **15.84**\n\nThese results confirm that GANQ consistently outperforms baselines with superior and more stable performance. While OmniQuant learns the quantization factors and is relatively robust, it still fails in some case (i.e, 3-bit for LLaMA-3.2-1B). RTN and GPTQ show significant sensitivity to weight distribution and degradation, with GPTQ notably unstable on 3B models. To explore this, we test GPTQ with a group size of 128 to mitigate outlier effects (same table head):\n\n||||||\n|-|:-:|:-:|:-:|:-:\n|GPTQ (g128)|15.37 / 72.00|1.1e2 / 1.1e3|16.03 / 28.31|49.14 / 6.9e2\n\nUsing a group size of 128 improves GPTQ's performance on 1B models but still performs poorly on 3B models, highlighting its sensitivity to weight distribution.\n\nWe initially did not include long-context or reasoning-intensive tasks as we evaluated base models without task-specific fine-tuning (e.g., LLaMA-2-7B). Following standard practice [1,2], we used Table 3 tasks to assess general capabilities.\n\nWe test LongBench and GSM8K on LLaMA-3.2 1B/3B-Instruct and their quantized versions.\n\n||1B-Instruct||3B-Instruct||\n|-|-|-|-|-|\n||LongBench|GSM8K (%)|LongBench|GSM8K (%)\n|FP16|11.5|32.90|12.7|64.97\n|RTN|0.2|4.17|12.1|37.68\n|GPTQ|error|11.14|error|~0.0\n|OmniQuant|8.9|16.53|11.5|54.74|\n|GANQ|11.7|27.75|12.5|60.50\n\nGANQ consistently outperforms baseline. For GPTQ, consistent with PPL results on 3B-Instruct, GSM8K accuracy is near zero. On LongBench, GPTQ failed with a \"skip_special_tokens\" error, despite identical settings across all methods and use of the official toolkit [5].\n\nWe plan to incorporate these results into the revision.\n\n**Response to Weakness 2**\n\nIn Table 5, we compare the efficiency of dequantization-based and LUT-based methods. Following [3], we use GPTQ as a representative baseline for dequantization-based method, as similar methods like RTN and OmniQuant can generally share the same inference kernel and offer comparable efficiency. We will clarify this in the revision.\n\nWe acknowledge that the GPTQ efficiency reported in Table 5 is lower than that in the original GPTQ paper. This discrepancy is mainly due to differences in the inference kernel, hardware, and model size, with the kernel likely being the key factor. \n\nAs noted in Section 4.1, we followed [1] and used the GPTQ-for-LLaMA [2] inference kernel for GPTQ-quantized models. The reported results reflect actual observations. However, since [2] is implemented in Triton, and our experiments were conducted on an NVIDIA RTX 4090, potential incompatibilities or limited hardware support may explain the slower inference in our setup.\n\nAs for metric, as noted in Section 4.1, we followed [1,3], using a batch size of 1 to generate 1024-token sequences and reporting total CUDA time.\n\nTo further validate this, we performed an additional test using GPTQ's official CUDA kernel implementation [1] (available only for 3-bit quantization and OPT models). The results (total CUDA time in seconds) are as follows:\n\n||FP16|GPTQ(3bit)|GANQ(3bit)\n|-|-|-|-|\n|OPT-6.7B|16.784|6.543|6.519\n\nThe results are consistent. We will incorporate these findings and clarifications into the revision to resolve potential confusion clearly.\n\nFinally, we would like to emphasize that our main contribution is the proposed LUT-based non-uniform quantization method, which improves quantization accuracy. For inference efficiency, it can similarly benefit from advances in LUT-based kernel implementations, as dequantization-based methods do with their kernels.\n\n**Response to Weakness 3**\n\nAs detailed in Section 4.1 (Lines 313–318, right column) and reiterated in Response to Weakness 2, we follow the evaluation setup used in prior works [1,3] to measure inference latency. Specifically, we report the total CUDA time required for the model to generate 1024 tokens, using a batch size of 1 and no initial input tokens. We will make these settings more explicit in the revised paper to improve clarity and reproducibility.\n\n[1] Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\"\n\n[2] Sun, Mingjie, et al. \"A simple and effective pruning approach for large language models.\"\n\n[3] Kim, Sehoon, et al. \"Squeezellm: Dense-and-sparse quantization.\"\n\n[4] https://github.com/qwopqwop200/GPTQ-for-LLaMa\n\n[5] https://github.com/THUDM/LongBench", "claims_and_evidence": "Please refer to \"*Methods And Evaluation Criteria*\".", "methods_and_evaluation_criteria": "Strengths:\n+ Introduces GANQ, an innovative GPU-adaptive, LUT-based non-uniform quantization method, demonstrating clear perplexity improvements for quantized OPT and Llama models.\n+ Provides promising inference efficiency improvements (up to 2.57× speedup) on real GPU hardware (NVIDIA RTX 4090).\n\nWeaknesses:\n- Accuracy evaluations are limited, lacking tests on more recent models (e.g., Llama-3.1), long-context scenarios (LongBench), and reasoning-intensive benchmarks (e.g., GSM8K).\n- Efficiency benchmarks only compare against GPTQ, with reported performance gains notably inconsistent (lower) compared to prior literature where GPTQ can gain 4x higher throughput.\n- Experimental setups for latency measurement lack critical details such as batch size and input token length, hindering clarity, reproducibility, and proper interpretation of the results.", "theoretical_claims": "Please refer to \"*Methods And Evaluation Criteria*\".", "experimental_designs_or_analyses": "Please refer to \"*Methods And Evaluation Criteria*\".", "relation_to_broader_scientific_literature": "This paper contributes to LLM quantization for efficient LLM inference.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to \"*Methods And Evaluation Criteria*\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces GANQ, a GPU-adaptive non-uniform quantization framework for large language models, addressing key limitations in existing quantization approaches. GANQ reformulates quantization as a layer-wise mixed-integer optimization problem optimized for LUT-based inference, yielding impressive perplexity improvements and up to 2.57× speedup over FP16 baselines. While reviewers initially raised concerns regarding comparisons to related LUT methods, evaluation breadth, and profiling consistency, the authors provided detailed responses, expanded benchmarks, and clarified the impact of kernel implementations and experimental setups. 
One remaining concern is that GANQ's efficiency gains, while solid, fall short of those reported for other 4-bit methods under optimized kernel settings, highlighting room for improvement in kernel-level integration. Given its empirical results and the authors’ thorough rebuttal and transparency, I recommend acceptance.
Decision: Accept (poster)