CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
Authors: Gunho Park
Jeongin Bae
Byeongwook Kim
Baeseong park
Jiwon Ryu
Hoseung Kim
Se Jung Kwon
Dongsoo Lee
Venue: NeurIPS 2025 poster ()
Keywords: Quantization
Matrix Multiplication
Large Language Models
Primary area: deep_learning

Abstract: Weight-only quantization is widely used to mitigate the memory-bound nature of LLM inference. Codebook-based methods extend this trend by achieving strong accuracy in the extremely low-bit regime (e.g., 2-bit). However, current kernels rely on dequantization, which repeatedly fetches centroids and reconstructs weights, incurring substantial latency and cache pressure. We present CodeGEMM, a codebook-centric GEMM kernel that replaces dequantization with precomputed inner products between centroids and activations stored in a lightweight Psumbook. At inference, code indices directly gather these partial sums, eliminating per-element lookups and reducing the on-chip footprint. The kernel supports the systematic exploration of latency–memory–accuracy trade-offs under a unified implementation.
On Llama-3 models, CodeGEMM delivers 1.83x (8B) and 8.93x (70B) speedups in the 2-bit configuration compared to state-of-the-art codebook-based quantization at comparable accuracy and further improves computing efficiency and memory subsystem utilization.

CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
Gunho Park, Jeongin Bae, Byeongwook Kim, Baeseong Park, Jiwon Ryu, Hoseung Kim, Se Jung Kwon, Dongsoo Lee
NAVER Cloud
(gunho. park3, dongsoo. leeeeeee@@lle wwwvercorp.pcoomm com github. com
www.comm//eeeeeeeeeeoeeeeeeeoaee hwwe//aaaaeeeeeee aaaver-aics//aaee codeggemm
Abstract
Weight-only quantization is widely used to mitigate the memory-bound nature LLM inference. Codebook-based methods extend this trend by achieving strong accuracy in the extremely low-bit regime (e.g.. 2-bit). However, current kernels rely dequantization, which repeatedly fetches centroids and reconstructs weights, incurring substantial latency and cache pressure. We present CodeGEMM, a codebook-centric GEMM kernel that replaces dequantization with precomputed inner products between centroids and activations stored in a lightweight Psum- book. At inference, code indices directly gather these partial sums, eliminating per-element lookups and reducing the on-chip footprint. The kernel supports the systematic exploration of eeeecy--meoryy merrryaaccurayy implementation. On Llama-3 models, CodeGEMM delivers 1.83x (8B) and 8.93 (70B) speedups in the 2-bit configuration compared sfate-of-the-ae codebook- based quantization at comparable accuracy and further improves computing effi- ciency and memory subsystem utilization.
accuracy trade-offs under a unified
1 Introduction
Driven by the power-law scaling principle, large language models (LLMs) have grown increasingly larger, delivering remarkable advancements in performance [9, 10). Notably, open-source models like Llama-3 [4] have reached a scale of 405 billion parameters, demonstrating significant improve- ments over their predecessors. However, deploying such massive models efficiently in production environments poses substantial challenges. For instance, the 405B model requires approximately 810GB of storage for its parameters alone, exceeding the memory capacity of single multi-GPU node configured with NVIDIA H100s, which offers a total of 640GB memory. To address these limitations, extensive research has been dedicated to developing various model compression techniques aimed at reducing model size while minimizing performance degradation [8, 24, 22].
Among compression strategies, quantization has emerged as particularly effective tool for cutting model size and bandwidth demands with minimal accuracy loss. In large language models (LLMs), challenges with activation quantization caused by activation outliers and the significant memory footprint weight parameters have spurred research into weight-only quantization [6]. These methods focus quantizing model weights to maximize memory efficiency without compromising performance.
State-of of-the-rrt techniques show that 4-bit weight-only quantization achieves nearly the same per- formance as unquantized models [13, 12], with notable progress even 3-bit quantization [21, 1]. However, in extreme low-bit (e.g., 2-bit) settings, uniform quantization suffers from significant per- formance degradation due to its limited representational capacity. Structured non-uniform methods
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Load Large Codebook
2
Redundant Comput.
1 Load Small
Psumbook Retrieval
Psumbook on Cache
Cache
Query
Cache
Retrivval
Psumbook
Codebook (FP16)
Impul (FP16)
Retrieval
:
Codebook (FP16)
Input
Psumbook
Code (INT2)
Dequantizatinn
Weight (FP16)
Output (FP16)
(FP16)
(FP16)
Code (INT2)
Output (FP16)
(a) On-the-fly dequantization
(b) Proposed
Figure I:
Comparison of matrix multiplication kernels for codebook-bar based quantized models. The iequantization-basee kernel performs on-the-fly dequantization, requiring the entire codebook be loaded into cache. In contrast, CodeGEMM precomputes partial sums and stores them in a Psumbook, eliminating dequantization overhead and redundant computation.
like Binary-Codde Quantization (BCQ) [30] improve performance over uniform quantization but still fall short of satisfactory results. These challenges emphasize the need for advanced non-uniform quantization techniques capable of maintaining high accuracy in extremely low-bit scenarios.
Codebook-based quantization has emerged as a promising algorithm capable of delivering superior performance in extremely low-bit environments [14, 16, 25, 26, 27]. Unlike traditional quantization methods that represent individual values with fewer bits, codebook-based quantization aims to represent data more efficiently by mapping input vectors to a limited set of centroid vectors. As a result, the original vectors are compressed and stored as codes pointing to the corresponding centroids in the codebook. However, this approach introduces a unique challenge: the efficient management of the codebook. As illustrated in Figure (()), unlike uniform quantization, codebook- based quantization relies on the codebook to reconstruct quantized values during computation. Without efficient codebook handling, the overhead associated with dequantization can negate the memory benefits of codebook-basee quantization, potentially reducing overall efficiency.
In this paper, we propose CodeGEMM, GEMM method designed to efficiently support codebook- based quantization, enabling practical speedups from extremely low-bit quantization. Unlike existing dequantization- based GEMM kernels, which load the entire codebook into programmable cache memory and rely on dequantization during computation, the proposed kernel precomputes all possible partial sum (Psum) results between the centroid and input data, storing these results as Psumbook in cache memory (Figure 1(b)). This approach eliminates the necessity for the traditional dequantization step, during which particular centroids are retrieved from the codebook through codes. Instead, the kernel directly retrieves precomputed Psums, reducing redundant computation and significantly decreasing the space complexity required for cache storage. Moreover, the proposed kernel supports a wide range of hyperparameters that define the configuration of codebook-base facilitating the execution of various quantized models within unified kernel. This flexibility enables users to explore and evaluate trade-offs among latency, memory usage, and accuracy, providing a versatile and efficient solution for quantized operations.
quantization,
The contributions of this work are as follows:
1. We introduce CodeGEMM, a new approach centered on codebooks to enhance the efficiency of GEMM in codebook-based quantized LLMs. CodeGEMM overcomes the drawbacks of existing methods, which require loading the entire codebook into programmable cache memory for dequantization-bant based operations and suffer from redundant computations.
2. CodeGEMM accommodates a broad spectrum of codebook hyperparameters, such as the number of codebooks, vector length, and group size, all within one kernel. This adaptability allows for investigating the trade-offs between latency, memory consumption, and accuracy in codebook-basee quantized models, facilitating better optimization for various use cases.
3. On 1lama-3.1 models, CodeGEMM delivers 1.83x (8B) and 8.93x (70B) speedups in the 2-bit configuration compared to state-of-the-art codebook-based quantization at comparable accuracy, and further improves computing efficiency and memory subsystem utilization.

---

Step I: Group Norm.
StepO: Clustering
Step 3 Centroid Assignment
16
-pa
+
+
o Px
0
2
1
E0
Centroid #0
o
Centroid
+
+++
+8+
-8-+
+8++
-
++
-8-
RF3
#1
3
0
E2
Centroid #2
x
2
3
F3
Centroid #3
-8+
-8-
+8+
-8--
Weight (FP16, M K)
Centroids (FP16, 2*)
Code (INT2, M KA) Codebook (FP16, 2"
Figure 2: Illustration quantization process of a (4 x 32) weight matrix with = m 1, v
and y 16.
2 Background
2.1 Weight-only quantization
As generative language models grow larger and demand greater memory, model quantization has become essential for reducing memory footprints, minimizing model size, and improving inference efficiency. Quantization is commonly applied two ways: (1) quantizing only the weights or (2) quantizing both weights and activations. However, activation quantization poses significant challenges due to the dynamic range of activation values and extreme outliers, termed massive activations [23]. which exceed typical values by over 2000x.
To address these challenges, many studies have focused on weight-only quantization. GPTQ [6] uses approximate second-order information compress weights 4 bits with minimal accuracy loss. AWQ [13] scales salient weights based on activation magnitude to reduce quantization-inde -inducce errors. Weight-only quantization enables compact model compression, reducing memory and accelerating inference with specialized kernels [6, 13, 20, 11]. However, at extremely low precision, still suffers notable accuracy degradation. For instance, QulP [3] uses random rotation matrices to mitigate this issue, achieving acceptable accuracy but still facing challenges at extremely low-bit levels. To overcome these limitations, non-uniform quantization methods have been introduced [5, 25, 14, 27J. Unlike traditional uniform quantization, non-uniform approaches offer greater flexibility in representing wider range of values, enhancing accuracy in extremely low-bit quantization scenarios [5].
2.2 Codebook- quantization
Extensive research has been conducted to utilize codebook-basee quantization for compressing large language models, utilizing its non-uniform properties and flexibility in representing wide range of values [5, 16, 25, 26, 27). In particular, GPTVQ [27] extended the GPTQ framework to incorporate codebook-basee quantization, interlcaving the quantization of one more columns with updates to the remaining unquantized weights. Similarly, AQLM [5] introduced an adaptation of multi-codebook quantization for large language models, proposing method to optimize the codebook using a calibration dataset. QulP# (25) adopts a smoothening approach similarly to QulP [3], applying rotation matrix to transform weights into a space that minimizes worst-case quantization error before mapping them to structured lattice codebooks. Similarly, QTIP [26] builds on this idea by combining rotation-base based smoothening with trellis-coded quantization further enhance performance.
In this work, we build upon the additive codebook quantization strategy adopted in AQLM [5], which represents weights as the sum of multiple centroid vectors. Compared to smoothening- based approaches that rely on matrix transformations such as rotations, additive codebook quantization offers more intuitive and inference-efficiee alternative by avoiding transformation overhead during runtime. This formulation enables fast lookup-basee inference and supports flexible trade-offs between accuracy, memory footprint, and latency through careful control of hyperparameters such as the number of codebooks and vector length.
Codebook based quantization reduces memory footprint by representing the original weights as set vectors and approximating each individual vector with a corresponding centroid vector. As illustrated in Figure 2, weight matrix of size (M x K) is quantized into codebooks and corresponding codes using several key hyperparameters, including the vector length (w), group size (g), the number of bits per code (b), and the number of codebooks (m). In the first step, the parameter specifies

---

the granularity at which the (M x K) weight matrix is partitioned into vectors. Each vector is then normalized by scaling factor to facilitate efficient clustering of weight vectors (Step normalization group size 9. which is always greater than or equal to u, defines the range over which normalization is performed, ensuring consistent scaling across all vector elements. Specifically, given that the length of each vector v, a total of g/v weight vectors are treated as single group for normalization purposes.
The
After normalization, vectors are clustered and mapped to centroid values, which serve as representative values for each cluster (Step A widely used approach for determining these centroid vectors is the K-means clustering algorithm, which partitions the data into predefined number of clusters [5, 27, 14]. The algorithm iteratively initializes cluster centroids and assigns each data point to the nearest cluster based on a chosen distance metric, repeating this process until convergence is achieved.
code. The number of clusters is set to 29, where denotes the number of bits per code, allowing representation of values ranging from to 2 - 1. A codebook that maps each code to its corresponding centroid vector is constructed (Step , and original weight values can subsequently be reconstructed by decoding the codes using this codebook. To minimize quantization error, multi-codebbok concept has been proposed [5, 2], which performs the quantization process times, constructing m codebooks and summing their values to represent the original weight value more accurately.
Each centroid vector is assigned unique index, referred
Since applying codebook- based quantization makes sized full full-precision values to be represented by m different codes b bit precision, the average per weight decided by these hyperparameters. Specifically, the number of bits per weight is calculated by dividing the total quantized bits by the number of weights. The total bit size, and consequently, the average bit per weight q, can be expressed as follows:
codebook + Scode + Snorm M . K
16 m 2 U + b. m M K + 16
M
K g
M.K
where the precision of each vector element in the codebook is set to FP16. Equation also reveals that multiple combinations of hyperparameters can achieve the same average bit per weight. For example, as shown Table 1, various combinations including (v, m, b. g) = (4, 1, 8, -1) and (v, m, g) (16, 8, 32) result in almost the same average bit per weight of 2. Since our goal is to maintain model accuracy under extremely low-bit quantization, achieving an optimal trade-off between bit precision and accuracy is essential. However, the impact of these hyperparameters accuracy and inference latency remains largely unexplored, necessitating extensive investigation through comprehensive experimental studies. In this paper, we analyze how different hyperparameter combinations with similar average bits per weight influence the trade-offs between memory footprint and performance, providing valuable insights into optimizing quantized models.
model
Table I: Average bits per weight for various quantization configurations. Gcode lcodebook. and Inorm represent the average bits allocated to codes, codebooks, and group normalization factors, respectively. A value of indicates row-wise group normalization.
m
b
Gende
Gendehook
fuorm
q
1
8
-1
2.0
0.001
0.004
2.005
8
2
8
-1
2.0
0.004
0.004
2.008
16
4
8
-1
2.0
0.016
0.004
2.020
8
1
8
16
1.0
0.002
1.000
2.002
16
3
8
32
1.5
0.012
0.500
2.012
2.3 Kernels for Quantized LIM
Recent weight-only quantization techniques, aimed at reducing bit-widths to sub-4-bit levels to en- hance memory efficiency, are also focused on achieving practical speedups. For instance, GPTQ [6] and AWQ [13] provide INT3-FP and INT4-FP kernels for uniform quantization, respectively, along- side their quantization methodologies. While these kernels achieve reduced latency relative to the

---

FPI6 cuBLAS baseline, they fall short of improving computational efficiency. This limitation arises as the benefits primarily stem from more efficient data movement facilitated by quantization, rather than from the computational performance itself. To address this issue, LUT-GEMM [20] introduces a kernel that reduces computational complexity by utilizing a lookup table, leveraging the BCQ format to represent quantized weights.
Especially in codebook-base quantization, the current kernel [5] relies on dequantization process. This process reconstructs the original weights by using the codes to retrieve the corresponding centroids from the codebook. These codes serve as indices for querying the codebook, where the corresponding centroids are retrieved and used as dequantized weights. The (M x K/u) code matrix retrieves centroid vectors of length v, reconstructing a dequantized matrix of the same shape as the original matrix (M x K). Subsequent matrix multiplication with the input matrix is then performed in the same manner as with the original weight matrix. To support efficient dequantization, current kernels load the entire codebook in programmable cache to enable rapid retrieval of centroid vectors. However, this approach leads to significant memory write overhead during the codebook load phase. Furthermore, as the codebook size increases, loading the entire codebook in programmable cache becomes more challenging. For instance, in the AQLM (1 X 16) kernel the codebook requires 2 centroids of length v 8, each represented in FP16. This requires 2 x x 8 x 2 bytes (=IMB) of shared memory, far exceeding the capacity of both A100 (164KB) and H100 (224KB). As result, the codebook cannot reside in shared memory and must be repeatedly fetched from DRAM, significantly increasing latency. Furthermore, while this dequantization approach effectively optimizes data movement, it does not reduce computational complexity, which remains identical to that of the original matrix multiplication.
3 CodeGEMM: Codebook-based GEMM
Motivation. Figure illustrates the matrix multiplication process in codehook- quantization and highlights two key challenges. First, loading the entire codebook into programmable cache incurs substantial overhead that scales with the size of the codebook and is repeated by each thread block, leading to performance degradation in large-scale deployments. Second, computations are often redundant, as each input vector interacts with limited set of centroids, resulting in repeated calculations across the output matrix. To address these challenges, we propose method that minimizes codebook load overhead and avoids redundant computation, enabling more efficient use of limited on-chip cache and improving overall efficiency.
Methodology. CodeGEMM introduces a codebook- approach to efficient matrix multiplica- tion in quantized LLMs. Instead of loading the entire codebook, CodeGEMM stores precomputed results of the inner products between the centroid vectors and input activations in the programmable cache. This approach reduces space complexity and eliminates the need for dequantization operations that retrieve centroids for each code from the codebook. Instead, the kernel repeatedly utilizes the precomputed results, significantly reducing the computational complexity.
Figure 3 illustrates the computation process of CodeGEMM, depicting a matrix multiplication operation between a weight tile of dimensions (t), x tw) and an input tile of dimensions (tw x 1). Initially, the input tile partitioned into inputs of dimensions (tw/ /v x v X 1) to facilitate efficient computation with centroids stored in the codebook (Step
These segmented inputs are represented and k E 1. V 1]. The corresponding code matrix is similarly partitioned into code tiles of dimensions (t,, x f (w//v) to align with the input tiles for effective computation. In our implementation, we set t. 32 and t, = 2048 to align with hardware-friende tiling strategies and to maximize reuse of the Psumbook within each thread block.
as of. = Foxj+h, where j E [0,
lww/v
Next, each segmented input computes inner products with the centroids to generate partial sums (Psums), which are then stored as a Psumbook the programmable cache (Step Eaac entry in the Psumbook, p/,, is calculated as follows:
p -
-
cl K=
x
iE 1.......,
2 1]
(2)
By storing the Psumbook the programmable cache instead of the full codebook, CodeGEMM replaces the conventional process of fetching and computing with centroids for each code with more
5

---

Step Input reshape
Step
Build Psumbook
Step Data retrieval & Accumulation
Programmable Cache
E0
Codo Sie
Codebook (FP18.
(NTTZ, & 4.-AV
Reshape
Programmable Cache
Store
Retrioval
Imput Ho
(FP16, 1,/6,
Input tie (FP16 1)
Paumbook /FP16,
Psumbook
Buch
(FP18,
Cuiputt
Partiel sum (FP116, & (a/
Figure 3: Overview of the CodeGEMM kernel operation for codebook-based quantized models. 1) Input data is reshaped into vectors to align with the codebook dimensions. Precomputed inner products between the codebook and input vectors are stored in the Psumbook within the programmable cache, significantly reducing computational overhead. During computation, codes query the corresponding partial sums from the Psumbook, which are then accumulated to generate the output efficiently without requiring on-the-fly dequantization.
efficient retrieval of precomputed Psums using code as a key. This only reduces computational complexity but also lowers space complexity, as only the scalar inner product results are cached rather than the full centroid vectors. Finally, the partial sums corresponding to each input are retrieved from the Psumbook via code-based indexing and then accumulated to generate the final output activations for the thread block (Step In contrast to existing kernels that load the entire codebook into the programmable cache, our approach of storing precomputed inner product results allows for significantly more efficient computations for codebook-based quantized LLMs.
Computational complexity. In a matrix multiplication operation with a weight matrix of size (M x K)) and an input matrix of size (K x N), the computational complexity of standard GEMM kernel is given as O(MNK) K). This complexity also applies to dequantization- based kernels, as they improve data movement efficiency through quantization but do not reduce the overall computation cost.
In contrast, CodeGEMM reduces the computational workload by precomputing all inner product results between the centroid vectors and input activations, storing them in a Psumbook, and replacing repeated computations with simple retrieval operations. This allows CodeGEMM achieve lower computational complexity compared to conventional methods. The computational complexity of CodeGEMM, assuming M > 2", expressed as follows:
K = O (m-2% v. .N +0 m M
-0(m
C = Obuild + Cread
N
=o coOmMNA
26 ++++
MNNK- m
where Cbviia and Cread represent the computational complexity of building the Psumbook and retrieving values from it, respectively. Consequently, CodeGEMM achieves computational reduction factor of (m/v) compared to conventional kernels, where m is the number of codebooks and v is the vector length-bott crucial for optimizing performance. This enables CodeGEMM enhance data movement efficiency through quantization, like traditional dequantization-bant kernels, while also reducing computational complexity for higher efficiency.
Space Complexity. Dequantization- based GEMM kernels load the entire codebook into the pro- grammable cache to perform computations. In this case, the space complexity is given by (mm-2h.)) which corresponds to the full size of the codebook. is O(m 2b lw//v), where L,, denotes the width of the weight tile. Since CodeGEMM stores pre- computed inner product results rather than full centroid vectors, its space complexity inversely proportional to the vector length v, resulting in smaller cache footprint. This reduction in memory
contrast, the space complexity of CodeGEMM

---

requirements allows CodeGEMM to achieve more efficient cache utilization, making it better suited for accelerators with limited programmable cache sizes.
4 Experiments
Setup. We evaluate CodeGEMM by exploring the trade-offs across key hyperparameters, focusing on three primary metries relevant to L.LM compression: memory footprint, lateney, and accuracy. The memory footprint is quantified using the average number of bits per weight q and computed according to Equation 1. Latency is measured to compare raw kernel performance for matrix multiplication, assuming the shape of linear layers used in Llama-3 [4], a widely adopted architecture. Specifically, latency is reported as the sum of kernel execution times for all linear layers in a single Transformer decoder block without layer fusion. All latency measurements are performed on an NVIDIA A100 80GB GPU. Throughput (or, equivalently, end-to-end latency) is additionally measured using the Llama implementation provided by the HuggingFace [29] library with layer fusion. Although this library is not optimized for high-throughput inference, it remains one of the most widely used frameworks and thus serves as a practical baseline. Accuracy is evaluated using the 1m-eval -harnss [7] benchmark suite across both zero-shot and 5-shot settings on standard tasks.
Memory Footprint vs. Latency. Table presents the kerne--eevel latency of various quantized matrix multiplication methods on 2-bit quantized Llama3 models (8B and 70B). LUTGEMM [20], de- signed for uniform quantization, achieves strong performance by eliminating redundant computations through efficient use of lookup tables. QulP# [25] and QTIP [26). which rely smoothing- -aaed transformations, mitigate the associated overhead through highly optimized fused kernels, demonstrat- ing competitive latency. AQLM [5] with the 2 x configuration improves data movement efficiency via quantization, resulting in moderate latency reduction despite retaining the same computational complexity as the FP 1 baseline. However, AQLM-I x 16 suffers from significantly higher latency due to inefficient dequantization caused by its large codebook (e.g., 216 entries), which exceeds on-chip cache capacity. In contrast, CodeGEMM achieves up to 2.18x and 1.64x speed-ups over the FP16 baseline and AQLM, respectively, at the same average bit precision. This improvement is due to both the use of precomputed inner products, which reduce computational complexity, and the efficient utilization of shared memory.
Figure 4(a) shows the relationship between memory footprint and latency of the Llama-3. 3-8B model under a single batch operation. As shown in Table 1, codebook- based quantization enables diverse configurations within a given memory footprint by adjusting hyperparameter settings. According to Equation 1, the group size g impacts the memory footprint. Smaller group sizes increase the total memory footprint, leading higher latency in memory-bount operations. For g 2 32, the fine-grained group normalization incurs minimal latency overhead despite the growing memory footprint. However, as g decreases further, the overhead becomes more pronounced, particularly in per-vector normalization (i.e., g = v), where the latency rises sharply. Additionally, this overhead amplified when the group normalization scale factor constitutes larger proportion of the total memory footprint, which is more likely when the number of codebooks m is small.
Table 2: Kernel-level latency (us) of quantized matrix multiplication in 2-bit quantized Llama3 models (8B and 70B). CodeGEMM achieves consistently lower latency than other methods.
cuBLAS (fp16)
LUTGEMM (q2-g128)
QulP# (c8p)
QTIP
AQLM (Ix16)
AQLM (2x8)
CodeGEMM (m2v8g128)
CodeGEMM (mIv4g128)
(r2)
8B
332.45
160.1 299.9
162.63
189.94
645.51
250.12
172.18 373.49
152.69
70B
1111.36
403.59
477.04
2285.5
674.67
293.82
Efficiency and Utilization. We measured DRAM traffic proxies and power efficiency using nvidia-smi [18] telemetry. Metrics were sampled 100 ms cadence over 10s window and averaged across trials. Memory utilization represents the fraction of the sampling interval during which device (global) memory was actively read from or written to [18]. As summarized Table 3, on a matrix multiplication workload with M=1, 286677,, and K-8192, CodeGEMM delivers sub- stantially higher compute efficiency (GFL.OP//) than dequantization-base kernels. Parenthetical values denote two-sigma error margins over 128 samples. Beyond lower latency, CodeGEMM also
7

---

shows higher energy efficiency and improved -umnorr--ubbyseen utilization, indicating reduced and more structured DRAM access relative dequantization- based approaches.
Table 3: Kernel-level Performance evaluation on GEMV with (M, N. K) = (1, 28672, 8192).
Performance metrics are obtained from nvidia-smi telemetry. CodeGEMM delivers higher power efficiency and improved memory utilization than dequantization-bant codebook kernels.
Method
TFLOPS
Power (W)
GFLOPS/W
GPU Util (%)
Mem Util (%)
cuBLAS
1,58
318.55 (#622)
4,95
96.87 (=0773)
96.94 (=:0.48)
AQLM- 166
0.75
126.54 (40.4)
5.93
99.00 (=0000)
6,00 (=0.00)
AQLM-2x8
2.59
254.20 (42447
10.18
92.84 (=1.5)
19.96 (23.39
CodeGGMMM-m22&81223
5.43
304.69 (16.1)
17.83
85.32 (.58)
43.75 (210.95
6.12
316.38 (18.37)
19.36
84.47 (=2.28)
49.80 (21))))
Memory Footprint vs. Accuracy. Figure 4(b) shows the relationship between memory footprint and accuracy of quantized models across various hyperparameter configurations. Perplexity, measured using the WikiText-2 dataset, is used to evaluate accuracy. CodeGEMM builds block-wise code- book optimization [5] and incorporates fine- rree-graned group normalization for enhanced performance. As average bits per weight increase, models demonstrate greater representational capacity, resulting in lower perplexity. Fine-grained group normalization reduces quantization error, but it increases memory footprint as the normalization factor grows. Under row-wise group normalization (g 1), increasing the number of codebooks (m) while keeping the average bit precision fixed leads to improved accuracy, as the model can better approximate the original weights with additive codebooks. However, as g becomes smaller (i.e., more fine-grained), this accuracy gain from increasing m diminishes, and models tend exhibit similar performance for given q, regardless of the number of codebooks.
0.35
20
CUBLAS (FP16)
WikiText-2 perplexity
Og 88 Og Vg= 128 g=
AQLM (2x8)
0g
09 Og
32 Vg 128
15
m1v4
m2v4
Latency
m2v8
m3v8
0.20
m4v8
m4v16
10
LV
miv4
m2v4
0.15
m2v8
m3v8
m4v8
m4v16
FP16 Bassllne Dy
0.10
2
Bits per weight (a) Latency vs. Memory footprint
Bits per weight
(b) Accuracy vs. Memory Footprint
Figure 4: Latency and accuracy trade-offs for the Llama-3.1-88 model under various configurations.
Throughput vs. Accuracy. While many studies on quantization have focused on the trade-off between memory footprint and accuracy to achieve better performance, the relationship between throughput (or, equivalently, decode- phase latency) and accuracy has often been overlooked. However, in real-world applications and service-level deployments, latency is often a critical constraint that directly impacts user experience and system efficiency.
Table 4 compares the accuracy of CodeGEMM against both uniform and codebook- based quantization methods. We include FlexRound [12] as a representative uniform quantization baseline and AQLM [5] for codebook-basee
quantization. To further improve accuracy, we also apply PV-Tuning [16],
recently proposed post-quantization calibration method for codebook- models. Figure 5 reports the resulting accuracy-throughyee trade-off's. Throughput for FlexRound is measured using LUT- GEMM [20], a state-of-the-aa art kernel optimized for uniformly quantized models. While Round demonstrates strong throughput at 2- precision, suffers from the worst accuracy among all methods, highlighting the limitations of uniform quantization in extremely low-bit settings. The recent rector-quantzzation approach VPTQ [14] achieves an average accuracy of 57.98, slightly higher than CodeGEMM without PV-Tuning. However, its kernel is implemented as straightforward dequantize-then muuliilyy pipeline without operator fusion, which introduces dequantization overhead and leads to lower throughput than FP16.
8

---

AQLM, with its dequantization based kernel, achieves notably higher accuracy in such settings but exhibits poor throughput due the overhead of repeatedly loading large codebooks. Specifically, AQLM (1 x 16) achieves the best accuracy under 2-bit memory budget by using large codebook (b= 16), but this configuration exceeds the capacity of the programmable cache. As a result, it incurs inefficient memory access and dequantization latency, leading to a suboptimal throughput-accure trade-off. AQLM (2x8) improves throughput over the FP16 baseline, but only marginally, because it still pays the dequantization cost and maintains similar computational complexity. In contrast, CodeGEMM delivers significantly better throughput while maintaining competitive accuracy. More- over, it shows strong synergy with PV-Tuning, achieving results comparable or better than AQLM with less computational overhead. Between different CodeGEMM configurations, the m1v4 variant consistently outperforms m2v8 in both throughput and accuracy, even though both configurations maintain a similar average bit precision. This aligns with the kernel-level latency trend observed in Figure 4(a), while diverging slightly from the perplexity trend in Figure 4(b), suggesting that end-to-end throughput-accural
behavior can differ from perplexity metrics. Overall, CodeGEMM achieves 1.83x end-to-end speedup over dequantization-basee kernels comparable accuracy (Figure 5())).
Table 4: Accuracy various quantization methods on the Llama-3.1-88 3-IBB-IIntttcct model. The Average column represents the mean accuracy across all tasks, including MMLU (5-shot) and and O-shot tasks such WinoGrande (WG), HellaSwag (HS), ARC-Easy (ARC-E), and ARC-Challenge (ARC-C).
Method
tok/s
MMLU
WG
HI
ARC-E
ARC-C
Avg.
FP16
16.000
103.8
68.39
73.95
79.2
79.63
55.03
71.26
1eesRound-g2g128 [12]
2.125
205.3
24.27
55.16
43.78
24.75
24.57
41.65
AQLM-2x8 [5]
2.005
124.5
42.29
58.25
61.40
46.25
30.89
47.82
+PV-Tuning
2,005
124.5
55.13
69.14
72.43
71.25
45.73
62.74
AQLM-Ix16 [5]
2,213
49.0
58.74
68.75
70.21
73.99
46.16
63.57
+PV-Tuning
2.213
49.0
60.72
70.24
74.33
74.92
48.89
65.82
CodeGEMM-m1144g12
2.126
228.3
45.16
58.96
63.07
63.97
38.48
53,93
+PV-Tuning
2.126
228.3
57.42
69.06
73.85
73.15
46.33
63.96
CudeGFMM-m22+8g123
2.127
214.4
42.83
60.54
62.84
60.14
37.03
52.67
+PV-Tuning
2.127
214.4
55.42
68.75
73.02
73.91
47.70
63.76
Scaling to Larger Models. To evaluate the scalability of CodeGEMM for larger language mod- els, we assess its performance on the Llama-3. 1-70B model under various quantization configura- tions. Table 5 compares CodeGEMM with uniform quantization methods such as GPTQ [6] and FlexRound [12], as well as the codebook-basee to 70B, latency improvements relative the AQLM become more pronounced. While the overall performance trends are consistent with those observed in the 8B model, we observe that (1 x 16) suffers from significantly degraded throughput 70B due the large codebook size (e.g.. 226 entries), which exceeds shared memory limits and introduces excessive dequantization overhead. In contrast, CodeGEMM achieves favorable trade-off by leveraging fine-grained group normalization to im- prove accuracy with minimal increases in memory footprint and latency. The benefit of fine-grained normalization is evident in the widening accuracy gap between CodeGEMM and AQLM (2x x8), which uses row-wise normalization, at 70B relative to 8B. Consequently, CodeGEMM matches the accuracy of AQLM (1 x 16) while delivering an 8.93x throughput ad advantage (Figure 5(b).. These results demonstrate that CodeGEMM scales effectively to large models while maintaining competitive performance.
method AQLM. As model size increases from 8B
5 Related Work
Look-Up Table-Based Computation. Several prior works have leveraged look-up tables (LUTss improve computational efficiency, both the kernel level [20, 15] and in specialized hardware de- signs [28, 17, 19]. However, these approaches primarily support uniform or binary-coded quantization non-uniform quantization.
formats, and do not extend to codebook-ban
contrast, CodeGEMM designed to naturally support codebook-bas based quantization while also being generalizable to uniform and binary formats through appropriate codebook configurations. Specifically, uniform and binary quantization can be accommodated by defining centroids e {0, for uniform quantization and

---

Table 5: Accuracy of various quantization methods on the Llama-3.1-70B model.
Method
4
tok/s
MMLU
WG
HS
ARC-E
ARC-C
Avg.
FP16
16.000
0OM
78.58
79.64
85.03
86.66
64.93
78.97
GPTQ-92g128
2.125
41.7
26.35
53.04
49.04
48.95
29.52
41.38
FlexRoun-gQ2g128
2.125
41.7
26.70
53.59
50.67
25.13
24.83
36.58
AQLM-2x8
2.002
19.0
61.45
59.59
52.83
48.82
28.67
50.27
AQLM-Ix16
2.055
5.5
73.07
76.16
80.83
82.20
57.17
73.89
CodeGEMMM-m1v4g122
2.125
51.2
68.15
74.90
75.37
79.42
52.73
70.11
CodeGEMMM-m1v4g32
2.500
49.1
71.21
76.64
79.43
82.41
56.06
73.15
FlexRound AQLM-2x8 77 AQLM-1x16
GPTQ FlexRound AQLM-2x8 AQLM-1x16
* CodeGEMMM-m1v4g22 CodeGEMM-m2vvg122 128
* CodeGEMMM4m1v41222 * CodeGEMMM-m1w4g32
80
80
70
Accuracy (%)
60
Accuracy
8.93x Speed up
1.83x Speed up
40
50
20
60
120
180
240
20
40
60
Tokens per second (a) Llama-3.1-8B
Tokens per second
(b) Llama-3. 1-70B
Figure 5: Latency and accuracy trade-offss for the Llama-3.1-4 1--BB model under various configurations.
CE {-1 for binary quantization. This generality makes CodeGEMM particularly suitable for deployment as fixed-function function ASIC kernel, as it enables support for diverse quantization formats under a unified hardware architecture.
Codebook-based Quantization. Beyond additive codebooks, recent work has explored highly structured designs that integrate rotation with the codebook itself. QulP# [25] and QTIP [26] both pair their lattice- and trellis-coded codebooks with an inference-time smoothening rotation, and each work supplies carefully hand-tuned kernel that fuses the rotation with subsequent look-ups. In contrast, the additive codebooks we target require only a lightweight table lookup and accumulation without rotation, dequantization remains simple and fast. This design lets CodeGEMM deliver competitive 2-bit accuracy while retaining single, reusable kernel that scales across broad range of quantization settings.
Conclusion
Summary. This paper presents CodeGEMM, an efficient matrix multiplication method for codebook-based quantized models. Instead of loading the full codebook into programmable cache, CodeGEMM precomputes and stores results in a Psumbook, reducing both space and computational complexity. Experiments show that CodeGEMM outperforms state-of-the-art kernels latency while maintaining high accuracy in extremely low-bit quantization.
Limitations. CodeGEMM requires the Psumbook to reside in on-chip shared memory, which constrains the usable codebook size. Very large codebooks (c.g., b=16 with 216 entries) exceed current GPU cache limits. In practice, we therefore fix the per-codebook width to 6-8 and recover accuracy via fine-grained group normalization. which adds negligible latency and yields favorable -efuraccyefficiency trade-offs under realistic hardware constraints. A second limitation is large- batch throughput: CodeGEMM underperforms compared Tensor Core- based cuBLAS when the batch size is large (e.g., . >23).. This behavior is common to CUDA Core-based quantized GEMM kemnlss and reflects current commercial GPU architecture rather than an inefficiency itself. Overall, while CodeGEMM excludes extremely large codebooks and is not optimized for large batches, its hardware-ale algorrthhm co-design addresses key inefficiencies of traditional codebook quantization, reducing both computational and space complexity.
the algorithm
10

---

References
[1] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms, arXiv preprint AxXiv:2404.004500 2024.
[2] Artem Babenko and Victor Lempitsky. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 931-938, 2014.
[3] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[4] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[5] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.061 18, 2024.
[6] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprinl arXiv:2210. 2022.
[7] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite. Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
[8] Jung Hwan Hco, Jeonghoon Kim, Beomseok Kwon, Bycongwook Kim, Se Jung Kwon, and Dongsoo Lee. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. arXiv preprint arXiv:2309./553 2023.
[9] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Training compute-optimat large language models. arXiv preprint arXiv:2203. 15556, 2022.
[10] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv22001.08661. 2020.
[11] Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. Finequant: Unlocking efficiency with fine-grained weight-only quantization for 1lms, arXiv preprint arXiv:2308. 09723, 2023.
[12] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, pages 18913-18933 PMLR, 2023.
[13] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awe aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[14] Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. arXiv preprint arXiv:2409. (aee::22099770000 /7066, 2024.
[15] Saeed Maleki. Look-up mai gemm: Increasing ai gemms performance by nearly 2.5 via msgemm. arXiv preprint 2rX.v:2310.06178 2023.

---

[16] Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight- estimation for extreme IIm compression. arXiv preprint 205 2024.
[17] Zhiwen Mo, Lei Wang. Jianyu Wei, Zhichen Zeng, Shijie Cao, Lingxiao Ma, Naifeng Jing, Ting Cao, Jilong Xue, Fan Yang, al. Lut tensor core: Lookup table enables efficient low-bit Ilm inference acceleration. arXiv preprint arXiv:2408. 06000,, 2024.
[18] NVIDIA Corporation. Nvidia system management interface (nvidia-smi) documentation. https://docs aadii//iiai...iiiaa....aa www.co//////c deploy/pplyy//oo heppoy/nvidii....aeem /nwpdiiiiaaaaaiii/ smi/index. html, 2025, Accessed: 2025- 10-22.
[19] Gunho Park, Hyeokjun Kwon, Jiwoo Kim, Jeongin Bac, Baeseong Park, Dongsoo Lee, and Youngjoo Lee. Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 1098-1111. IEEE, 2025.
[20] Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Bycongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models, arXiv preprint arXiv:2206.0955 2024.
[21] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprint 130.::2.30..33137, 2023.
[22] Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. /1796, 2024.
[23] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024.
[24] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
[25] Albert Tseng. Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXtv:2402.043396 2024.
[26] Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37:59597-59620. 2024.
[27] Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization. arXiv preprint arXiv: 15319, 2024.
[28] Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. T-mac: Cpu renaissance via table lookup for low-bit Ilm deployment on edge. In Proceedings of the Twentieth European Conference on Computer Systems, pages 278-292, 2025.
[29] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: -tate-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, October 2020. Association for Computational Linguistics.
[30] Haoran You, Yipin Guo, Yichao Fu, Wei Zhou, Huihong Shi, Xiaofan Zhang, Souvik Kundu, Amir Yazdanbakhsh, and Yingyan Celine Lin. Shiftaddllm: Accelerating pretrained lmms via post-training multiplication-les reparameterization. arXiv preprint arXiv:2406. 05981, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly made the main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
All theoretical results are accompanied by their full assumptions and complete
proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All key implementation and configuration details are provided to ensure reproducibility of the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and data are publicly available, with clear instructions for reproducing the main results provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
httgg//ddd//oeeeuumnnmmmm cesiiissssiiiiyyiii for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //tips.c/pppii pcc/public/guutc gewgge/ideeeeeeeoo /CodeSubmi ssisss ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant training and evaluation details are clearly specified in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are omitted because the observed variance was minimal and had no impact on the conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The type of hardware and execution time are specified for all experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eeerrpp.eeeee cc/public/ Ceticc//Ehiittiiiiee chhuidelieeeees
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics and does not raise ethical concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper addresses low-level efficiency improvements for LLM inference and is not directly tied to applications with societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper presents infrastructuur-llvell optimiaaions without releasing any high-risk models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all external code and datasets used in the paper, and ensure their licenses and usage terms are properly followed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswithcode com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code introduced in the paper is publicly released with clear documen- tation and usage instructions.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: human subjects or crowdsourcing were involved this study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects were involved in this study.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This work does not involve LLMs in the core methodology or contributions.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp://eee...ppeee oc/Confferences/22222 for what should or should not be described.
A
Appendix
A.1 Psumbook Build vs. Read Breakdown
We quantify the relative cost of constructing and consuming the Psumbook by isolating execution to single SM and splitting runtime into two phases: building the Psumbook for each tile and reading during the main accumulate loop. We sweep tile width tu and batch size M while fixing (N, K) indicated, and report the percentage of execution cycles devoted to each phase for two kernel variants, m2v8 and m1v4.
Across settings, the build/read split is stable with respect M at a fixed two which indicates that Psumbook construction amortizes across the batch. At a fixed M, larger to increases the build share for small matrices and decreases it for large matrices. Typical ranges are 288 build and ~54-72% read for m2v8, and 20 42% build and ~58- 80%% read for m1v4.
Table 6: Cycle share (%) spent building reading the Psumbook under varying ty and M.
M
N
K
Lu
Psumbook Phase (%)
Proposed m2v8
Proposed m1v4
4096
4096
32
Building Reading
30.5 69.5
20,3 79.7
4096
4096
64
Building Reading
33.0/67.0
28.5
4096
4096
128
Building Reading
31.2/68 68.8
30.7 69.3
8192
8192
32
Building Reading
45.4/
41.2/58.8
8192
8192
64
Building Reading
45.6 54.4
39.7 / 60.3
8192
8192
128
Building Reading
28.3 71..7
29.5 70.5
4096
Building Reading
30.4/ 69.6 /
20.7 79.3
4096
4096
Building Reading
20.4 79.6
8192
8192
Building Reading
45.7 54.3
41.3 58.7
8192
8192
32
Building Reading
53.9
41.6/58.
A.2 Tile Size Sensitivity
We revisited our heuristic choices for the tile dimensions and conducted a systematic sweep over fw (32, 64, 128} and fz, {2048, 4096] across representative shapes. We find that ty, =2048 consistently yields the best performance over broad set of workloads, supporting our original choice. For horizontal tile, smaller values such as L,,, =2 work well for relatively small matrices, whereas -4 tends to perform better on large matrices. We attribute this coarser tiling reducing kernel-launch overhead and improving partial-sum reduction efficiency scale. Table 7 summarizes representative
19

---

results for (N, K) E 4006), (8192, 8192)} at M-1. Overall, these observations justify our default choice (tm: (b) (32, 2048) for small and medium shapes, with a.- =64 preferred matrices grow.
Table 7: Effect of tile dimensions on end-to-end latency (18).
M
K
tw
Proposed m2v8
Proposed v4
4096
4096
32
2048
26.57
25.07
4096
4096
64
2048
26.76
25.40
4096
4096
128
2048
29.61
26.81
4096
4096
32
4096
28.95
27.60
4096
4096
64
4096
28.49
27.68
4096
4096
128
4096
37.58
32.87
8192
8192
32
2048
39.04
36.02
8192
8192
64
2048
37.23
35.33
8192
8192
128
2048
40.09
38.54
8192
8192
32
4096
37.78
36.17
8192
8192
64
4096
38.29
37.70
8192
8192
128
4096
45.40
42.75
A.3 Effect of Higher Bit Precision
We additionally measured latency for higher average bit precisions using the kernel configura- tion (g=128, b=8, f. =32, tn 20488.. For reference, FP16 cuBLAS latency is included. As
expected, increasing the effective bits per weight (via larger numbers of codebooks m smaller vector length v) generally raises latency, and the trend is more pronounced for larger matrices (c.g.. M=1, -=922, Even at higher bit precisions, CodeGEMM remains competitive with FP16 baselines while offering a flexible accuracy efficiecyf trade-off through the (m, v) configuration.
Table 8: Matrix multiplication latency (μs) for higher effective bit precisions under 1g=128, b=8, to 32, tA 2048). FP16 cuBLAS is shown for reference.
M
N
K
m
bit
lateney
4096
4096
N/A
N/A
16,000
28.118
4096
4096
2.126
25.074
4096
4096
2
4
4.127
27.009
4096
4096
I
8
1.127
24.015
4096
4096
2
8
2.129
26.574
4096
4096
8
3.126
27.385
4096
4096
8
4.127
29.797
8192
8192
N/A
N/A
16.000
95.785
8192
8192
4
2.125
36.020
8192
8192
2
4
4.125
49.636
8192
8192
1
8
1.125
31.883
8192
8192
2
8
2.126
39,040
8192
8192
3
8
3.126
47,210
8192
8192
4
8
4.127
58.364
A.4 Batch-Size Sensitivity and Fair cuBLAS Accounting
For fair comparison, we include the dequantization stage when reporting FP16 CuBLAS latency, since dequantization is required to precede GEMM in codebook-basee pipelines. Under this accounting, CodeGEMM remains competitive even at batch sizes of and 16. In data-center deployments, continuous batching can aggregate requests and increase the effective batch size during decoding; yet, many scenarios still operate with small batches (e.g., on-device inference). The batch-size sensitivity observed in CUDA-core quantized kernels reflects architectural constraints such as occupancy limits and shared-man memoryy bandwidth. This limitation is shared by recent methods (e.g., QulP# and QTIP) and does not indicate algorithmic inefficiency. Table reports linear latency Llama-3-8B.
20

---

Table 9: Aggregate latency of linear layers (uU) within a Llama-3-8B decoder block as function of
batch size.
cuBLAS Dequant
AQLM (Ix16)
AQLM (2x8)
QUTP# (e8p)
QTIP
Proposed (m2v8)
Proposed (mlv4)
BS
cuBLAS
Dequant
(12)
332
1027
1360
646
250
163
190
172
4
333
1027
1361
2373
794
445
550
491
405
8
336
1027
1364
4695
1515
818
1034
909
744
16
340
1027
1367
9267
2959
1554
1991
1748
1416
A.5
Additional Benchmarks Across Problem Sizes
We expanded our evaluation to broad sweep of matrix shapes N,) Latency is measured end to end on a fixed hardware and software stack, with all kernels compiled under identical toolchains. Table 10 reports the full results. cuBLAS runs Tensor Cores and tends to maintain relatively stable latency as the batch size M grows, whereas recently proposed quantized ker- nels-including eurs-execute
CUDA cores and often show increased latency with larger M. Within this CUDA-core class, CodeGEMM consistently performs well on large matrices, where arithmetic intensity and memory diverse (M, N, K) settings and is particularly effective for large-scale matrix multiplications that dominate LLM inference.
high. In practice, the method remains competitive across
Table 10: Kernel latency (uI) across diverse (M,N,K) configurations.
AQLM (Ix16)
AQLM
Proposed (m2v8)
Proposed (mlv4)
QUIP (c8p)
QТЛР
M
N
K
cuBLAS
(2x8)
(r2)
2048
2048
19.82
28.84
20.55
20.75
20.66
19.47
19.44
4
2048
2048
19.99
74.67
43.31
44.04
41.92
36.71
36.00
8
2048
2048
19.79
135.36
73.03
75.18
69.72
59.44
57.87
8192
2048
30.57
28.84
28.83
25.94
26.70
25.52
27.08
4
8192
2048
31.31
74.67
76.15
63.97
65.36
60.70
66.18
8192
2048
31.70
135.36
138.09
115.39
116.11
107.85
118.99
2048
8192
27.52
60.47
30.93
24.28
23.81
23.44
24.90
2048
8192
29.82
203.86
82.18
56.21
$2.57
51.91
59.03
8
2048
8192
28.69
396.44
149.98
98.92
90.73
89.91
103.24
4096
4096
28.00
63.13
32.28
24.76
24.97
23,96
26.74
4096
4096
28.54
210.03
89.76
60,58
$7.79
$3,92
62.74
8
4096
4096
28.11
396.37
165.49
108.16
103.92
93.43
110.84
14336
4096
88.67
168.12 12
64.76
38.85
37.51
38.91
$1,30
4
14336
4096
89.08
632.69
217.68
111.20
106.90
113.28
161.23
8
14336
4096
89.29
1252.55
422.89
211.37
196.68
212.55
308.37
4096
14336
86.31
169.31
58.70
36.15
33.92
37.27
43.85
4096
14336
86.51
635.74
193.41
103.15
92.61
106.63
133.36
4096
14336
86.49
1253.11
372.97
192.63
170.16
199.31
252.12
8192
8192
96.40
188.91
62.50
37.99
35.45
38.31
49.86
8192
8192
100.41
713.24
208.11
111.00
98.66
111.08
157.26
8
8192
8192
95.45
1408.68
402.29
207.73
184,25
208.29
299.24
28672
8192
297.74
625.53
181.54
86.48
76.71
101.98
134.03
28672
8192
303.10
2462.88
684.92
305.47
264.31
366.74
492.14
8
28672
8192
295.11
4913.52
1355.70
597.22
514.85
718.13
970.35
8192
28672
302.42
618.61
180.38
86.20
76.50
101.13
124,90
4
8192
28672
292.59
2437.82
679.24
305.14
263.70
361.95
455.84
8192
28672
293.69
4860.85
1344.49
596.63
515.12
710.94
897.41
21


Reviews:
[{"review_id": "pSB5EpYKFM", "reviewer": "Reviewer_2GP4", "summary": "The paper proposes a codebook-based quantization method to represent weights using 2-bit configurations. Unlike previous approaches that stored the entire codebook in memory for dequantization, the proposed method stores the product-sum results, eliminating the need for dequantization. Hyperparameters for codebook selection such as group sizes, vector length, and number of codebooks are explored to find the best trade-off between performance accuracy and hardware complexity. The new approach achieves a 2.29× speed improvement compared to previous codebook-based quantization methods such as AQLM (2×8).", "questions": "Is there any limitation on the size of the product-sum values stored in memory for large language models such as LLaMA 3.1 with 405B parameters?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1- The new approach is explored across various hardware constraints such as memory footprint, latency, and throughput, and it demonstrates the benefits of product-sum codebook-based quantization compared to previous approaches.\n\n2- The paper is well-written, and the experiments are elaborated in detail.\n\nWeaknesses:\n\n1- The accuracy degradation of the codebook-based approach is high compared to the FP16 approach in Table 3 (approximately 7%). The author is advised to elaborate on this accuracy degradation.\n\n2- It is suggested to include accuracy vs. latency and accuracy vs. throughput plots corresponding to Table 3 and Table 4.\n\n3- It is recommended that the comparison with these previous approaches [1, 2] be demonstrated in the paper.\n\n[1] Liu, Yifei, et al. \"Vptq: Extreme low-bit vector post-training quantization for large language models.\" arXiv preprint arXiv:2409.17066 (2024).\n \n[2] Liu, Zechun, et al. \"ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization.\" arXiv preprint arXiv:2502.02631 (2025).", "was_revised": true, "final_justification": "The author provides new experiments to compare with the previous work and also addressed my question regarding the limitations of the new approach for large language models such LLaMA 3. Therefore, I raised my score.", "rebuttal": "## Weaknesses\n\n> 1- The accuracy degradation of the codebook-based approach is high compared to the FP16 approach in Table 3 (approximately 7%). The author is advised to elaborate on this accuracy degradation.\n\n- Thank you for the insightful feedback. We agree with your observation. In the context of LLM post-training quantization, 2-bit quantization still leads to notable accuracy degradation, especially under extreme compression. While advanced techniques such as PV-Tuning help mitigate this issue to some extent, a performance gap remains. We believe that with sufficient resources, quantization-aware training could be a promising direction to further narrow this gap.\n- Our primary goal with CodeGEMM is to serve as a practical kernel-level bridge between model compression and real-world speedup. We are optimistic that as better quantization algorithms continue to emerge, CodeGEMM will enable them to realize their full latency-throughput benefits on modern hardware.\n\n> 2- It is suggested to include accuracy vs. latency and accuracy vs. throughput plots corresponding to Table 3 and Table 4.\n- We agree that figures are better suited than tables for illustrating trade-offs. Although we are unable to modify the manuscript during the rebuttal phase, we will include these plots in the camera-ready version, if accepted. We believe the visual representation will clearly highlight the accuracy-efficiency Pareto front achieved by CodeGEMM under different quantization settings.\n\n> 3- It is recommended that the comparison with these previous approaches [1, 2] be demonstrated in the paper.\n- Thank you for the helpful suggestion. While the reviewer recommended evaluations on both ParetoQ and VPTQ, we were only able to conduct experiments with VPTQ due to the lack of publicly available LLaMA 3 checkpoints for ParetoQ. VPTQ, which is also a vector-quantization-based method, shows comparable accuracy. However, similar to AQLM, it relies on dequantization-based kernels and therefore exhibits lower throughput than the FP16 baseline. We appreciate the suggestion, which helped enrich our study, and we plan to include this result in the camera-ready version if accepted.\n\n| Method                      | 𝑞̄     | tok/s | MMLU  | WG    | HS    | ARC-E | ARC-C | Avg.  |\n|----------------------------|--------|-------|-------|-------|-------|--------|--------|--------|\n| FP16                       | 16.000 | 34.3  | 68.39 | 73.95 | 79.2  | 79.63 | 55.03 | 71.26 |\n| FlexRound-q2-g128 [12]     | 2.125  | 34.4  | 24.27 | 55.16 | 43.78 | 24.75 | 24.57 | 41.65 |\n| AQLM-2x8 [5]               | 2.005  | 20.6  | 42.29 | 58.25 | 61.40 | 46.25 | 30.89 | 47.82 |\n| +PV-Tuning                 | 2.005  | 20.6  | 55.13 | 69.14 | 72.43 | 71.25 | 45.73 | 62.74 |\n| AQLM-1x16 [5]              | 2.213  | 20.8  | 58.74 | 68.75 | 70.21 | 73.99 | 46.16 | 63.57 |\n| +PV-Tuning                 | 2.213  | 20.8  | **60.72** | **70.24** | **74.33** | **74.92** | **48.89** | **65.82** |\n| CodeGEMM-m1-v4-g128    | 2.126  | 36.8  | 45.16 | 58.96 | 63.07 | 63.97 | 38.48 | 53.93 |\n| +PV-Tuning                 | 2.126  | **36.8**  | 57.42 | 69.06 | 73.85 | 73.15 | 46.33 | 63.96 |\n| CodeGEMM-m2-v8-g128    | 2.127  | 36.2  | 41.53 | 55.72 | 62.08 | 65.07 | 39.51 | 52.78 |\n| +PV-Tuning                 | 2.127  | 36.2  | 56.64 | 69.06 | 72.58 | 73.06 | 46.76 | 63.62 |\n| **VPTQ**                       | 2.300  | 31.7 | 44.87 | 69.06 | 70.83 | 64.94 | 40.19 | 57.98 |\n\n\n## Questions\n> Is there any limitation on the size of the product-sum values stored in memory for large language models such as LLaMA 3.1 with 405B parameters?\n- Our approach is independent of the total model size and scales well to extremely large models such as LLaMA 3.1–405B. The only memory constraint lies in the size of on-chip memory used to store the Psumbook. The size of Psumbook depends only on codebook parameters (e.g., number of codebooks and vector length), not on the number of parameters. Importantly, the performance of CodeGEMM improves with larger problem sizes. As shown in the table below, CodeGEMM achieves greater speedup over cuBLAS as the matrix dimensions grow. Each matrix shape corresponds to configurations used in LLaMA 3 models of varying sizes, including 1B, 8B, 70B, and 405B. (Values in parentheses indicate ±2σ error over 128 measurement samples.) \n\n| M | N     | K     | cuBLAS ($\\mu s$)     | CodeGEMM_m2v8 ($\\mu s$) | Speed up | CodeGEMM_m1v4 ($\\mu s$) | Speed up |\n|---|--------|--------|------------------|----------------------|-----------|----------------------|-----------|\n| 1 | 2048  | 2048  | 20.1 (±0.9)       | 20.8 (±1.4)           | ×1.0      | 20.7 (±0.9)           | ×1.0      |\n| 1 | 8192  | 2048  | 31.2 (±1.3)       | 25.9 (±0.9)           | ×1.2      | 26.7 (±0.9)           | ×1.2      |\n| 1 | 2048  | 8192  | 27.6 (±1.2)       | 24.3 (±0.8)           | ×1.1      | 23.8 (±1.7)           | ×1.2      |\n| | | | | | | | | \n| 1 | 4096  | 4096  | 27.8 (±1.5)       | 24.8 (±0.8)           | ×1.1      | 25.0 (±1.0)           | ×1.1      |\n| 1 | 14336 | 4096  | 88.8 (±6.7)       | 38.8 (±1.1)           | ×2.3      | 37.5 (±1.2)           | ×2.4      |\n| 1 | 4096  | 14336 | 86.2 (±6.9)       | 36.2 (±1.5)           | ×2.4      | 33.9 (±1.1)           | ×2.5      |\n| | | | | | | | | \n| 1 | 8192  | 8192  | 96.4 (±7.2)       | 38.0 (±1.0)           | ×2.5      | 35.4 (±1.1)           | ×2.7      |\n| 1 | 28672 | 8192  | 297.9 (±11.5)     | 86.5 (±1.1)           | ×3.4      | 76.7 (±0.8)           | ×3.9      |\n| 1 | 8192  | 28672 | 302.0 (±6.3)      | 86.2 (±0.8)           | ×3.5      | 76.5 (±1.8)           | ×3.9      |\n| | | | | | | | | \n| ***1*** | ***16384*** | ***16384*** | 340.6 (±8.4)      | 97.2 (±1.3)           | ×3.5      | 97.0 (±2.4)           | ×3.5      |\n| ***1*** | ***53248*** | ***16384*** | 1023.5 (±10.3)    | 263.9 (±1.9)          | ×3.9      | 263.7 (±2.4)          | ×3.9      |\n| ***1*** | ***16384*** | ***53248*** | 1060.0 (±10.8)    | 263.9 (±1.3)          | ×4.0      | 263.8 (±1.2)          | ×4.0      |\n\n\n***We sincerely appreciate the insightful suggestions provided. Throughout the rebuttal period, we have made our best effort to address the reviewers' feedback and strengthen the paper accordingly. That said, we recognize that there may still be areas for improvement. If there are additional suggestions or concerns, we would be more than happy to incorporate them. Thank you again for your thoughtful and constructive comments.***"}, {"review_id": "foSxR5ekdd", "reviewer": "Reviewer_wnqh", "summary": "The paper presents CodeGEMM, a new GPU kernel for weight-only, codebook-based quantization in large language models (LLMs). Instead of performing on-the-fly dequantization by loading full centroid vectors into cache, CodeGEMM precomputes all possible inner products between centroids and input subvectors—storing them in a compact Psumbook. During GEMM, it uses each code to index directly into this Psumbook, eliminating repeated dequantization and reducing both computational and space complexity. \nThe kernel is parameterized by vector length $v$, number of codebooks $m$, bits per code $b$, and tile width $t_w$, enabling exploration of the latency memory accuracy trade-off.", "questions": "- Can you provide detailed microbenchmarks separating the time spent building versus reading the Psumbook, especially for varying $t_w$ and batch sizes?\n- How does CodeGEMM fare when coupled with codebooks generated by TurboQuant (https://arxiv.org/abs/2504.19874)?\n- Have you measured DRAM traffic reductions, and can you provide quantitative comparisons to dequantization-based kernels?\n- Do you envision streamed or hierarchical Psumbook designs that allow $b>8$ without cache overflow, and what challenges do these impose?", "limitations": "See Weaknesses", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Precomputing and caching inner products removes the need to fetch full centroids for each code, directly addressing the shared-memory capacity bottleneck in existing kernels \n- Kernel-level benchmarks demonstrate up to 2.18× improvement over FP16 cuBLAS and 1.64× over AQLM on A100 GPUs, and throughput evaluations confirm real-world speedups in the HuggingFace Llama pipeline \n\n**Weaknesses:**\n- The reliance on fitting the Psumbook in shared memory forces $b\\le8$ and precludes exploration of larger codebook widths (e.g., $b=16$), limiting accuracy potential in extreme low-bit regimes .\n- The focus is on latency and perplexity; energy consumption and DRAM bandwidth savings which are critical in production are unreported.\n- It remains unclear how much time is spent constructing the Psumbook versus retrieving entries during GEMM. A breakdown would clarify whether build overheads offset retrieval gains at different batch sizes or tile shapes.\n- While NeurIPS checklist items are cited, the paper omits error bars (claimed negligible variance) and details on code release versioning, which may hinder exact reproduction .", "was_revised": true, "final_justification": "I thank the authors for their response and have read it carefully. While I acknowledge that some concerns may have been addressed to the satisfaction of other reviewers, my own vote is a borderline rejection.", "rebuttal": "## Weaknesses\n> The reliance on fitting the Psumbook in shared memory forces $b\\le8$ and precludes exploration of larger codebook widths (e.g., $b=16$), limiting accuracy potential in extreme low-bit regimes .\n- We agree, and this is indeed a clear limitation of our current work. We believe that model compression serves different goals depending on the deployment scenario. If the primary objective is to minimize memory footprint with minimal accuracy degradation, then using larger codebooks such as b = 16 is certainly effective.\n- However, when practical speedup is also a goal, algorithm-hardware co-design becomes essential. To support this, we deliberately introduced constraints during the design of our compression algorithm to ensure compatibility with on-chip memory and efficient kernel execution.\n- As shown in Table 3, recent calibration methods such as PV-Tuning offer promising ways to recover much of the accuracy lost under aggressive quantization. Notably, PV-Tuning was particularly effective in the b = 8 regime, where degradation was previously much higher compared to b = 16. We expect that as more advanced tuning and compression algorithms emerge, the remaining accuracy gap will continue to shrink even under shared memory constraints.\n\n> While NeurIPS checklist items are cited, the paper omits error bars (claimed negligible variance) and details on code release versioning, which may hinder exact reproduction .\n- We apologize for the omission. To support reproducibility, we provide the exact version information of the software environment used in our experiments: python==3.12.3, CUDA==12.6, torch==2.6.0, transformers==4.46.3, lm-eval==0.4.5\n- We also fully agree that including error bars improves transparency. For this rebuttal, we have added 2-sigma error margins over 128 measurement samples to all reported numbers in our responses to the reviewers. While we are unable to modify the manuscript at this stage, we will include error bars in all relevant results in the camera-ready version.\n\n## Questions\n\n> The focus is on latency and perplexity; energy consumption and DRAM bandwidth savings which are critical in production are unreported.\n\n> Have you measured DRAM traffic reductions, and can you provide quantitative comparisons to dequantization-based kernels?\n- Thank you for the suggestion. We agree on the importance of hardware utilization and conducted a set of experiments to compare the efficiency of different kernels.\n- To evaluate DRAM traffic and power efficiency, we used nvidia-smi queries [1] sampled every 100 milliseconds for 10 seconds, then averaged the results. The results show that CodeGEMM achieves significantly better compute efficiency (FLOPS per watt) compared to dequantization-based kernels.\n- The table below summarizes the results for various methods on a matrix multiplication workload with M = 1, N = 28672, and K = 8192. All values in parentheses indicate 2-sigma error margins over 128 measurement samples.\n\n| Method         | Latency (us)     | TFLOPS | Power (W)         | GFLOPS/W | GPU Util (%)      | Mem Util (%)      |\n|----------------|------------------|--------|--------------------|----------|--------------------|--------------------|\n| cuBLAS         | 297.74 (±11.93)  | 1.58   | 318.55 (±6.26)     | 4.95     | 96.87 (±0.73)      | 96.94 (±0.48)      |\n| aqlm_1x16       | 625.53 (±10.30)  | 0.75   | 126.54 (±0.49)     | 5.93     | 99.00 (±0.00)      | 6.00 (±0.00)       |\n| aqlm_2x8        | 181.54 (±1.83)   | 2.59   | 254.20 (±2.47)     | 10.18    | 92.84 (±1.58)      | 19.96 (±0.39)      |\n| codegemm_m2v8   | 86.48 (±1.13)    | 5.43   | 304.69 (±6.11)     | 17.83    | 85.32 (±1.58)      | 43.76 (±0.95)      |\n| codegemm_m1v4   | 76.71 (±0.75)    | 6.12   | 316.38 (±8.37)     | **19.36**    | 84.47 (±2.28)      | 49.80 (±1.21)      |\n\n- These results highlight that CodeGEMM not only achieves lower latency but also delivers higher energy efficiency and better memory subsystem utilization, suggesting reduced and more structured DRAM access compared to dequantization-based approaches. We will include this analysis in the final version.\n\n[1] https://docs.nvidia.com/deploy/nvidia-smi/index.html\n\n> Can you provide detailed microbenchmarks separating the time spent building versus reading the Psumbook, especially for varying $t_w$ and batch sizes?\n- Thank you for the insightful suggestion. While it is challenging to measure precise cycle-level timing for each kernel phase due to concurrent execution across multiple SMs on the GPU, we performed controlled measurements by isolating execution to a single SM in order to estimate the relative cost of building and reading the Psumbook.\n- The table below reports the percentage of execution cycles spent on each phase for various tile widths $t_w$ and batch sizes M. For a fixed $t_w$, increasing M shows that the ratio between building and reading remains largely stable, indicating that the Psumbook construction cost is effectively amortized across the batch. For a fixed M, increasing $t_w$ tends to increase the proportion of time spent on building in small matrices, but decrease it in large matrices.\n\n| M | N    | K    | $t_w$ | Psumbook Phase (%)       | m2v8         | m1v4         |\n|---|------|------|------|--------------------------|--------------|--------------|\n| 1 | 4096 | 4096 | 32   | Building / Reading       | 30.5 / 69.5  | 20.3 / 79.7  |\n| 1 | 4096 | 4096 | 64   |                          | 33.0 / 67.0  | 28.5 / 71.5  |\n| 1 | 4096 | 4096 | 128  |                          | 31.2 / 68.8  | 30.7 / 69.3  |\n| 1 | 8192 | 8192 | 32   |                          | 45.4 / 54.6  | 41.2 / 58.8  |\n| 1 | 8192 | 8192 | 64   |                          | 45.6 / 54.4  | 39.7 / 60.3  |\n| 1 | 8192 | 8192 | 128  |                          | 28.3 / 71.7  | 29.5 / 70.5  |\n| 1 | 4096 | 4096 | 32   |                          | 30.5 / 69.5  | 20.3 / 79.7  |\n| 4 | 4096 | 4096 | 32   |                          | 30.4 / 69.6  | 20.7 / 79.3  |\n| 8 | 4096 | 4096 | 32   |                          | 30.7 / 69.3  | 20.4 / 79.6  |\n| 1 | 8192 | 8192 | 32   |                          | 45.4 / 54.6  | 41.2 / 58.8  |\n| 4 | 8192 | 8192 | 32   |                          | 45.7 / 54.3  | 41.3 / 58.7  |\n| 8 | 8192 | 8192 | 32   |                          | 46.1 / 53.9  | 41.6 / 58.4  |\n\n> How does CodeGEMM fare when coupled with codebooks generated by TurboQuant (https://arxiv.org/abs/2504.19874)?\n- Thank you for the valuable suggestions. TurboQuant is an online vector quantization method primarily designed for KV‑cache compression. This aligns well with CodeGEMM, which dynamically operates on precomputed codebooks and codes, replacing on-the-fly dequantization with psum lookups. CodeGEMM is agnostic to how the codebooks are trained; it only requires the centroids, codes, and the associated hyperparameters (m, b, v) to construct a Psumbook for the current input tile. As such, codebooks generated by TurboQuant can be directly used with CodeGEMM, provided that the per-codebook bitwidth and vectorization choices fit within on-chip memory constraints.\n- While TurboQuant is designed for online KV‑cache quantization, CodeGEMM naturally extends to this use case. In attention matmuls with quantized activations, the Psumbook is constructed between the current query tile and the key or value centroids, followed by a lookup using the stored codes. This maintains the same memory and compute characteristics, although the Psumbook must be rebuilt for each new query tile.\n\n\n> Do you envision streamed or hierarchical Psumbook designs that allow $b>8$ without cache overflow, and what challenges do these impose?\n- We agree that fitting the entire Psumbook in shared memory limits the maximum codebook size $b$. We are exploring two complementary extensions: streamed or hierarchical Psumbook designs. First, a streamed Psumbook that partitions the table along the codebook dimension and prefetches chunks from global memory with double buffering, overlapping data transfer and lookup. This avoids cache overflow but introduces additional bandwidth demand and synchronization overhead. Second, a hierarchical Psumbook that decomposes a wide codebook into coarse and residual sub-codebooks. Partial sums are built separately for each level and then aggregated, which reduces the $2^b$ memory blow-up at the cost of extra lookups and combination logic. The main challenges include hiding global memory latency, preventing bank conflicts under larger tiles, and managing register and shared memory usage to sustain occupancy. Additionally, if the computational overhead of Psumbook building grows beyond a certain point, the net performance gain of CodeGEMM may diminish.\n\n***We sincerely appreciate the insightful suggestions provided. Throughout the rebuttal period, we have made our best effort to address the reviewers' feedback and strengthen the paper accordingly. That said, we recognize that there may still be areas for improvement. If there are additional suggestions or concerns, we would be more than happy to incorporate them. Thank you again for your thoughtful and constructive comments.***"}, {"review_id": "VirwA6TEmI", "reviewer": "Reviewer_DeVt", "summary": "CodeGEMM is a GPU GEMM kernel designed for large language models that eliminates the costly dequantization step by pre-computing all inner products between input activations and centroid vectors, storing the results in a small shared-memory “Psumbook.” At inference time, the kernel merely indexes this table with the quantized codes, reducing cache traffic and computational complexity while accommodating a broad range of quantization hyperparameters (bits per code, number of codebooks, vector length, group size) without kernel rewrites. Evaluations on 2-bit Llama-3 (8 B and 70 B) show up to 2.27× kernel-level speedups and 1.8× higher end-to-end throughput over state-of-the-art codebook methods, illustrating that a codebook-centric GEMM can translate the theoretical storage gains of extreme low-bit quantization into real-world LLM inference performance.", "questions": "1. Can you provide more benchmark results for the kernel for different problem sizes (M, N, K)? \n2. Line 175 says \"each input vector interacts with a limited set of centroids\". Can you provide any evidence for this assumption? \n3. If higher bits per weight were used, what would the speedup be like?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- This paper is overall well written and easy to follow. The method and results are clearly presented.\n- Motivation and the design of the kernel are visualised with side-by-side diagrams (Figure 1) and a step-wise schematic (Figure 3). \n- The authors derive both computational and space complexity, showing CodeGEMM reduces the multiply-accumulate count by a factor ≈ m/v. \n- It provides up to 2.27× speed-up compared to other codebook-based quantization methods with comparable accuracy in the 2-bit configuration.\n\n\nWeaknesses: \n- It only provides benchmark results of the kernel under two different model sizes. Since this work is all about kernel optimization, more benchmark results under different problem sizes are supposed to be given. \n- Implementation details such as choosing t_w = 32, t_h = 2048 are mentioned but not justified. \n- Because the Psumbook must reside in shared memory, very large codebooks (b = 16) are out of scope; the authors themselves list this as a limitation.", "was_revised": true, "final_justification": "My concerns were addressed by the additional benchmark data. So, I'd like to maintain my score.", "rebuttal": "## Weaknesses\n> It only provides benchmark results of the kernel under two different model sizes. Since this work is all about kernel optimization, more benchmark results under different problem sizes are supposed to be given.\n\n> Can you provide more benchmark results for the kernel for different problem sizes (M, N, K)?\n- Thank you for the suggestion, and we apologize for not including a more comprehensive set of benchmarks earlier. We have now conducted additional experiments across a wide range of (M, N, K) configurations. (Values in parentheses denote ±2σ error margins of 128 samples.)\n- Unlike cuBLAS, which utilizes Tensor Cores and maintains relatively stable latency across batch sizes, recently proposed quantized kernels, including ours, run on CUDA cores and tend to exhibit increased latency as the batch size (M) grows. Among these state-of-the-art kernels, CodeGEMM consistently demonstrates strong performance on large-scale matrix multiplications, where its compute and memory efficiency are best utilized.\n\n| M | N | K  | cuBLAS | AQLM_m1v8b16  | AQLM_m2v8 | CodeGEMM_m2v8  | CodeGEMM_m1v4| QUIP#_e8p | QTIP|\n|---|------|------|--------|----------|-----------|------------------|------------------|------------------|----------------|\n| 1 | 2048 | 2048 | 19.82 (±0.84)| 28.84 (±4.05)  | 20.55 (±0.52) | 20.75 (±1.43) | 20.66 (±0.90) | 19.47 (±0.85) | ***19.44 (±1.46)***|\n| 4 | 2048 | 2048 | 19.99 (±1.95)| 74.67 (±9.95)  | 43.31 (±0.64) | 44.04 (±0.71) | 41.92 (±1.11) | 36.71 (±1.74) | 36.00 (±0.76)|\n| 8 | 2048 | 2048 | 19.79 (±1.09)| 135.36 (±6.28) | 73.03 (±1.02) | 75.18 (±1.13) | 69.72 (±2.45) | 59.44 (±1.06) | 57.87 (±1.11)|\n| 1 | 8192 | 2048 | 30.57 (±1.36)| 28.84 (±4.05)  | 28.83 (±0.67) | 25.94 (±0.89) | 26.70 (±0.88) | ***25.52 (±0.68)*** | 27.08 (±0.82)|\n| 4 | 8192 | 2048 | 31.31 (±1.33)| 74.67 (±9.95)  | 76.15 (±1.90) | 63.97 (±1.17) | 65.36 (±1.03) | 60.70 (±0.56) | 66.18 (±0.90)|\n| 8 | 8192 | 2048 | 31.70 (±1.54)| 135.36 (±6.28) | 138.09 (±1.00)| 115.39 (±1.34)| 116.11 (±1.61)| 107.85 (±0.89)| 118.99 (±17.80)|\n| 1 | 2048 | 8192 | 27.52 (±1.13)| 60.47 (±18.39) | 30.93 (±0.74) | 24.28 (±0.84) | 23.81 (±1.72) | ***23.44 (±0.81)*** | 24.90 (±1.19)  |\n| 4 | 2048 | 8192 | 29.82 (±1.37)| 203.86 (±10.30)| 82.18 (±1.29) | 56.21 (±1.24) | 52.57 (±1.21) | 51.91 (±1.33) | 59.03 (±0.89)  |\n| 8 | 2048 | 8192 | 28.69 (±1.03)| 396.44 (±13.32)| 149.98 (±1.90)| 98.92 (±0.96) | 90.73 (±1.12) | 89.91 (±1.28) | 103.24 (±0.78) |\n| 1 | 4096 | 4096 | 28.00 (±1.51)| 63.13 (±15.02) | 32.28 (±0.92) | 24.76 (±0.82) | 24.97 (±0.96) | ***23.96 (±0.89)*** | 26.74 (±0.96)  |\n| 4 | 4096  | 4096 | 28.54 (±0.94)| 210.03 (±17.94) | 89.76 (±1.03)  | 60.58 (±1.28) | 57.79 (±1.17) | 53.92 (±1.37) | 62.74 (±0.97) |\n| 8 | 4096  | 4096 | 28.11 (±0.74)| 396.37 (±19.42) | 165.49 (±0.80) | 108.16 (±1.15)| 103.92 (±0.93)| 93.43 (±1.04) | 110.84 (±0.83)|\n| 1 | 14336 | 4096 | 88.67 (±7.59)| 168.12 (±6.50)  | 64.76 (±2.44)  | 38.85 (±1.15) | ***37.51 (±1.24)*** | 38.91 (±0.69) | 51.30 (±1.15) |\n| 4 | 14336 | 4096 | 89.08 (±6.37)| 632.69 (±16.72) | 217.68 (±7.61) | 111.20 (±1.35)| 106.90 (±1.51)| 113.28 (±0.85)| 161.23 (±0.99)|\n| 8 | 14336 | 4096 | 89.29 (±6.19)| 1252.55 (±17.89)| 422.89 (±10.00)| 211.37 (±2.31)| 196.68 (±2.60)| 212.55 (±0.86)| 308.37 (±0.97)|\n| 1 | 4096   | 14336 | 86.31 (±6.76)  | 169.31 (±4.36)    | 58.70 (±0.63) | 36.15 (±1.46) | ***33.92 (±1.12)*** | 37.27 (±1.01) | 43.85 (±0.70)  |\n| 4 | 4096   | 14336 | 86.51 (±6.66)  | 635.74 (±9.95)    | 193.41 (±0.65)| 103.15 (±2.33)| 92.61 (±2.05) | 106.63 (±0.73)| 133.36 (±1.32) |\n| 8 | 4096   | 14336 | 86.49 (±7.32)  | 1253.11 (±11.63)  | 372.97 (±1.77)| 192.63 (±3.34)| 170.16 (±2.65)| 199.31 (±0.83)| 252.12 (±0.94) |\n| 1 | 8192   | 8192  | 96.40 (±7.16)  | 188.91 (±4.24)    | 62.50 (±0.96) | 37.99 (±1.03) | ***35.45 (±0.92)***    | 38.31 (±0.65) | 49.86 (±1.00)  |\n| 4 | 8192   | 8192  | 100.41 (±6.41) | 713.24 (±4.30)    | 208.11 (±0.81)| 111.00 (±1.63)| 98.66 (±1.79) | 111.08 (±0.66)| 157.26 (±0.91) |\n| 8 | 8192   | 8192  | 95.45 (±8.13)  | 1408.68 (±5.48)| 402.29 (±2.42)  | 207.73 (±2.37)| 184.25 (±2.18)| 208.29 (±0.91)| 299.24 (±1.55) |\n| 1 | 28672  | 8192  | 297.74 (±11.93)| 625.53 (±10.30)| 181.54 (±1.83)  | 86.48 (±1.13) | ***76.71 (±0.75)*** | 101.98 (±2.07)| 134.03 (±1.10) |\n| 4 | 28672  | 8192  | 303.10 (±7.94) | 2462.88 (±3.48)| 684.92 (±1.66)  | 305.47 (±1.43)| 264.31 (±1.45)| 366.74 (±3.86)| 492.14 (±1.77) |\n| 8 | 28672  | 8192  | 295.11 (±10.35)| 4913.52 (±6.10)| 1355.70 (±3.20) | 597.22 (±1.71)| 514.85 (±1.72)| 718.13 (±5.03)| 970.35 (±1.71) |\n| 1 | 8192   | 28672 | 302.42 (±6.26) | 618.61 (±2.04) | 180.38 (±1.32)  | 86.20 (±0.84) | ***76.50 (±1.82)*** | 101.13 (±2.12)| 124.90 (±1.03) |\n| 4 | 8192   | 28672 | 292.59 (±6.29) | 2437.82 (±2.30)| 679.24 (±2.01)  | 305.14 (±1.48)| 263.70 (±2.13)| 361.95 (±3.70)| 455.84 (±1.19) |\n| 8 | 8192   | 28672 | 293.69 (±7.43) | 4860.85 (±3.99)| 1344.49 (±1.70) | 596.63 (±2.25)| 515.12 (±3.34)| 710.94 (±4.49)| 897.41 (±1.38) |\n\n> Implementation details such as choosing $t_w = 32, t_h = 2048$ are mentioned but not justified.\n- Thank you for pointing this out. Initially, we chose the tile dimensions $t_w = 32$ and $t_h = 2048$ based on heuristic tuning. However, prompted by your feedback, we conducted a more systematic analysis of these implementation parameters.\n- We found that $t_h = 2048$ consistently provided the best performance across a wide range of workloads, supporting our original choice. Interestingly, for $t_w$, smaller values such as $t_w = 32$ worked well for relatively small matrix sizes, whereas larger values like $t_w = 64$ yielded better performance on large-scale matrix multiplications. We believe this is because larger matrices benefit from coarser tiling, which reduces launch overhead and improves partial sum reduction at scale.\n- We will include a brief discussion of these observations in the implementation section of the final version.\n\n| M | N    | K    | t_w | t_h  | CodeGEMM_m2v8   | CodeGEMM_m1v4   |\n|--|----|----|-----|------|-------------|-------------|\n|1|4096|4096|32 |2048|26.57 (±1.40)|25.07 (±1.28)|\n|1|4096|4096|64 |2048|26.76 (±0.93)|25.40 (±1.08)|\n|1|4096|4096|128|2048|29.61 (±0.83)|26.81 (±0.99)|\n|1|4096|4096|32 |4096|28.95 (±0.88)|27.60 (±0.86)|\n|1|4096|4096|64 |4096|28.49 (±1.29)|27.68 (±1.01)|\n|1|4096|4096|128|4096|37.58 (±1.02)|32.87 (±0.86)|\n|1|8192|8192|32 |2048|39.04 (±1.19)|36.02 (±0.98)|\n|1|8192|8192|64 |2048|37.23 (±0.99)|35.33 (±0.83)|\n|1|8192|8192|128|2048|40.09 (±0.80)|38.54 (±1.43)|\n|1|8192|8192|32 |4096|37.78 (±0.81)|36.17 (±1.18)|\n|1|8192|8192|64 |4096|38.29 (±1.08)|37.70 (±0.67)|\n|1|8192|8192|128|4096|45.40 (±0.70)|42.75 (±1.20)|\n\n> Because the Psumbook must reside in shared memory, very large codebooks (b = 16) are out of scope; the authors themselves list this as a limitation.\n- We agree, and this is indeed a known limitation of our current implementation. As noted, very large codebooks such as b=16 can offer better accuracy but exceed the capacity of on-chip memory (e.g., GPU shared memory), making them impractical in our current design. However, as shown in Table 3, recent calibration techniques such as PV-Tuning have proven highly effective at mitigating accuracy degradation even under smaller codebooks like b=8, which previously showed substantial loss. We are optimistic that as more advanced tuning and quantization algorithms are developed, the remaining accuracy gap will narrow further, without requiring excessively large codebooks.\n\n## Questions\n> Line 175 says \"each input vector interacts with a limited set of centroids\". Can you provide any evidence for this assumption?\n- Thank you for pointing this out. The phrasing in Line 175 may have caused confusion. As illustrated in Figure 1 of the paper, each output activation is computed by combining the input activation with all $2^b$ centroids per codebook, meaning there is no restriction on which centroids are involved in the computation.\n- However, when the number of output channels $N$ exceeds the number of centroid combinations $2^b$, which is common in practice, the pigeonhole principle implies that multiple output channels will share the same centroid indices. This leads to reuse opportunities in the partial sum computation. CodeGEMM exploits this by precomputing the Psumbook, which enables efficient reuse of these dot products across multiple output channels.\n\n> If higher bits per weight were used, what would the speedup be like?\n- Thank you for the suggestion. We also conducted latency measurements for higher-bit quantization settings using the kernel configuration ($g = 128, b = 8, t_w = 32, t_h = 2048$). For reference, FP16 cuBLAS latency is also included.\n- As shown in the results, higher average bit precision generally leads to increased latency, especially when the number of codebooks (m) increases. This trend is more pronounced for larger matrix sizes, such as M = 8192. The measurements confirm that even at higher bit precisions, CodeGEMM maintains competitive performance compared to FP16 baselines, while offering a flexible trade-off between accuracy and efficiency depending on the (m, v) configuration.\n\n| M | N | K | m | v | bit | latency (us) |\n|--|----|----|----|----|------|--------------|\n|1|4096|4096|N/A|N/A|16.000|28.118 (±0.993)|\n|1|4096|4096|1|4|2.126|25.074 (±1.281)|\n|1|4096|4096|2|4|4.127|27.009 (±0.685)|\n|1|4096|4096|1|8|1.127|24.015 (±0.870)|\n|1|4096|4096|2|8|2.129|26.574 (±1.400)|\n|1|4096|4096|3|8|3.126|27.385 (±0.701)|\n|1|4096|4096|4|8|4.127|29.797 (±0.545)|\n|1|8192|8192|N/A|N/A|16.000|95.785 (±6.836)|\n|1|8192|8192|1|4|2.125|36.020 (±0.980)|\n|1|8192|8192|2|4|4.125|49.636 (±1.365)|\n|1|8192|8192|1|8|1.125|31.883 (±0.870)|\n|1|8192|8192|2|8|2.126|39.040 (±1.190)|\n|1|8192|8192|3|8|3.126|47.210 (±0.985)|\n|1|8192|8192|4|8|4.127|58.364 (±0.988)|\n\n***We sincerely appreciate the insightful suggestions provided. If there are additional suggestions or concerns, we would be more than happy to incorporate them.***"}, {"review_id": "VuIBy0f56J", "reviewer": "Reviewer_DEUr", "summary": "This paper introduces CodeGEMM, an optimized matrix multiplication kernel for codebook-based quantized large language models. Traditional codebook-based quantization methods suffer from significant latency due to the need for repeated dequantization and frequent codebook access. \n\nCodeGEMM precompute and storing inner product results between codebook centroids and input data in the programmable cache, eliminating the need for on-the-fly dequantization. The kernel is designed to be flexible across various quantization hyperparameters and model configurations.", "questions": "see weaknesses", "limitations": "Provide a small-scale analysis or discussion of possible failure cases where Psumbook overhead could become a new bottleneck (e.g., if codebook structure is highly non-uniform or input batch sizes are extreme).\n\nIf possible, add error bars or a variance table to the main throughput/accuracy results.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The Psumbook-based approach is a practical and innovative solution to a key bottleneck in codebook-based quantization, significantly reducing both space and compute complexity.\n\n2. The kernel is adaptable to a wide range of quantization settings, supporting diverse codebook sizes, vector lengths, and group normalization strategies, making it practical for real deployment.\n\nWeaknesses\n1. CodeGEMM is constrained by on-chip cache size, making very large codebooks (e.g., b=16) infeasible. The workaround (smaller codebooks, fine-grained group normalization) is reasonable, but may limit certain applications.\n\n2. The performance improvements are well justified empirically, but the paper could benefit from a more formal treatment of complexity, cache usage, and the limits of the Psumbook design.\n\n3. Figure 1 - Figure 3 appears to be a non-vector image. It would be better to replace it with a vector graphic for improved clarity and scalability.", "was_revised": true, "final_justification": "After reading the author's rebuttal and the comments from other reviewers, I maintain my original score.", "rebuttal": "## Weaknesses\n> CodeGEMM is constrained by on-chip cache size, making very large codebooks (e.g., b=16) infeasible. The workaround (smaller codebooks, fine-grained group normalization) is reasonable, but may limit certain applications.\n- We agree, and this is indeed a known limitation of our current implementation. As noted, very large codebooks such as $b=16$ can offer better accuracy but exceed the capacity of on-chip memory (e.g., GPU shared memory), making them impractical in our current design. However, as shown in Table 3, recent calibration techniques such as PV-Tuning have proven highly effective at mitigating accuracy degradation even under smaller codebooks like $b=8$, which previously showed substantial loss. We are optimistic that as more advanced tuning and quantization algorithms are developed, the remaining accuracy gap will narrow further, without requiring excessively large codebooks.\n\n> The performance improvements are well justified empirically, but the paper could benefit from a more formal treatment of complexity, cache usage, and the limits of the Psumbook design.\n- Thank you for pointing this out. While Section 3 provides a high-level discussion of the complexity, cache behavior, and architectural constraints of the Psumbook design, we agree that a more rigorous and detailed analysis could enhance the clarity and completeness of the paper. Our current version emphasizes conceptual clarity and accessibility to help readers understand the motivation behind CodeGEMM. That said, your suggestion is well taken, and we plan to incorporate a more formal treatment in the final version.\n\n> Figure 1 - Figure 3 appears to be a non-vector image. It would be better to replace it with a vector graphic for improved clarity and scalability.\n- Thank you for pointing this out. We agree that vector graphics would improve readability and scalability. While we are unable to modify the manuscript during the rebuttal phase, we will update these figures with vectorized versions in the camera-ready version if accepted.\n\n> If possible, add error bars or a variance table to the main throughput/accuracy results.\n- We appreciate this suggestion. Although we cannot revise the main figures at this stage, we have annotated all numerical results used during rebuttal with ±2σ error margins to ensure transparency and reliability. We will include proper error bars and/or variance tables in the camera-ready version.\n\n***We sincerely appreciate the insightful suggestions provided. Throughout the rebuttal period, we have made our best effort to address the reviewers' feedback and strengthen the paper accordingly. That said, we recognize that there may still be areas for improvement. If there are additional suggestions or concerns, we would be more than happy to incorporate them. Thank you again for your thoughtful and constructive comments.***"}]

Author rebuttal:

Meta-review: The work introduces a matrix multiplication kernel for codebook-based quantized models. Unlike traditional methods that incur latency by repeatedly loading the codebook for dequantization, the proposed approach CodeGEMM eliminates the dequantization step by precomputing inner products into a Psumbook. It supports flexible hyperparameters to balance latency, memory, and accuracy. Experiments show that CodeGEMM achieves a speed-up over state-of-the-art codebook-based quantization methods while maintaining comparable accuracy in 2-bit settings.

This is an interesting algorithm-hardware co-design work. We decided to accept this paper and also suggests the following changes:
- Explicitly point out the limitation of this approach in the final version, e.g., the coodbook size cannot be too large; why CodeGEMM underperforms Tensor Core–based cuBLAS in large-batch operations.
- Narrow down the application area of the proposed approach and retune the claim in the abstract
- Include the additional experimental comparison in the final version.
Decision: Accept (poster)