Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
Authors: Xi Zhang
Xiaolin Wu
Jiamang Wang
Weisi Lin
Venue: NeurIPS 2025 poster ()
Keywords: LLM compression
lattice vector quantization
Primary area: deep_learning
TL;DR: We propose a grouped lattice vector quantization (GLVQ) framework that tailors a customized lattice codebook to each weight group, boosting quantization accuracy and efficiency for LLM.

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities but typically require extensive computational resources and memory for inference. Post-training quantization (PTQ) can effectively reduce these demands by storing weights in lower bit-width formats. However, standard uniform quantization often leads to notable performance degradation, particularly in low-bit scenarios. In this work, we introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook, defined by a learnable generation matrix. To address the non-differentiability of the quantization process, we adopt Babai rounding to approximate nearest-lattice-point search during training, which enables stable optimization of the generation matrices. Once trained, decoding reduces to a simple matrix-vector multiplication, yielding an efficient and practical quantization pipeline. Experiments on multiple benchmarks show that our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines, highlighting its effectiveness in deploying large models under stringent resource constraints. Our source code is available on GitHub repository: https://github.com/xzhang9308/GLVQ.

Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
Xi Zhang'
Xiaolin Wu
Weisi Lin 2 Alibaba Group Southwest Jiaotong University
Jiamang Wang²
Nanyang Technological University
axi.zhang wslin] melin]ontu.. entu.edu.sg
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities but typically require extensive computational resources and memory for inference. Post-training quantization (PTQ) can effectively reduce these demands by storing weights in lower bit-width formats. However, standard uniform quantization often leads notable performance degradation, particularly in low-bit scenarios. In this work, we introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook, defined by a learnable generation matrix. To address the non diffdeffeeeniibiiity of the quantization process, we adopt Babai rounding approximate nearest- tetss-latttceooooii search during train- ing, which enables stable optimization of the generation matrices. Once trained, decoding reduces to simple matrix-vector multiplication, yielding an efficient and practical quantization pipeline. Experiments on multiple benchmarks show that our approach achieves better trade-off between model size and accuracy compared existing post-training quantization baselines, highlighting its effectiveness in deploying large models under stringent resource constraints. Our source code is available GitHub repository: https:/ggthub..omkkkaang/D/O0000
Introduction
Large language models (LLMs) have achieved extraordinary success across wide range of natural language processing tasks, including machine translation, question answering, text generation, and summarizatte
summarization (55
These models, exemplified by the GPT 43), BERT [12), Llama (50),
Qwen (2) and DeepSeek 32 are at the forefront of AI innovation. Beyond supervised pretraining, reinforcement learning techniques, most notably reinforcement learning from human feedback (RLHF), have further enhanced LLMs by aligning them with human preferences and improving their safety, helpfulness, and controllability [49 40 3 33
34 [6]. However, their exceptional performance comes exorbitant computation and memory costs, which hinders their deployment on edge devices and in other resource-limee limited scenarios 7 For instance, state-of-the -ate-of-the-art models with billions of parameters often require hundreds of gigabytes of memory and expensive hardware for inference
posing serious challenges for scalability and accessibility.
Quantization has emerged as a key technique to alleviate the resource burdens by reducing the precision of model weights from 16-bit or 32-bit floating-poin representations to low-hit integers [28] 35 This significantly reduces the memory footprint and computation load, improving model economy and inference speed. A relatively simple method is Post-training quantization (PTQ), which
quantizes pre-trained LLM model without retraining (29 [8]. However, PTQ has difficulties in achieving ultra-low bit-widths, such as 2 or 3 bits per weight 9 Severe quantization errors can lead to significant degradation in model performance, such as higher perplexity in language tasks [18) or noticeable drops in downstream quality, thereby limiting the practical utility of low-bit quantization.
M Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Group-Specific Companding Functions
Group-Specific
Lattice Codebooks
LLM Weight Matrix
Bit Allocation
Group
4 bits
2 bits
bits
Figure 1: An overview of the GLVQ framework. The LLM weight matrix is partitioned into column-wise groups, each assigned a aalience-determined bitwidth. Each group undergoes a tailored companding transformation and is quantized using learned group-specifie lattice codebook, enabling fine-grained control over precision and compression efficiency.
Recently, some progresses have been made in LLM weight quantization. including vector quantization (VQ) techniques [4 [52] and salience-awe bit allocation [23]. Vector quantization methods, such as QulP# [5T, leverage structured codebooks (e.g.. E8 lattice) to improve quantization fidelity by aligning the codebook structure with the statistical properties of model weights. However, QulP# constrained by the use of fixed lattice designs across the entire model, which fails account for the diversity of parameter distributions across different groups or layers. This uniform approach can lead to suboptimal quantization for certain groups, ultimately limiting both compression performance and overall model quality. In contrast, AQLM (14) proposes learning free-form VQs for different groups, which allows for more flexible quantization. However, this approach has the drawback that decoding requires the lookup operation, which is computationally more expensive than QUIP# and other existing PTQ methods.
In this paper, we propose novel framework, Grouped Lattice Vector Quantization (GLVQ) (illus- trated in Fig. [T.. aiming to increase the L.LM compression ratio by reducing quantization resolution to be below 3 bits per weight. Specifically, GLVQ end-to-end optimizes the geometric structure of the lattice codebook for each quantization group based on that group's 's unique parameter distribution. By shaping the lattice to fit the data distribution of each group, we reduce quantization distortion, pre- serve critical information better, and improve model fidelity over nonadaptive lattices under extreme compression regimes. Compared to free-form VQ used in AQLM 14, GLVQ employs a structured codebook, hence enjoys the operational advantage of much lower quantization complexity. Further investigations reveal that LLM parameter distributions unfortunately happens, optimizing GLVQ alone has limited effect, because the lattice cells of each group have congruent shape, albeit optimized, and have regular configurations. To counteract this, we incorporate group-specific companding transformations. These functions reshape each group's 's parameter distribution into more uniform space before applying the lattice quantizer, thereby reducing distortion the low-magnitude regions and enabling the lattice to allocate codepoints more effectively. Together, group-specific lattice codebook optimization and companding form powerful synergy that maintains high accuracy while significantly reducing bit-width requirements.
be highly skewed or heavy-tailed. If this
We validate our approach on popular LLM benchmarks, demonstrating its effectiveness in improving perplexity and inference efficiency compared to state-of-tt of-the----e-rrt quantization methods [18] 51 14 [
Our results show that GLVQ not only enables higher compression ratios but also maintains robust performance across diverse language tasks, making it a practical solution for deploying LLMs in censtrce- environments. Furthermore, framework provides insights into the interplay between parameter distribution and quantization fidelity, opening new directions for research in efficient model compression.
This paper makes the following key contributions:
We propose a novel grouped lattice vector quantization approach that dynamically adapts lattice structures to fit the unique distribution of each weight group.

---

We introduce group-specific companding mechanism that reshapes each parameter distri- bution prior lattice quantization, further reducing quantization distortion and improving overall performance.
Extensive experiments on post-training quantization benchmarks for LLMs show that GLVQ achieves a superior trade-off between accuracy and inference efficiency under extreme compression regimes.
2 Background and Related Work
2.1 LLM Post- Trai-ina Quantization (PTQ)
Recent surveys [99 [1 have emphasized the importance of post-training quantization (PTQ) methods in the context of large language models (LLMs), particularly as model scales reach billions of parameters. Classic approaches such as AdaRound [38, BitSplit [56), AdaQuant (24], BRECQ [29]. and OBQ 17 have been shown to work well for smaller networks. However, applying these techniques directly LLMs often incurs prohibitive computational overhead due to their reliance on expensive solvers and per-layer fine-grained adjustments.
In effort to address scalability challenges, multiple PTQ frameworks have emerged. For instance, GPT3.int8() [9]. ZeroQuant [61], and Lut-gemm [41] employed straightforward round-to-neared (RTN) quantization strategies and introduced granular control to manage the trade-off between model size and accuracy. GPTQ [8] proposed a more data-aware layer-wise wise optimization to minimize the l2 reconstruction error. Follow-up studies [11] argued that 4-bit RTN might be near-optimal for certain classes of networks, but datae schemes like GPTQ can exceed this limit without compromising efficiency. Meanwhile, 8-bit quantization of both weights and activations has also been investigated ( (58 61), revealing that "outlier features" can substantially degrade performance, and spurring specialized outlier-handlngg techniques.
Subsequent efforts prioritized addressing such outliers in weight matrices. For example, SpQR [00 maintains sparse, higher-precision representation of large-magnitde weights to mitigate their outsized influence. Similarly, AWQ [30] employs per-channel scaling to handle channels with large activation magnitudes, while SqueezeLLM (27 adopts diagonal Fisher approximations and K-means clustering to perform non-uniform quantization, selectively preserving critical weights in full precision. A notable result is QuIP [6), which strategically rotates (e.g., via Hadamard transform) and then projects weights onto a lattice to control outliers, enabling 2-bit per-parameter compression with only minor increases in perplexity.
Beyond these developments, recent research introduces vector quantization (VQ) to further enhance LLM compression. Transformer-VO VQ [31) applies VQ to the Attention key vectors, effectively reduc- ing Attention complexity to linear time. Other VQ-based approaches generally optimize discrete code assignments (often in the form of one-hot vectors) and their associated codebooks [4. For in- stance, AQLM [4] proposes an additive quantization strategy that is input-adaptive, while QuIP# [51] leverages highly symmetric Eg lattice handle weights that follow (approximately) a spherical sub-Gaus Gaussian distribution. GPTVQ [4]] interleaves column-wise quantization steps with updates to the unquantized parameters, guided by the Hessian of the per-layer output reconstruction MSE, and then employs SVD and integer quantization for additional codebook compression. PV-Tuning [36] shows how relying on straight-through estimators (STE) alone can be suboptimal, proposing a stage- wise method that separately optimizes continuous parameters (e.g., scales, codebooks) and discrete assignments. QTIP (52) decouples the notion of codebook size from both bitrate and dimensionality through a stateful decoding, thereby facilitating novel PTQ scheme for weights and activations that is based on self-similar nested lattices.
VQ. NestQuant [46) proposes
2.2 Lattice Vector Quantization (LVQ)
Scalar quantization maps each parameter to a discrete code individually. Vector quantization, instead, quantizes group of parameters as vector by mapping the nearest code vector from a codebook [22]. Lattice-based quantization [8] 20 15) goes step further, leveraging a structured grid in the high-dimensional space for efficient quantization. Formally, let G gddd be a generation matrix, which defines a full-rank lattice: A = G E zd). Each column of G acts as a basis vector in Rd any lattice point can be written an integer combination of these basis vectors.
3

---

Given a vector x E RR, the encoding operation aims to find an integer vector 2 E Zd such that the corresponding lattice point G 2 is close x in the Euclidean sense:
2 arg min ||X G Gzll- 2E2
(1)
In general, solving the nearest-lattice lateee---atiicc-ointt problem is both computationally prohibitive for large d and non-differentiable, motivating the use of efficient approximate methods such as Babai round- ing []] Once the integer lattice index is found and stored, multiplying back: & = G 2, yielding a point on the lattice that approximates the original vector x.
decode the quantized vector by
Learnable LVQ. Recent studies have shown that learning or adapting the generation matrix G to given data distribution can significantly reduce quantization error at a fixed bit budget 226 60 [59 (44] This is especially beneficial in low-bit regimes where per-dimension scalar methods often struggle. Additionally, practical implementations of LVQ can incorporate companding or tranpossttaannsfrrmation steps to further align the data with the lattice geometry [15, 62
2.3
Companding for Quantization
Many neural network weight distributions exhibit high dynamic range and non-uniform behavior (Gaussian or sub-Gaussian). which can amplify quantization errors in the low-magnitude regime. Companding (derived from "compressing" and "expanding") 48 25 [3 offers a simple, yet effective
strategy to reshape these distributions before quantization, thereby reducing distortion in critical regions. Typically, the companding process comprises two stages:
Given a value æ, a nonlinear function F()) is applied before quantization to produce = Fxx). and the inverse function is applied after quantization to obtain the reconstruction where 2 denotes the quantized value (e.g., using scalar or sattice-based quantization). This two-step compress- andds-an-eeee -andee reduces errors near zero, where neural parameters are often concentrated, and avoids over-al wer-allooation of code space for very large very small values.
F- (2).
well-known example of F is the p-law transformation processing:
45
4 originally used in audio signal
In(1 + n(
Fx(x) sgggn)))))
1n(1 + pt)
where > 0 is hyperparameter controlling curvature. This two-step process allocates finer resolution near zero and avoids overuse code space for large outliers.
3 Method
The key idea of GLVQ to endow each weight group with its own lattice codebook, thereby matching the heterogeneous statistics observed across different portions of an LLM. Unfortunately. jointly optimizing both the size (bit-width) and the structure (lattice basis) of every codebook is combinatorial and intractable. We therefore decompose the problem into two sequential sub-tasks:
1. Bit allocation. Given global bit budget, we first determine the codebook size for each group by assigning an integer bit-width by. A lightweight heuristic prefers higher precision for groups whose weight statistics exhibit larger dynamic ranges or higher sensitivity, under the constraint Eg by lg < Budget.
2. Lattice codebook learning. With 6, fixed (implying a lattice codebook of 2dx points), we reshape every weight group W, E "R into a matrix of size d x bg (d is the lattice dimension). We then learn group-specific generation matrix G, E [Rddd and the corresponding integer indices Z, E Zdxt, can be obtained via Babai rounding 1 Hence the quantized weight matrix is
W, = (W,,) = =G, Zg.
Fixing the size beforehand simplifies the search space and lets the optimizer focus on discovering a lattice struciure that best fits the local weight distribution. The Babai rounding algorithm does not always return the exact nearest lattice point; nevertheless, its approximation error formally bounded, proved in Appendix A

---

Group-specific companding, To further curb quantization error, we learn a nonlinear companding function (-)
(+) per group. Each weight group undergoes
W (Q
where Qg is the lattice quantizer defined above. This adaptive companding allocates finer resolution near weight values that are most influential for the layer's output, yielding markedly lower error than a fixed uniform quantizer.
3.1 Salienee-Determinet Bit Allocation
To achieve optimal bit allocation under strict resource constraints, adopt the Salience-Determiin Bil Allocation (SDBA) mechanism proposed in Slim-LLM [23], which dynamically assigns bit-widthh to each group based its importance.
Given the weight matrix W. calibration imput X. and a target average bit-width N, SDBA solves the following constrained optimization problem:
arg min DRI. (WX Wx), bg....be
G GEus و = and 19v+1l 19N-11:
(3)
where by z N, N++1,N111
is the bit-width allocated group g. 9N+1 - to by - and GN-1 {g by = NN= This constraint ensures balanced allocation: the number of groups assigned N + bits equals those assigned - bits, while the majority receive N bits. For example, in 2-bit quantization, the highest-salience groups use bits, equal number of low-salience groups use bit, and the rest use 2 bits. To solve this efficiently, we leverage Slim-LLM's 's double-pointer search algorithm. For weight matrix with m output channels and group size 128, the search space is limited to [0, m/ (2g)). requiring only (log m) iterations.
+ 1]
3.2 Lattice Codebook Learning
The key innovation behind GLVQ the ability to adaptively learn specialized lattice structure for each weight group, rather than imposing fixed, universal lattice (such as the Eg lattice employed by QulP# (51). Recent studies, such as Slim-LLM [23], show that even after standard preprocessing. large language models (LLMs) exhibit significant heterogeneity across weight groups, motivating the need for group-specific quantization schemes. Specifically, GLVQ learns generation matrix Rdd for each weight group 9, where the dimension d (e.g., chosen from {8, 16, 32. }) controls both computational complexity and the granularity of quantization. Crucially, the integer lattice indices Zo not directly optimized, but rather determined via Babai rounding, making G, the only explicitly learned parameter.
Gg E
Consider the weight matrix for the g-th group, W, X",, We reshape it into matrix with dimension x Ey. by partitioning it into ly contiguous sub-blocks of dimension and stacking these blocks as columns, such that d mgllg. Quantization then approximates each sub-block using lattice structure defined by Ggi
w GG
where Zy E dx holds the integer lattice indices computed by rounding each column of G W Given calibration inputs X E |g,,x, XN (consisting of N samples), GLVQ minimizes the reconstruction error of the layer outputs after quantization:
Cg = W W x| | X G
(5)
Due to the discrete nature of Zg we solve 5 by alternating between efficient steps:
1. Lattice index assignment (fix G.). Fixing the lattice basis Gy. we update integer indices using Babai rounding [I:
z: G Wx
(6)
for each column w, of W g. This assignment is computationally efficient with complexity O(d)).
5

---

2. Generation matrix optimization (fix zg). With fixed lattice indices Zg we optimize Gg via gradient descent. The gradient is given by:
- - - G,,
(7)
Spectral normalization is applied after each update to constrain the singular values of Gg within a stable range [mii Jmax
To avoid overfitting to the finite calibration set, GLVQ employs a Frobenius regularization term that penalizes deviations from the initial lattice structure:
Leg - G G 2 = 0.1,
(8)
where G (0)
is initialized using the Cholesky decomposition of the group's s covariance matrix, aligning initial lattice directions with the group's 's principal weight distributions.
3.3 Group-Specific Companding
The weight distributions of LLMs are typically heavy-tailed and highly non-uniform; directly quan- tizing such values wastes many code-points on rare outliers. GLVQ therefore inserts an element-wise companding stage before lattice quantization and the corresponding expansion after decoding. Dis- tinct from prior work that uses single global non-linearity, we learn an independent curvature parameter Pg for every group that the transformation adapts to local statistics.
For group we adopt the f-llaw transformation:
In(1 1 + My Myl
(v) = sgn(y) (1 + Mg Mul
(x) = (g)
(9)
1m(1 + Mg)
Hg
where (g 0 controls the compression strength. Applied to the reshaped weights, the overall encode-decode chain is
Wg g (W
Zg
G Wg
W = F (G,Z,)
(10)
Given calibration inputs X E XN the reconstruction loss proposed in Eq. 5 can be updated with the new definition of W, in Eq. accompanying with a Frobenius penalty on the lattice basis:
Cg W X W. w,X1[ A Goll2. -
==
(11)
Because both Fg and g are differentiable in Mgg. update Vg jointly with Gg via gradient descent, while Z, is refreshed by Babai rounding at every iteration.
We initialise
00) 100 (anb((x/100)
(12)
where Kig is the sample kurtosis of W so that heavier-tailed groups start with stronger companding. After each update we project Mg onto the practical range [10, 255] to ensure numerical stability.
3.4 Quantization Pipeline and Runtime Decoding
We present the full pseudocode for GLVQ in Algorithm Starting from initial values G and (0) each iteration (i) reshapes the weight block, applies the group-specific -lw companding, and produces latent vectors Y (ii) quantizes these vectors via Babai rounding to obtain integer codes Zy: (iii) reconstructs provisional weights W g by inverse companding the lattice outputs; and (iv) minimizes reconstruction loss augmented with a Frobenius penalty on Gg. Gradients update the generation matrix and curvature parameter, while Zy implicitly refreshed at the next rounding step. The loop stops when the relative loss reduction falls below E, returning the final compact representation W that combines group-specifie lattice precision with adaptive companding. This design delivers state-of-the-an accuracy under tight bit budgets while meeting the memory and latency constraints of real-world LLM deployment.
Hg
Offline compression. After the optimization Alg. | each weight group represented by (i) an integer-code tensor and (ii) small set of side parameters d' FP16 generation matrix plus one

---

Algorithm Pseudo code of GLVQ algorithm.
Require: Group weights W E Rm*,*%, calibration inputs X E 2R initial generation matrix G (0) R gdxd, initial companding parameter M4
1: Gg + G Ma t Pg
2: repeat
3:
Wg + F (reshape W (d)
apply companding: / m
4:
Zg round G W.)
(Babai lattice rounding)
5:
W9 f reshape F (GZzg), [mg, nol)
(inverse compand reshape)
6:
Lg + [W,X W x F
2
>/| G
rreconstruction + regularizer) {update generation matrix) (update companding parameter] [convergence check)
7:
Gy f- Gy n1 Vc, fag
8:
Me My Tr V
9:
until ( Lg C(--1) (((-1) <E
10:
return W,
FP16 scalar Mg. Because d is modest (8-32) and the matrix is shared by all codes the group, the storage overhead is tiny. For the default configuration used in our experiments (d-16, 4-bit weights, m==4096, ng 112),, the side information adds only 0.2% to the weight codes. Aggregated over the entire Llama 2-7B model, this amounts to roughly 2 MB of extra storage top of 1.1 GB 4-bit payload versus 13.4 GB in FP16. detailed derivation is provided in Appendix
On-the-fly decoding. During inference we materialise just a handful of sub-blocks, apply w F F- and release the data immediately after use. This streaming strategy cuts peak activation- plus- wusshw memory by 10x versus the common practice of pre-decompressing an entire layer. The additional computation is modest-d multiplies per sub-block-and increases end-to-end latency by only 2-3% relative to standard 4-bit uniform PTQ baseline.
Experiments
In this section, evaluate the performance of our proposed grouped Lattice vector quantization (GLVQ) framework on the Llama 1 and models (50). which range from billion to 70 billion parameters. To ensure fair comparison, we follow the experimental setup used in QulPH. Specifically, we evaluate perplexity on the Wikitext-2 (37] and C4 44 datasets, utilizing context lengths of 2048 for Llama 1 and 4096 for Llama 2 models. For zero-shot tasks, we use the LM Eval framework to measure accuracy on tasks such as ARC, PIQA, and the Winograd Schema Challenge (Wino). We report both perplexity and zero-shot accuracy for all methods offer a comprehensive evaluation. We implement our method using PyTorch (42) and CUDA 39], with all experiments conducted on NVIDIA A100 GPUs. For timing experiments, we use an NVIDIA RTX 4090 GPU. We adopt 4M tokens from the RedPajama IT dataset [57) as the calibration sequences in our experiments. We implement two variants of our model with lattice dimensions d 8 and d 32, referred as GLVQ-8D and GLVQ-32D, respectively. For baselines, we compare our method against several state-of-the-arr PTQ approaches, including OmniQuant AWQ 30 QulP# [1] AQLM [4] and
QTIP 52
4.1 Perplexity Results on Llama Models
Our evaluation focuses on perplexity over the Wikitext-2 37 and C4 datasets, using context length of 2048 for Llama and 4096 for Llama Results are reported in Table [ Across all settings, GLVQ consistently achieves lower perplexity than existing methods, particularly in low-bit regimes. For example, under the 2-bit quantization regime on Llama 2-70B, GLVQ-32D achieves perplexity of 3.36 on Wikitext-2, significantly lower than QTIP (3.78) and QulP# (3.91), highlighting that the performance gap becomes most pronounced under extreme compression levels. We also observe consistent improvements across model scales and datasets. On Llama 1- 1-13B, GLVQ-32D improves upon QulP# by 0.41 perplexity points (5.38 vs. 5.79) at 2-bit on Wikitext-2, and 0.53 points (6.95 vs. 7.48) C4. These results validate the robustness of our method across both small and large I.lama variants. Notably, the performance gain more pronounced as the bit-width decreases, highlighting the advantage of GLVQ's group-specific lattice codebooks and companding
7

---

Table 1: Perplexity (1) of competing methods Wikitext2 and C4 for Llama 1 and 2. All results a context length of 2048 for Llama and 4096 for Llama 2.
Method
Bits
Wikitext
2
C4
1-7
1-13
1-30
1-65
2-7
2-13
2-70
1-7
1-13
1-30
1-65
2-7
2-13
2-70
Fp16
16
5.68
5.09
4.10
3.53
5.12
4.57
3.12
7.08
6.61
5.98
5.62
6.63
6.05
4.97
OmniQ
2
15.5
13.2
8.71
7.58
24.9
18.3
13.9
10.8
QulP#
6.86
5.97
5.02
4.36
6.19
5.35
3.91
8.36
7.48
6.71
6.19
8.16
7.20
QTIP
6.52
5.80
4.21
5.91
5.26
3.78
7.99
7.31
6.56
6.08
7.76
6.99
5.56
GLVQ-8D
6.28
5.64
4.57
4.01
5,69
5.02
3.62
7.83
7.17
6.48
5,94
7.33
6.56
5.25
GLVQ-32D
2
6.00
5.38
4.32
3.81
5,41
4.80
3.36
7.61
6.95
6.22
5,80
7.14
6.33
5.04
Table 2: Zero-shot accuracy (ace in LM Eval, not acc acc_norm) for Llama 2 models.
2-70
2-13
2-7
AreC
PIQA
ArcE
PIQA
AreC
PIQA
WINO
IP16
16
51.1
77.7
81.1
77.0
16
45.6
73.3
73.5
69.6
16
40.0
69.1
78.5
67.3
OmniQ
49.8
75.8
70.2
78.4
37.9
77.1
QulP
47.0
74.3
80.3
76.0
44.9
73.3
79.0
69.7
4
AQLM
51.0
78.1
81.4
76.9
72.2
78.6
70.4
40.3
77.7
67.3
QulPa
50.6
78.1
81.4
77.1
45.5
73.9
78.9
69.9
40.5
69.1
78.4
67.6
QTIP
50.0
77.6
81.5
77.0
45.9
73.2
78.6
69.9
4
40.4
69.2
78.5
67.4
GLVQ-SD
67.8
OmniQ
47.6
75.7
79.7
73.5
42.0
69.0
77.7
65.9
3
35.3
62.6
73.6
63.6
QuIP
46.3
74.6
70.4
76.9
69.9
AQUM M
50.0
77.6
81.3
77.2
43.6
73.5
77.8
67.6
38.7
67.8
76.6
68.4
QulP#
50.9
81.4
76.4
44.0
72.5
78.4
69.1
39.2
68.4
77.3
66.5
QTTP
77.5
76.3
72.5
78.2
GLXQ-SD
50.8
78.0
81.6
77.1
46.5
74.2
79.3
71.0
40.9
69.8
78.8
67.9
OmniQ
28.7
51.5
QulP
2
34.0
62.2
74.8
67.5
23.5
45.2
62.0
$2.8
2
19.4
26.0
S1.6
51.8
AQLM
2
47.9
80.4
75.9
38.5
67.0
75.1
33.6
62.8
73.5
64.6
QulPa
48.7
77.3
80.3
75.9
39.5
69.3
77.3
67.7
2
34.6
64.6
75.1
64.9
QTIP
2
48.1
76.9
80.1
76.5
39.2
70.6
77.8
71.0
2
35.3
63.9
75.3
66.7
GLYQ-SD
functions. The combination of adaptive vector quantization and nonlinear transformation helps
mitigate the quantization error that typically worsens at extreme compression levels. Furthermore, GLVQ-32D consistently outperforms GLVQ-8D, demonstrating that larger lattice dimensions improve quantization fidelity by offering a more expressive codebook structure. This supports our design choice of learning group-specific lattices with adjustable resolution.
4.2 Zero-Shot Accuracy on Llama Models
We report zero-shot accuracy of GLVQ and baseline methods on Llama-2 models under 4-bit, 3-bit and 2-bit quantization across four tasks from LM Eval: ARC-Challenge, ARC-Easy, PIQA, and Winogrande. Results are summarized in Table 2 GLVQ-8D consistently matches or outperforms existing methods across all bit-widths and model sizes. At 4-bit precision, GLVQ-8D achieves the highest accuracy
tasks, e.g., 51.2% ARC-Challenge and 81.6% on PIQA (2-70B), slightly surpassing QTIP and QuIP#. As the bit-width decreases, the performance gap becomes more evident. At 2-bit precision, GLVQ maintains strong performance-c.g e.g. 40.0% ARC-Challenge and 78.0% on PIQA 2-133))ootperformmig QuIP# (39.5 (39.5%, 77.3%) and QTIP 339.2%, 77.8%). On the smallest model (2-7B), GLVQ-8D again leads with the highest scores on all tasks, demonstrating its robustness under extreme compression. These results confirm that GLVQ preserves downstream task accuracy more effectively than existing vector quantization methods, especially low-bit regimes.
4.3 Fractional and Sub-2-Bit Quantization
To assess GLVQ in more extreme compression regimes, we further evaluate its performance under fractional and ultra-low bit-width settings (1.5 and 1.0 average bits per weight), using the same protocol as in the main experiments, Fractional rates are naturally supported GLVQ via groep-wise bit allocation: each group is assigned integer bit-width, and the global rate is the arithmetic mean across groups, allowing any rational target (e.g., 1.5 bit by mixing 1-bit and 2-bit groups). No architectural or algorithmic modification is required to support such mixtures.

---

Table 3: Perplexity (1) of fractional and sub-2-bit quantization on Wikitex12 (Llama-2). GLVQ
achieves strong performance 1.5-bit and remains competitive even at 1.0-bit.
Method
7B
13B
70B
Bits
PPL
A to GLVQ
Bits
PPL
A
Bits
PPL
BiLLM
1.08
48
+24.65
1.10
16.77
+10.66
1.08
8.41
+2.30
One
1.00
9.73
+1.90
1.00
8.76
+1.17
PV-Tuning
1.02
8.28
+0.45
0.97
7.96
+0.37
1.00
6.50
+0.39
GLVQ (ours)
1.00
7.83
0.00
1.00
7.59
0.00
1.00
6.11
0.00
PB- PB-LLM
1.70
69.20
+62.19
1.70
151,09
+144,98
1.70
28.37
+23.38
PV-Tuning
1.58
7.32
+0.31
1.37
6.65
+0.54
1.14
5.52
+0.53
GLVQ (ours)
1.50
7.01
0.00
1.50
6.11
0.00
1,50
4.99
0.00
Table 4: Inference throughput (TOK/s), MATVEC MEM BW (GB/S), and perplexity WikiText2
dataset tested on a NVIDIA RTX 4090 for various methods on 2- Llama 2-7B and 2-70B models. OOM indicates that the method ran out of memory the tested model.
2-7B
2-70B
Method
TOK/S
MEM BW
Perplexity (L)
TOK/S
MEM Perplexity (L)
Original Model
33.1
7.0
OOM
OOM
Scalar Quantization
OmniQuant GPTVQ-ID
125.0
37.4
35.1
7.81
143.5
10.1
40.2
4.82
Vector Quantization
GPTVQ(4D)
24.0
7.22
10.4
4.39
AQLM
20.6
6.59
8.27
3.94
QUIP#
106.3
697 GB/s
6.66
25.9
867 GB/s
4.16
QTIP
105.2
628 GB/s
5.91
25.8
840 GB/s
3.78
GLVQ-
102.3
615 GB/s
5.87
24.9
836 GB/s
3.72
GLVO-32D
100.2
608 GB/s
5.55
24.1
831 GB/s
3.49
GLVQ-8D
88.4
544 GB/s
5.69
20.1
721 GB/s
3.62
GLVQ-32D
82.0
521 GB/s
5.41
18.7
702 GB/s
3.36
Table reports results WikiText2 for Llama-2 1.5 bit and 1.0 bit. At 1.5 bit, GLVQ achieves perplexities of 7.01 (7B), 6.11 (13B), and 4.99 (70B), outperforming PV-Tuning at matched rates and significantly surpassing PB-LLM. Notably, GLVQ approaches the performance of 2-bit baselines despite using substantially fewer bits. Even at the extremely aggressive 1.0 bit regime, GLVQ remains competitive: it attains 7.83 (7B), 7.59 (13B), and 6. (70B), outperforming BiLLM and OneBit by large margins on 7B and 13B and closely matching PV-Tuning at similar budgets.
These findings highlight the robustness of GLVQ's group-specific lattice design and salience-driven allocation under extreme compression, where competing methods degrade substantially. Unlike approaches that depend retraining fine-tuning: GLVQ preserves strong performance even in the ultra-low-rate regime, making a practical solution for deployment in severe memory- and energy-constraine environments.
4.4 Inference Throughput
We measure inference throughput on a single NVIDIA RTX 4090 GPU, using a batch size of and a 2- bit quantization setup for all competing methods. The results summarized in Table We compare both uniform-precisian GLVQ-u) do not use salience-drive bit allocation and instead apply a fixed bit-width across all groups Despite this simplification, GLVQ-8D-u and GLVQ-32D-u achieve throughput on par with QTIP (c.g., 100.2 vs. 105.2 TOK/s on Llama 2-7B), while consistently delivering lower perplexity (c.g.. 5.55 vs. 5.91), demonstrating the effectiveness of group- spup-speecfiic lattice quantization and companding. The full GLVQ variants, which leverage aalience-determine bit allocation, achieve even better perplexity (e.g., 5.41 with GLVQ-32D vs. 5.91 with QTIP) at the cost of moderate decrease in speed (e.g., 82.0 vs. 105.2 TOK/s). The reduction in memory bandwidth usage also confirms improved efficiency (e.g., 521 GB/s vs. 628 GB/s). These results highlight that GLVQ offers flexible trade-off between accuracy and efficiency, and preciipre configuration particularly advantageous for accuracy-sensitive scenarios such as language generation summarization.
and mixed-precision variants of GLVQ. The uniform versions (denoted as

---

4.5
Ablation Studies
To quantify the contribution of each design component in GLVQ, we perform controlled ablations and examine their effects perplexity. All corresponding results are reported in Appendix Tables 6 [3
Bit allocation. We replace the salience-driven bit allocation with uniform bit-width across all groups (Appendix D 6) This forces all groups to share the same precision regardless of their sensitivity. Perplexity increases consistently across model scales and bit-widths; for example, on Llama 1-13B at 2-bit it rises from 5.64 to 5.79, and on Llama 2-70B from 3.62 to 3.72. The gap persists at 3-bit and remains visible even at 4-bit, where the precision constraint is relatively loose. These results confirm that group-aware precision allocation is essential for preserving accuracy when operating under tight bit budgets.
Group-specific lattice codebook. We then enforce a single shared lattice basis across all groups instead of learning group-specific bases (Appendix [.] Table 7 This substitution consistently harms performance, especially under 2-bit quantization (e.g., I.lama 1-13B increases from 5.64 to 5.89; Llama 2-70B from 3.62 to 3.88), and the degradation persists 3- and 4-bit. These findings indicate that different weight groups exhibit distinct geometric statistics, and a shared codebook cannot align to them simultaneously. Group-specific lattice learning therefore necessary to realize the full benefit of lattice quantization.
Group-specific companding, Next, we remove the group-wise μ-laww companding and apply fixed global transformation to all groups (Appendix [F) Table 8). Perplexity increases across all settings, with the strongest effect again observed at low-bit precision (e.g.. Llama 1-13B 2-bit: 5.64 to 5.87; Llama 2-70B: 3.62 to 3.88). This validates the hypothesis that heavy-tailed and skewed weight distributions must be linearized at the group level to reduce quantization error.
Group size. We ablate the number of columns per group using (32, 64, 128, 256, 512) (Appendix G Tables 10). Very small groups (32/64) yield slightly lower perplexity but incur significant code- book/storage overhead, whereas very large groups (256/512) compromise accuracy by averaging out distributional differences within the group. A moderate group size of 128 consistently provides the best balance between the adaptation power and additional overhead and is therefore adopted as the default option in the proposed GLVQ.
Calibration-set size. We vary the number of calibration tokens from 100K to 8M (Appendix H Table TT.. Perplexity improves steadily up to about 4M tokens and then saturates; for instance, Llama 1-13B (2-bit) perplexity is 5.64 at 4M and 5.65 at 8M. Importantly, GLVQ remains competitive with 500K or IM tokens, and degradation is gradual even at 100K. This demonstrates both the data efficiency and the robustness of the method in low-calibration regimes.
Babai rounding vs. GCD. Finally, we replace Babai rounding with greedy coordinate descent (GCD) for index assignment (Appendix[] []] Tables [2 13). GCD consistently yields worse perplexity; for example, on Llama 1-13B at 2-bit the perplexity rises from 5.64 (Babai) to 5.92 (GCD). Zero-shot results on Llama 2 benchmarks show the same pattern. This confirms that Babai rounding offers both better convergence stability and superior final accuracy, and should be preferred for basec quantization.
Conclusion
We have introduced a novel approach, grouped lattice vector quantization (GLVQ), for low-bit mixed- precision quantization of Large Language Models (LLMs). By integrating group-wise bit allocation, lattice codebook learning, and companding transformations, GLVQ achieves superior rate-distortion performance compared to scalar or fixed-pree preeiicciiioon baselines. Our experimental results demonstrate the effectiveness of GLVQ on both language modeling and summarization tasks, highlighting its potential as a practical solution for deploying LLMs in resource-constrained environments. Future research will focus on exploring hierarchical grouping strategies and designing hardware-efficine lattice codebooks.
Limitations future work. GLVQ still assumes a fixed column partition, which may under-utilize layer-wise sensitivity structure; data-driven grouping may improve rate-distortion trade-offs. The current work also compresses only weights; extending latrice-based quantization to activations would unlock further memory and energy savings but is complicated by dynamic activation statistics.
10

---

Acknowledgements
This research is supported by the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award 12301 E0026), administered by STAR,,, well supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global S-Sttainaability CorpLab (ANGEL), This work is also supported in part by the National Natural Science Foundation of China (No.62301313)
References
[1] László Babai. On Lovász' lattice reduction and the nearest lattice point problem. Combinatorica, 6(1):1-13, 1986.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint 3rXiv:2309.166000 2023.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforeement learning from human feedback, arXiv preprint arXiv:2204. 258605 2022.
[4] MA Bhagyaveni, R Kalidoss, and KS Vishvaksenan. Introduction to analog and digital communication. River Publishers, 2022.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems,
2023.
[7] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282. 2017.
[8] John Horton Conway and Neil James Alexander Sloane. Sphere packings, lattices and groups, volume 290. Springer Science & Business Media, 2013.
[9] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318 8-0332, 2022.
[10] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.03077 2023.
[11] Tim Dettmers and Luke Zeulemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pages 7750-7774. PMLR, 2023.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume 1 (long and short papers), pages 4171 171-4186, 2019.
[13] Xin Ding, Xiaoyu Liu, Zhijun Tu. Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, et al. Chq: Cross-block quantization for large language models. arXiv preprini 1 2023.
[14] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118, 2024.
11

---

[15] Uri Erez and Ram Zamir. Achieving 1/2 log (1 + snr) on the awgn channel with lattice encoding and decoding. IEEE Transactions on Information Theory, 50(10):2293-2314, 2004.
[16] Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. Serl: Self-play reinforcement learning for large language models with limited data. arXiv preprint wwXiv:2555..0:44 2025.
[17] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475- 4488, 2022.
[18] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq; Accurate post-training quantization for generative pre-trained transformers. arXiv preprini arXiv:22 10. 1733,, 2022.
[19] Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pages 291-326. Chapman and Hall/CRC, 2022.
[20] Jerry D Gibson and Khalid Sayood. Lattice quantization. In Advances in electronics and electron physics, volume 72, pages 259-330. Elsevier, 1988.
[21] Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Yang Yong. Shiqiao Gu, Haotong Qin, Jinyang Guo, et al. A survey of low-bit large language models: Basics, systems, and algorithms. Neural networks, page 107856, 2025.
[22] Robert Gray. Vector quantization. IEEE Assp Magazine, 12):4-29, 1984.
[23] Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint vxXiv2245..14917 2024.
[24] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International conference on machine learning, pages 4466 4475. PMLR, 2021.
[25] Hisashi Kaneko. A unified formulation of segment companding laws and synthesis of codecs and digital compandors. Bell System Technical Journal, 49(7):1555-1155 1970.
[26] Ahmed Khalil, Robert Piechocki, and Raul Santos- Modros-Rodriguez. Ll-vq-vae: Learnable lattice vector-quantization for efficient representations. arXiv preprint arXiv:2310.09388, 2023.
[27] Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense- andse-and- quantization. arXiv preprint arXiv:2306.07629, 2023.
[28) Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: whitepaper. arXiv preprint arXiv:1806.08342. 2018
[29] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv: 102.05426, 2021.
[30] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device 11m compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
[31] Lucas D Lingle. Transformer-vq: Linear- time transformers via vector quantization. arXiv preprint arXiv:2309. 16354, 2023.
[32] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX4v:2412.19437, 2024.
12

---

[33] Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, and Dacheng Tao. A survey of direct preference optimization. arXiv preprint arXiv:2503.1 1701, 2025.
[34] Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Wendong Fan, Ge Zhang, Jiajun Shi, Weihao Xuan, et al. Verigui: Verifiable long-chain gui dataset. arXiv preprint arXiv:2508. 04026, 2025.
[35] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092-281000 2021.
[36] Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond traight-througe estimation for extreme Ilm compression. Advances in Neural Information Processing Systems, 37:5074-5121, 2024.
[37] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint /00V:v6600007555 2016.
[38] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International conference machine learning. pages 7197-7206. PMLR, 2020.
[39] NVIDIA. CUDA Toolkit. https: avidiaa.comcuuaatoolkii 2020. Ver- sion 10.2.89.
[40] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[41] Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. arXiv preprint ar arXiv:2206.0920 095577, 2022.
[42] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
[45] CCITT Recommendation. Pulse code modulation (pem) of voice frequencies. In ITU, 1988.
[46] Semyon Savkin, Eitan Porat, Or Ordentlich, and Yury Polyanskiy. Nestquant: Nested lattice quantization for matrix products and 1lms. arXiv preprint arXiv:2502.09720. 2025.
[47] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu. Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao. and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprint arXiv:2308. /3137, 2023.
[48] Bernard Smith. Instantaneous companding of quantized signals. Bell System Technical Journal, 36(3):653-709, 1957.
[49] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
13

---

[50] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 1397 2023.
[51] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprins 200V:2240..04396 2024.
[52] Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37:59597-59620, 2024.
[53] YP Tsividis, V Gopinathan, and L Toth. Companding in signal processing. Electronics Letters, 26(17):1331-1332 1990.
[54) Mart Van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for IIm quantization. arXiv preprint wrXiv:2402.15319 2024.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[56] Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning, pages 9847-9856. PMLR, 2020.
[57] Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:1 (6662-116492,, 2024.
[58] Guangxuan Xiao, Ji Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pages 38087-38099, PMLR, 2023.
[59] Hao Xu, Xiaolin Wu, and Xi Zhang. Improving 3d gaussian splatting compression by scene- adaptive lattice vector quantization. arXiv preprint arXiv:2509. / 3382,, 2025.
[60] Hao Xu, Xiaolin Wu, and Xi Zhang, Multirate neural image compression with adaptive lattice vector quantization. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7633 1143-764,,21 2025.
[61] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in neural information processing systems, 35:27168-27188 183, 2022.
[62] Xi Zhang and Xiaolin Wu. Lvqae: Lattice vector quantization coupled with spatially adap- tive companding for efficient learned image compression. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10239-10248, 2023.
[63] Xi Zhang and Xiaolin Wu. Learning optimal lattice vector quantizers for end-to-end neural image compression. In Advances in Neural Information Processing Systems, volume 37, pages 106497-106518. 2024.
[64] Xi Zhang and Xiaolin Wu. Receptive field expanded look-up tables for vision inference: Advancing from low-level to high-level tasks. arXiv preprint
2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction match theoretical and experimental results and accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discussed the limitations of the work: GLVQ currently relies on fixed grouping and is applied only to weights.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: The paper provide the full set of assumptions and a complete and correct proof for all theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes],
The source code will be released once we obtain the necessary approval from Alibaba.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results reported in the paper are averaged across several independent experiments,
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://euurppss.ccocm httpe/rrpppp.cpppupcb//eeeccccce itw.
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed both potential positive societal impacts and negative socictal impacts of the work performed in the appendices.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners of assets (e.g., code, data, models) used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p///earrrppccc/ooooeereeee/ /ct/////oeeeeeccc//22////.. for what should or should not be described.
21

---

Technical Appendices and Supplementary Material
A
A Brief Theoretical Derivation of Babai Rounding's Error Bound
A.1 Algorithm Description
Babai's rounding algorithm is a classical method for finding an approximate closest lattice point to given target vector. Let
B = b.......
..... bn] I=
be a lattice basis and t E R. be the target vector. The algorithm proceeds in three steps:
1. Compute the real coordinate vector: x = B 't. This step expresses the target vector in terms of the lattice basis B.
2. Round each coordinate: integer Gj
[x: + 0.5]. Here, each coordinate x; is rounded to the nearest
3. Return the lattice point: Bc. This gives the approximate closest lattice point.
The error vector e is defined as the difference between the target vector and the approximate lattice point:
= - B(x - c) Bd,
(13)
where 5 = (61,..-;6,,) and |6:| 2 due rounding.
A.2 Gram-Schmidt Orthogonalization
To analyze the error, we first orthogonalize the lattice basis using the Gram-Schmidt process. Let
B bi...... by
be the orthogonalized basis, where each b, can be expressed as:
b, = bj
Hjabj.
(14)
Here,
(b: bj)
Mj. b
represents the projection coefficient of b, onto bz. This decomposition ensures that b is orthogonal all previous bj for j i.
A.3 Error Vector Decomposition
Next, we decompose the error vector e in terms of the orthogonal basis B Starting from the definition of e, substitute the expression for by:
b,b;
Σd; i=1
6i
by
Σ
(15)
j=1
By rearranging the terms, we can rewrite as:
8y +
8sptis
bj.
(16)
Let 0; Ōū + Σi 8114,,1. Then,
a ajb
(17)
22

---

A.4
Norm Calculation
Since the vectors bj are orthogonal, the squared norm of is:
||e12 Σ a 133 ==1
(18)
A.5 Coefficient Bound
To bound |/e]|, we first derive an upper bound for each coefficient oz. Lemma A.I. For any j n, the coefficient oj satisfies:
layl S
Σ
(19)
1=++1
Proof. Using the triangle inequality and the fact that |6,| S 1/2 for all i, we have:
1
1 n
lay1 S 15,1 +
+
Σ
lpsal-
(20)
2
2
1=1+1
A.6 LLL-Reduced Basis Case
When the basis B is LLL-reduced (with parameter 8 33/4), the projection coefficients satisfy:
Maja S
Vj i.
(21)
Thus, for each j,
Σ 2=++1
1p S n
(22)
A.7
Final Error Bound
Combining the above results, obtain:
1+ Σ luyal ))
||||
2
S
||b;1|?
(23)
2
j=l
imj+1
For an LLL-reduced basis, using the bound
| Σi
s n-1
2
have:
1 12
j 2
|e| 2 =
||bع||
(24)
4
2
j=1
Taking the square root of both sides, the final error bound is:
1 |||| s
Σ j=1
+
n -
2
|bb 11.
(25)
2
2
23

---

B Storage Overhead Analysis
For each group g we store the integer codes for Mgng weights (b, bits per weight) and a small amount of side information: d x d FP16 generation matrix plus one FP16 scalar,
Mg
Bytes
Bytessia = 2d + 2.
(26)
The overhead ratio therefore
Bytes 16d + 16
OH2 Bytes (g)
(27)
Representative cases. Table 5 reports the side-information overhead for fixed row-count my 4096 and two typical column widths (ng 1 256), evaluated at lattice dimensions E {8. 16, 32) and bit-widths by {2, 3, 4]. Even in the most demanding configuration (2-bit, 32, ng = 128), the FP16 lattice parameters add only = 1.6% to the weight codes, while the default setting used in our main experiments (d 16, "g 128, 4-bit) incurs less than 0.2% overhead- that the side information is negligible across all practical scenarios.
Observation. Even with the largest lattice (d=32) and the most aggressive 2-bit setting, the side- information overhead is below 1.6%. For the default configuration used in our main experiments (d=16, 1288 44), the overhead is under 0.2%, confirming that the FP16 lattice parameters contribute a negligible addition to the overall model size.
Table 5: Per-group storage overhead (%) with explicit inclusion of the row count mg. Results reported for mig 4096, column widths ng E {128, 256), lattice dimensions d E {8, 16, 32). and bit-widths by E 3, 3, 43.
mg
fg
Overhead (%)
by = 2 / 3
4096
128
0.10
0.07
0.05
8
4096
256
0.05
0.03
0.02
16
4096
128
0.39
0.26
0.20
16
4096
256
0.20
0.13
0.10
32
4096
128
1.56
1.04 0.78
32
4096
256
0.78
0.52 0.39
Values computed via Eq. (27).
C
Broader Impact
Potential positive societal impacts.: (i) Energy and carbon reduction. By enabling 2-4 bit deployment large language models (LLMs) on commodity GPUs or edge SoCs, our method lowers inference energy per token and reduces the overall carbon footprint of Al services. (ii) Edge privacy and accessibility. Supporting on-device inference allows sensitive user data to remain local, strengthening privacy guarantees, and makes advanced language capabilities available on low-resource or offline devices, fostering digital inclusion. (iii) Cost-effective R&D. Lower hardware requirements democratise experimentation for academia, small enterprises, and regions with limited compute infrastructure, broadening participation in AI research and application development.
Potential negative societal impacts.: (i) Easier large-scale misuse. Compressing powerful LLMSs lowers the barrier to deploying them disinformation, spam, or malicious content generation pipelines. (ii) Amplified bias under quantisation. Quantisation can introduce distribution shift that disproportionately affects under-represented dialects or demographic groups, potentially worsening fairness metrics if not carefully audited. (iii) Expanded surveillance capabilities. Running compact LLMs on edge cameras or mobile devices may increase ubiquitous monitoring and privacy intrusion when deployed without appropriate safeguards.
24

---

D Ablation Study of Bit Allocation
We assess the impact of group-wise bit allocation by disabling it and assigning a uniform bit- width across all groups. This prevents GLVQ from adapting to the varying importance of different weight groups, resulting in a measurable drop in accuracy. As shown in Table 3 (see Appendix), removing bit allocation consistently increases perplexity across Llama models and bit-widths. For instance, on Llama 1-13B with 2-bit quantization, perplexity increases from 5.64 to 5.79. The degradation becomes more evident at lower bit-widths, where precision budgets are more constrained. These results confirm that salience-driven bit allocation plays a critical role preserving model accuracy under aggressive compression.
Table 6: Ablation study on salience-determined bit allocation. We compare GLVQ with and without bit allocation across multiple Llama models on Wikitext2 (perplexity 4).
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
GLVQ-8D w/ bit allocation (2-bit)
6,28
5.64
4.57
4.01
5.69
5.02
3.62
GLVQ-8D w/o bit allocation (2-bit)
6.44
5.79
4.65
4.10
5.87
5.19
3.72
GLVQ-8D bit allocation (3-bit)
5.72
5.10
4.11
3.56
5.13
4.48
3.14
GLVQ-&D w/o allocation (3-bit)
5.81
5.18
4.17
3.61
5.21
4.55
3.21
GLVQ-8D w/ bit allocation (4-bit)
5.70
5.08
4.09
3.52
5.01
4.44
3.02
GLVQ-8D w/o bit allocation (4-bit)
5.74
5.13
4.14
3.56
5.08
4.49
3.08
E
Ablation Study of Group-Specifie Lattice Quantization
We evaluate the influence of group-specific lattice learning by replacing each group's learned genera- tion matrix with a fixed lattice basis (shared across all groups). As summarised in Table removing adaptive lattice optimisation consistently increases perplexity for every model size and bit-width. The degradation most evident in the 2-bit setting, where the fixed lattice cannot align with group-wise statistics (c.g.. Llama 1-13B rises from 5.64 to 5,89). Even at 4 bits, shared lattice still hurts accuracy, confirming that the learned, group-specific basis is an essential component of GLVQ.
Table 7: Ablation study on group-specific lattice codebook learning. We compare GLVQ with learned vs. fixed lattice bases across multiple Llama models WikiText-2 (perplexity 1).
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
GLVQ-SD w/ adaptive lattice (2-hit)
6,28
5.64
4.57
4.01
5.69
5.02
3.62
GLVQ-&D w/ fixed lattice (2-bit)
6.53
5.89
4.82
4.26
5.95
5.28
3.88
GLVQ-&D w/ adaptive lattice (3-bit)
5.72
5.10
4.11
3.56
5.13
4.48
3.14
GLVQ-8D 8D w/ fixed lattice (3-bit)
5.87
5.25
4.26
3.70
5.27
4.62
3.28
GLVQ-SD adaptive lattice (4-bit)
5.70
5.08
4.09
3.52
5.01
4.44
3.02
GL.VQ-DD fixed lattice (4-hit)
5.78
5.16
4.17
3.60
5.09
4.52
3.10
F
Ablation Study of Group-Specific Companding
We also examine the contribution of group-specific p-law companding by replacing with fixed transformation across all groups. As shown in Table 4 (see Appendix), removing this component significantly degrades performance, particularly low-bit settings where weight distributions are more diffficult to approximate with uniform quantizers. For example, at 2-bit quantization Llama I-13B, turning off companding raises the perplexity from 5.64 to 5.87. This confirms that companding helps mitigate quantization error by better accommodating heavy-tailed weight distributions. Overall, these ablation results validate the necessity of group group-specific companding as a core component the proposed GLVQ.
25

---

Table 8: Ablation study group-specific j-law companding. We compare GLVQ with and without companding across multiple Llama models on Wikitext2 (perplexity J).
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
GLVQQ-D w/ companding (2-bit)
6.28
5.64
4.57
4.01
5.69
5.02
3.62
GLVQ-8D w/o companding (2-bit)
6.58
5.87
4.71
4.18
5.97
5.23
3.88
GLVQ-8D w/ companding (3-bit)
5.72
5.10
4.11
3.56
5.13
4.48
3.14
GLVQ-8D 8D w/o companding (3-bit)
5.84
5.20
4.19
3.63
5.23
4.58
3.23
GLVQ-8D companding (4-bit)
5.70
5.08
3,52
5.01
4.44
3.02
GLVQ-80 8D w/o companding (4-bit)
5.75
5.14
4.12
3.54
5.06
4.48
3.06
G
Ablation Study of Group Size
vary the number of columns per group {32, 64, 128, 256, 512) to study the trade-off between granularity and overhead (see Tables Mobles9and and 10). Smaller groups (32, 64) offer slightly lower perplexity by adapting more precisely to local weight distributions, but they inflate model size and decoding cost because many lattice codebooks must be stored and reconstructed. Conversely, very large groups (256, 512) minimise codebook overhead yet fail to capture fine-gred lineggrined distributional variation, yielding noticeably higher perplexity- -arr-puuarttlllyy at 2- and 3-bit precision. A group size of 128 consistently provides the best balance, delivering the lowest (or near-lowest) perplexity while keeping storage and runtime overhead moderate; therefore adopt 128 the default in GLVQ.
Table Ablation study of GLVQ with different group sizes on Wikitext2 using Llama 1-7B and 2-7B. We report perplexity (1) under different bit- bittws.ths.
Llama 1-7B (Wikitex12)
Llama 2-7B (Wikitex(2)
Group Size
2-bit
3-bit
4-bit
2-hit
3-bit
4-bit
32
6.26
5.71
5.68
5.65
5.10
4.97
64
6.26
5.72
5.70
5.66
5.13
5.00
128
6.28
5.72
5.70
5.69
5.13
5.01
256
6.34
5.78
5.75
5.84
5.19
5,08
512
6.47
5.85
5.82
5.91
5.28
5.17
Table 10: Ablation study of GLVQ with different group sizes C4 using Llama 1-7B and 2-7B. We report perplexity (L) under different bit bithw..ths.
Llama 1-7B (C4)
Llama 2-7B (C4)
Group Size
2-bit
3-bit
4-bit
2-bit
3-bit
4-bit
32
7.80
7.16
7.04
7.30
6.38
6.28
64
7.81
7.17
7.06
7.32
6.39
6.31
128
7.83
7.17
7.08
7.33
6.40
6.31
256
7.95
7.25
7.15
7.42
6.50
6.39
512
8.14
7.32
7.18
7.51
6.66
6.54
H Ablation Study of Calibration Dataset Size
To evaluate data efficiency, we vary the calibration corpus from 100 K M RedPajama tokens (Appendix Table [T]. Perplexity drops steadily until about 4 M tokens, after which gains saturate an M-token set brings no further improvement and occasionally introduces minor noise. Crucially, GLVQ remains robust with far fewer samples: reducing the set to M or 500 K tokens incurs only marginal increase in perplexity, and even a 100 K-token subset delivers competitive accuracy. These results confirm that GLVQ attains its best performance with roughly 4 M calibration tokens and degrades gracefully when calibration data are searce, underlining practicality in low-resource scenarios.
26

---

Table 11: Perplexity (4) of GLVQ under different calibration set sizes, evaluated Wikitext2 and C4 using Llama models (2-bit quantization, context length 2048). Results show that GLVQ achieves best performance 4M tokens and remains stable across wide range of calibration sizes.
Calib Size
Tokens
Wikitext2
C4
1-7
1-13
1-30
1-65
2-7
2-13
1-13
1-30
1-65
2-7
2-13
2-70
GLVQ-8D
SM
6.30
5.65
4.60
4.02
6.50
5.95
7.34
6.57
5.26
GLVQ-8D
4M
6.28
5.64
4.57
4.01
5.69
5.02
7.83
7.17
6.48
5.94
7.33
6,56
5,25
GLVQ-8D
2M
6.34
5.69
4.63
4.06
5.74
5.07
7.87
7.23
6.54
6.00
7.38
6.61
5,30
GLVQ-8D
1M
6.38
5.73
4.68
4.10
6.59
6.05
7.43
6.67
GLVQ-8D
500K
6,45
5.80
4.74
4.16
5.84
5.19
8.00
7.36
6.67
6.11
7.50
6.74
5,42
GLVQ-8D
100K
6.58
5.94
4.85
4.29
5.97
5.33
8.17
7.52
6.84
6.27
7.67
6.89
5.60
I Babai Rounding vs. Greedy Coordinate Descent (GCD)
We replace Babai rounding with greedy coordinate descent (GCD) and re-train the quantizer (Ap- pendix Tables T2 and 13). GCD exhibits unstable optimisation and poorer final perplexity across all model sizes: for 2-bit Llama 1-13B, perplexity rises from 5.64 (Babai) to 5.92 (GCD), and similar gaps persist at 3- and 4-bit settings. The results underline Babai rounding's 's advantage in both convergence speed and final quality, and we therefore retain it as the default index-assignment method in GLVQ.
Table 12: Comparison GLVQ with Babai Rounding vs. GLVQ with Greedy Coordinate Descent (GCD) Llama & 2. Perplexity (L) is reported on Wikitext2 and C4 for 4-, 3-, and 2-bit quantization.
Method
Bits
Wikitext 2
C4
1-7
1-13
1-30
1-65
2-7
2-13
1-7
1-13
1-30
1-65
2-7
2-13
GLVO-8D-babai
5.70
5.08
4.09
3,52
5,01
4.44
7.08
6.55
5,92
5.58
6.31
5.81
GLVQ-8D-GCD
5.90
5.32
4.28
3.72
5.23
4.65
7.30
6.78
6.18
5.82
6.62
6.04
GLVO-8D-bbbbll
5.72
5.10
4.11
3.56
5.13
4.48
7.17
6.60
5.91
5.58
6.40
5.85
GLVQ-&D-GCD
5.95
5.37
4.36
3.82
5.38
4.72
7.41
6.83
6.13
5.78
6.70
6.08
GLVO-8D-babai GLVQ-8D-GCD
6.28
5.64
4.57
4.01
5.69
5.02
7.83
7.17
6.48
5.94
7.33
6.56
6.72
6.08
5.01
4.45
6.14
5.46
8.30
7.62
6.96
6.39
7.81
7.00
Table 13: Zeroshot Accuracy comparison of GLVQ with Babai Rounding vs. GLVQ with Greedy Coordinate Descent (GCD) on Llama
2-13
2-7
Method
Bits
ArcC
ArcE
PIQA
WINO
Bits
ArcC
ArcE
PIQA
WINO
Original Model
16
45.6
73.3
73.5
69.6
16
40.0
69.3
78.5
67.3
GI-VO-8D-bahai
4
46.0
74.0
79,2
70.5
4
40.7
69.5
78.6
67.8
GLVQ-&D-GOE
45.3
73.4
78.4
69.8
4
39.9
68.9
77.8
67.0
GLVO-8D-bull
3 46.5
74.2
79.3
71.0
3
40.9
69.8
78.8
67.9
GLVQ-8D-GCU
45.7
73.5
78.5
70.2
3
40.0
69.0
77.9
67.1
GLVQ-8D-bbbbu GLVQ-8D-GCD
40.0
70.8
78.0
71.3
2
35.5
64.5
75.5
67.2
38.9
69.4
76.7
70.0
2
34.0
63.2
73.9
66.0
27


Reviews:
[{"review_id": "h6Vn51Em8H", "reviewer": "Reviewer_Zc3F", "summary": "The submitted work proposes a lattice-based vector quantization scheme applicable post training to LLMs. It claims to work in particular very well with low number of bits per weight.", "questions": "* As mentioned under \"Strengths & Weaknesses\" there are more VQ techniques available and I strongly suggest to add those to the comparison (NestQuant, VQ-LLM, etc.). Not only, but in particular, a comparison with a non-PT VQ tecnique like VQ-LLM seems to be very interesting in contrast.\n\n* Lines 29-30 claims that PTQ has difficulties in achieving ultra-low bit-widths. As your technique promises (and also proves) to perform better than other methods, it would be good to see performance extended to below 2 bits-per-weight. NestQuant also supports fractional bit ratios and VQ-methods in general suggest that. Therefore it would be a great addition to the paper to also see a sweep over a larger range of bits-per-weight, e.g. 0.5 to 4 including fractional settings (I think this should happen due to calculating the average # of bits per weight anyway).\n\nNOTE: addressing these two main weaknesses properly would raise the overall rating from 4 to 5.\n\n* Figure 1: the two matrix configurations for \"group-specific companding functions\" and \"group-specific lattice codebooks\" are not different. If understood correcetly, it may be better to start with different shades in the full matrix to the left, carry those slices to the middle before companding and then show the companded shades to the right, then being visually different to those before companding. Is there a better way to visualize the selection of the lattice on the right?\n\n* Lines 153-156: does the bit budget constraint actually lead to a knapsack problem chosing the optimal bit allocation for group matching the budget?\n\n* Lines 261-265: selecting only 8D and 32D configurations seems arbitrary and Tables 1 & 2 in addition show little variance in results. Please choose and add additional extreme corner cases like 2D, 3D or 4D to show \"some\" effect.", "limitations": "Yes, there's an Appendix (A.3) addressing these and the inclusion of the Appendix certainly adds to the quality of the submission.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The strength of this work is the solid idea that seems to work well according to the experimental results.\n\nSome weaknesses are in clarity and the scientific depth. As the main idea (application of lattice-based VQ) is admittently obvious it would be good to cover that topic in more depth. There are other PT lattice-based approaches, e.g. NestQuant: https://arxiv.org/abs/2502.09720, and also non-PT lattice-based approaches, e.g. VQ-LLM: https://arxiv.org/abs/2503.02236 to compare with.\n\nAnother weakness is that there are only integer number of bits per weight. This is hard to believe as Figure 1 shows different bit budgets per group and it is expected that the average bits per weight is fractional.", "was_revised": true, "final_justification": "Based on the very detailed rebuttal I'm increasing individual ratings and the overall rating by 1 point.", "rebuttal": "----\n### **[W1 & Q1, Comparison with NestQuant, VQ-LLM]**   \nWe thank the reviewer for highlighting recent lattice-based quantization works, including *NestQuant* and *VQ-LLM*. We respectfully clarify that these two papers play fundamentally different roles in the LLM quantization landscape.  \n(i) **NestQuant** is a lattice-based post-training quantization (PTQ) method that constructs nested lattice structures for compressing both weights and activations in LLMs. It is directly comparable to GLVQ in terms of quantization performance and perplexity.  \n(ii) **VQ-LLM**, in contrast, is not a quantization algorithm, but rather a high-performance kernel generation framework for accelerating inference on already vector-quantized models (e.g., those produced by QuIP#, CQ, or GPTVQ). It focuses on efficient dequantization and computation fusion, and does not propose a new quantization method.\n\nIn the revised version, we will include a discussion of both works to clarify their respective scopes. To enable a direct comparison with NestQuant, we evaluated GLVQ on Llama-2 models (7B, 13B, and 70B) under 4-bit quantization using the WikiText2 benchmark. The results, summarized in Table 1, show that GLVQ consistently improves perplexity over NestQuant by 0.3--0.5 points across all model sizes. These results demonstrate the effectiveness of GLVQ’s group-specific, learnable lattice design in achieving stronger quantization performance than fixed nested lattices at low bit-widths.\n\nTable 1. Perplexity ↓ Comparison with NestQuant on WikiText2 (Llama-2).\n| Method         | Bits | Llama-2-7B | Llama-2-13B | Llama-2-70B |\n|----------------|------|------------|-------------|-------------|\n| NestQuant      | 4    | 5.53       | 4.93        | 3.38        |\n| GLVQ (ours)    | 4    | 5.01       | 4.44        | 3.02        |\n\n\n----\n\n\n### **[W2 & Q2, Fractional bit-widths and below 2 bits setting]**   \nWe appreciate the reviewer’s suggestion to evaluate GLVQ under fractional and ultra-low bit-width settings.\n\n**Fractional bit-widths are fully supported in our framework.** While our main paper reports results at integer average budgets (e.g., 2, 3, 4 bits per weight) for comparability with prior work, GLVQ’s salience-driven bit allocation (SDBA) operates at the group level. Each group is assigned an integer bit-width $b_g \\in \\{1, 2, 3, 4\\}$, and the global bit-rate is given by the average $\\bar{b} = \\frac{1}{G} \\sum_{g=1}^G b_g$. This allows us to realize any fractional target $b_{\\text{avg}}$ (e.g., 1.5) by mixing different group-wise precisions (e.g., 50% 1-bit and 50% 2-bit). Importantly, the design of our codebooks and decoding process is invariant to such mixtures and requires no modification.\n\n**New experiments at fractional and sub-2-bit settings.** To address the reviewer’s concern, we conducted additional experiments at 1.5 and 1.0 average bits per weight, using the same protocol as in the main paper. Results on WikiText2 for Llama-2 models are shown below:\n\nTable 2. Perplexity ↓ results of GLVQ and competing methods at fractional and sub-2-bit settings on WikiText2 (Llama‑2).\n\n| Method        | Bits | PPL (7B) | Bits | PPL (13B) | Bits | PPL (70B) |\n|---------------|------|----------|------|-----------|------|-----------|\n| BiLLM         | 1.08 | 32.48    | 1.10 | 16.77     | 1.08 | 8.41      |\n| OneBit        | 1.00 | 9.73     | 1.00 | 8.76      | --   | --        |\n| PV-Tuning     | 1.02 | 8.28     | 0.97 | 7.96      | 1.00 | 6.50      |\n| **GLVQ (ours)** | 1.00 | **7.83** | 1.00 | **7.59**   | 1.00 | **6.11**   |\n| PB-LLM        | 1.70 | 69.20    | 1.70 | 151.09    | 1.70 | 28.37     |\n| PV-Tuning     | 1.58 | 7.32     | 1.37 | 6.65      | 1.14 | 5.52      |\n| **GLVQ (ours)** | 1.50 | **7.01** | 1.50 | **6.11**   | 1.50 | **4.99**   |\n\nThese results confirm that GLVQ naturally supports fractional and ultra-low bit-width quantization. At an average of 1.5 bits per weight, GLVQ achieves perplexities of 7.01 (7B), 6.11 (13B), and 4.99 (70B), consistently outperforming PV-Tuning and significantly surpassing PB-LLM by large margins. Notably, GLVQ approaches the performance of 2-bit baselines despite using substantially fewer bits.\n\nEven at an aggressive 1.0 bit-per-weight budget, GLVQ remains competitive, achieving perplexities of 7.83 (7B), 7.59 (13B), and 6.11 (70B). This represents a clear improvement over prior 1-bit methods such as BiLLM and OneBit, and is on par with or better than PV-Tuning at similar bit rates. Compared to BiLLM, GLVQ reduces perplexity by over 24 points on 7B and over 10 points on 13B under similar bit budgets.\n\nThese findings highlight the robustness of GLVQ’s group-specific lattice design and salience-driven bit allocation strategy. Unlike methods that require calibration or fine-tuning, GLVQ achieves strong performance even in the ultra-low-bit regime, making it a compelling solution for extreme compression scenarios.\n\n\n----\n\n\n### **[Q3, Better figure configuration]**   \nThank you for pointing out the visual ambiguity in Fig.~1. In the current version, the three sub-matrices (\"before companding,\" \"after companding,\" and \"lattice-quantized\") are rendered using the same color scheme, which indeed makes it difficult to visually distinguish the effects of *group-specific companding* and *group-specific lattice quantization*.\n\n**Planned revision.**  \nWe will revise Fig.\\,1 into a more intuitive and visually informative pipeline illustration that clearly differentiates each transformation stage:\n\n1. **Original weights.**  \n   The full weight matrix will be shown with *distinct color hues* for each group. Groups with heavier-tailed distributions will use darker shades to visually convey statistical diversity across groups.\n\n2. **Group-specific companding.**  \n   Each group is processed through its corresponding companding function $F_{\\mu_g}$. The output matrix will be recolored using a *unified pastel palette*, emphasizing that group-wise dynamic ranges have been normalized post-companding.\n\n3. **Group-specific lattice quantization.**  \n   For each group, we will overlay its corresponding lattice grid, and include a small 2D inset that visualizes the shape of the learned generator matrix $G_g$, with axes scaled and rotated differently per group. The quantized outputs will be shown as discrete markers, clearly distinguishable from the continuous pre-quantized values.\n\nThis redesigned figure will leverage both color (distributional contrast, companding effect) and spatial structure (lattice overlay, generator insets) to clearly communicate the impact of group-specific companding and the subsequent lattice selection. We believe this will fully address the reviewer’s concern and significantly improve the figure’s explanatory clarity.\n\n----\n\n### **[Q4, knapsack problem]**   \nAt first glance, selecting per-group bit-widths $b_g$ under a global bit budget may resemble a discrete knapsack problem. However, our setting introduces additional structure that enables a highly efficient and *closed-form solution*:\n\n1. **We restrict each group’s bit-width to a small discrete set:** $\\(b_g \\in \\{N{-}1,\\,N,\\,N{+}1\\}\\)$, and enforce a symmetric constraint as defined in Eq.~10. This ensures the average bit-width remains exactly $N$.\n\n2. **Each group is associated with a scalar salience score** $\\mathcal{S}_g$, reflecting its relative importance. By sorting all $G$ groups by salience $O(G\\log G)$, we obtain the optimal assignment directly: the top $k$ groups receive $N{+}1$ bits, the bottom $k$ groups receive $N{-}1$ bits, and the remaining groups receive $N$ bits. This selection mechanism follows the double-pointer scheme introduced in Slim-LLM.\n\nBecause the bit choices are limited and symmetrically constrained, the search space reduces to identifying a single threshold index, rather than solving a general 0--1 knapsack. The overall algorithm runs in $O(G\\log G)$, is deterministic, and yields an allocation that is provably optimal under our model.\n\n----\n\n### **[Q5, Extreme corner cases GLVQ-2D & -4D]**   \nWe appreciate the reviewer’s suggestion. In the original submission, we focused on 8D (canonical \\(E_8\\)) and 32D (stacked \\(E_8\\) blocks) because they (i) align with SIMD register widths on modern GPUs and (ii) empirically offered a strong trade-off between accuracy and speed. That said, we agree that including smaller lattice dimensions provides better insight into how dimensionality impacts performance.\n\nWe have now added evaluations at 2D and 4D under identical settings. Results are shown in Table 1.\n\nTable 3. Perplexity ↓ of competing methods on WikiText2 for Llama-1 and Llama-2.\n| **Method**  | **Bits** | **1-7** | **1-13** | **1-30** | **1-65** | **2-7** | **2-13** | **2-70** |\n|-------------|----------|---------|----------|----------|----------|---------|----------|----------|\n| OmniQ       | 2        | 15.5    | 13.2     | 8.71     | 7.58     | --      | --       | --       |\n| QuIP#       | 2        | 6.86    | 5.97     | 5.02     | 4.36     | 6.19    | 5.35     | 3.91     |\n| QTIP        | 2        | 6.52    | 5.80     | 4.83     | 4.21     | 5.91    | 5.26     | 3.78     |\n| GLVQ-2D     | 2        | 6.65    | 5.92     | 4.96     | 4.32     | 6.05    | 5.38     | 3.84     |\n| GLVQ-4D     | 2        | 6.48    | 5.75     | 4.72     | 4.12     | 5.84    | 5.18     | 3.70     |\n| GLVQ-8D     | 2        | 6.28    | 5.64     | 4.57     | 4.01     | 5.69    | 5.02     | 3.62     |\n| GLVQ-32D    | 2        | 6.00    | 5.38     | 4.32     | 3.81     | 5.41    | 4.80     | 3.36     |\n\nThese results reveal a consistent trend: increasing the lattice dimension leads to better perplexity across all model sizes. Compared to QTIP, GLVQ-2D underperforms slightly, while GLVQ-4D achieves comparable accuracy. From 4D to 8D, perplexity improves noticeably, and the 32D variant provides further gains with diminishing returns."}, {"review_id": "9r90OfB90d", "reviewer": "Reviewer_4RXG", "summary": "This work proposes learnable lattice quantization for LLM compression, assigning a specific learnable matrix to each group of weights. Babai rounding is adopted as an approximate solution for the closest lattice point search problem. To enhance performance, the authors adopt a pointwise transformation and saliency-driven mixed-precision quantization. The efficacy of the proposed GLVQ method is tested on models from the Llama-½ family, for 2-, 3-, and 4-bit compression targets.", "questions": "- Since GLVQ is a method for optimizing per-layer output error, it would be interesting to compare its output error with the baselines (AQLM, QuIP#, QTIP). QTIP was claimed to be close to the lower error bound achievable from information-theoretical principles. How close is GLVQ?\n- Lattice quantization with Babai rounding can be thought of as an affine transformation followed by rounding to the nearest value (i.e., scalar quantization in a transformed basis). Why should this be more expressive than vector quantization, which can span an arbitrary shape?\n- Which framework is used for inference throughput measurements? Is it Hugging Face Transformers? The inference results depend heavily on the specific choice of framework (HF, vLLM, SGLANG). Which version of the library and what context length are used to produce the numbers in Table 3?\n- The provided list of benchmarks may not be sufficient for a reliable assessment of compressed model performance. I would suggest adding generative evaluation for Llama-3, for example mmlu_cot_llama and gsm8_llama from [1].\n- How large is the overhead caused by applying the μ-law transformation to weights? It appears that this function must be applied to each weight entry at every decoding step (i.e., O(d_ou td_in) applications of a non-linear function, where d_out and d_in are the output and input dimensions of the weights, respectively), which may incur significant computational overhead.\n\n[References]\n- [1] https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/llama3", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The overall approach is sound and well-motivated. The specific design for learning lattices and weight reparametrization appears to be novel in the context of LLM compression.\n- The achieved speed-up is quite decent.\n\n**Weaknesses**\n\n- There seems to be an issue with the evaluation procedure. One would expect that a PTQ-quantized model can at most match the uncompressed model in terms of quality. However, the provided results suggest that for some compression levels (e.g., 4-bit compression), the GLVQ-compressed model may significantly outperform the uncompressed one. Specifically, the 4-bit quantized Llama-2-7B with GLVQ-32D reportedly achieves the same perplexity (Table 1) as the almost 2x larger Llama-2-13B. Such an outcome is extremely unusual and indicates a possible flaw or inconsistency in the comparison between different methods and the fp16 baseline.\n- In the caption of Table 2, it is claimed that Llama-2 uses a context length of 4k for evaluation. However, the fp16 baseline results seem to correspond to a 2k context length (see [1]). For a 4k context length, the baseline result should be 5.12 for Llama-2-7b, for instance [2].\nThe reported improvement on 0-shot benchmarks from lm-eval-harness also looks suspicious. It's important to note that results are sensitive to the specific evaluation package used. Which one was adopted to produce the numbers? One must ensure it is the same as the one used for the baselines.\n- The evaluation is conducted only on rather outdated Llama-½ models. I would suggest additionally evaluating a more recent model family, such as Llama-3, Qwen-2.5, or Qwen-3. There are baselines [3], [4] that provide evaluation results on Llama-3 models.\nGLVQ can produce models with different bitwidths. However, this is not described anywhere in the method formulation. Do I understand correctly that the value zi in Equation 6 is clamped to some qmin⁡, qmax⁡ (with qmin⁡=−2bits−1, qmax⁡=2bits−1, for instance)? One should elaborate on this further.\n- L41-42: However, this approach has the drawback that decoding requires a lookup operation, which is computationally more expensive than QuIP#. Both AQLM and QuIP# require a lookup operation in the decoding phase. The difference is that the AQLM codebook is arbitrary, while QuIP# has a specific parameterization.\n- The speed-up numbers for AQLM cited in Table 3 are incorrect. AQLM achieves throughput up to 114.1 tok/s (see Table 14 in [2]) on an RTX 3090, which is slower than the RTX 4090.\n\nOverall, I appreciate the method, but the evaluation procedure has to be fixed as the current experimental results look implausible.\n\n[References]\n- [1] Tseng, Albert, et al. \"Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks.\" arXiv preprint arXiv:2402.04396 (2024).\n- [2] Egiazarian, Vage, et al. \"Extreme compression of large language models via additive quantization.\" arXiv preprint arXiv:2401.06118 (2024).\n- [3] Tseng, Albert, et al. \"Qtip: Quantization with trellises and incoherence processing.\" Advances in Neural Information Processing Systems 37 (2024): 59597-59620.\n- [4] Malinovskii, Vladimir, et al. \"Pv-tuning: Beyond straight-through estimation for extreme llm compression.\" Advances in Neural Information Processing Systems 37 (2024): 5074-5121.", "was_revised": true, "final_justification": "After reviewing the responses addressed to both myself and the other reviewers, I have decided to raise my score to Borderline accept.\n\nThe method is sound, and the claimed performance numbers for extreme quantization appear to be quite strong. While I still have some concerns—namely, a possible mismatch between the `lm-eval` versions used for the baselines and the proposed method—I believe this paper deserves acceptance.", "rebuttal": "----\n### **[W1 & W2, Issue with evaluation procedure]**  \nFirst, We thank the reviewer for catching citation discrepancy. The reviewer is correct: in the original manuscript, we inadvertently cited FP16 baseline results for Llama-2 under a 2k context length instead of the correct 4k context used during evaluation. We have now corrected this in the revised version. \nImportantly, we emphasize that this correction does **not** affect the results or conclusions of our method (GLVQ). \n\nBesides, it is indeed uncommon for a post-training quantized (PTQ) model to outperform its original FP16 baseline. However, such outcomes are not unprecedented and can occur under certain conditions, as observed in prior work. For example, Table~6 in the QTIP paper (NeurIPS’24) shows that several 4-bit quantized LLaMA checkpoints achieve higher zero-shot accuracy than their FP16 counterparts. This has been attributed to mild regularization effects from quantization, or improved alignment between quantizer and inference distribution.\n\n**Fine-tuning explains small improvements.**  \nIn our GLVQ pipeline, a *lightweight fine-tuning step* is optionally performed *after* quantization. This step fine-tunes only the group-specific lattice generation matrices \\(G_g\\) and companding parameters \\(\\mu_g\\), using a small external corpus of ~8M tokens. This fine-tuning allows the quantizer to better adapt to actual inference-time statistics and introduces a regularization-like effect, sometimes yielding perplexity *slightly* better than FP16. \n\n**GLVQ without fine-tuning still outperforms.**  \nTo ensure that our main gains are not solely due to the optional refinement, we also evaluate a pure PTQ variant of our method, named GLVQ-w/o-ft, where no fine-tuning is applied post-quantization. The table below shows that GLVQ-w/o-ft still consistently outperforms recent state-of-the-art PTQ methods at 2–4 bits precision.\n\nTable 1. GLVQ results with and without fine-tuning on WikiText2.\n| **Method**            | **Bits** | **1-7** | **1-13** | **1-30** | **1-65** | **2-7** | **2-13** | **2-70** |\n|------------------------|----------|---------|----------|----------|----------|---------|----------|----------|\n| Fp16                  | 16       | 5.68    | 5.09     | 4.10     | 3.53     | 5.12    | 4.57     | 3.12     |\n| QTIP                  | 4        | 5.72    | 5.15     | 4.15     | 3.58     | 5.17    | 4.61     | 3.16     |\n| GLVQ-32D-w/o-ft       | 4        | 5.70    | 5.10     | 4.12     | 3.51     | 5.10    | 4.55     | 3.16     |\n| GLVQ-32D           | 4        | 5.55 | 4.94 | 3.95 | 3.39 | 4.88 | 4.31 | 2.98 |\n| QTIP                  | 3        | 5.85    | 5.24     | 4.26     | 3.68     | 5.29    | 4.71     | 3.26     |\n| GLVQ-32D-w/o-ft       | 3        | 5.75    | 5.14     | 4.18     | 3.61     | 5.19    | 4.66     | 3.21     |\n| GLVQ-32D           | 3        | 5.58 | 4.98 | 4.00 | 3.42 | 4.99 | 4.34 | 3.02 |\n| QTIP                  | 2        | 6.52    | 5.80     | 4.83     | 4.21     | 5.91    | 5.26     | 3.78     |\n| GLVQ-32D-w/o-ft       | 2        | 6.18    | 5.55     | 4.51     | 3.97     | 5.58    | 5.00     | 3.55     |\n| GLVQ-32D           | 2        | 6.00 | 5.38 | 4.32 | 3.81 | 5.41 | 4.80 | 3.36 |\n\n----\n\n### **[W3, New results on Llama-3]**  \nWe have reproduced our pipeline on *Llama‑3‑8B* and *Llama‑3‑70B* under the same 2–4 bit settings on WikiText2. Results are summarized below and will be added to the revised paper.\n\nTable 2. Perplexity ↓ results on WikiText2 of Llama‑3.\n| Method        | Bits | Llama-3-8B | Llama-3-70B |\n|---------------|------|------------|-------------|\n| BF16          | 16   | 5.54       | 2.59        |\n| QUIP#         | 4    | 5.81       | 2.99        |\n| QTIP          | 4    | 5.67       | 2.75        |\n| GLVQ (ours) | 4    | 5.58   |  2.69   |\n| QUIP#         | 3    | 6.27       | 3.59        |\n| QTIP          | 3    | 6.01       | 3.18        |\n| GLVQ (ours) | 3    | 5.86   | 3.02    |\n| QUIP#         | 2    | 7.84       | 5.77        |\n| QTIP          | 2    | 7.33       | 4.97        |\n| GLVQ (ours) | 2    | 7.00   | 4.61    |\n\n----\n\n### **[W5, Speed-up numbers for AQLM]**  \n Our Table 3 cited the value **20.6 TOK/s** for “AQLM (2‑bit, LLaMA‑2‑7B, RTX 4090)” based on Table 7 of the QTIP arXiv v1 paper (page 8), which referred to the same experimental setup. However, the original AQLM paper (arXiv v1) did not report throughput; this was added later in arXiv v3.\n\nIn arXiv v3, Table 14 reports two throughput numbers on RTX 3090: 65.3 TOK/s and 114.1 TOK/s. The latter corresponds to a faster variant using a different kernel, but the 65.3 TOK/s figure matches the exact configuration used to report perplexity and accuracy. Therefore, the 65.3 number is the appropriate one for comparison under consistent conditions.\n\nTo resolve this ambiguity, we have re-run the official AQLM code on our own RTX 4090 setup using the same batch size and context length (batch = 1, sequence length = 2048) as all other methods in Table 3. Under this unified setting, AQLM achieves 91.7 TOK/s.\n\n\n----\n\n\n### **[Q1, Output error]**  \nWe would like to offer the following points to clarify GLVQ’s relationship to the information-theoretic lower bound:\n\nFirst, GLVQ is explicitly designed to reduce group-wise reconstruction error through the use of learnable lattices and companding parameters. This aligns closely with QTIP’s goal of minimizing layer-wise distortion, though our method achieves this via group-specific vector quantization rather than Hessian-weighted scalar quantization.\n\nSecond, although we do not report NMSE explicitly, GLVQ consistently achieves better perplexity than QTIP in all 2-bit settings. Since previous works (including QTIP) have established a strong empirical correlation between NMSE and perplexity, this improvement indirectly suggests that GLVQ attains comparable or lower reconstruction error at the output level.\n\n----\n\n### **[Q2, Lattice quantization]**  \nIndeed, given a full‑rank generation matrix $G_g \\in \\mathbb{R}^{d \\times d}$ and Babai rounding, GLVQ performs a *scalar* quantization in the *latent* orthonormal basis $G_g^{-1}$.  \nHowever, two properties make this seemingly simple scheme highly expressive in practice:\n\n1. **Learnable affine map.**  \n   By optimising $G_g$ per group we can  \n   (a) *rotate* the data so that major covariance directions align with lattice axes, and  \n   (b) *scale* each axis independently.  \n   The resulting Voronoi cell is a *parallelotope* whose orientation and aspect ratio are free parameters—already a rich family covering the optimal vector quantizers for any *elliptically contoured* distribution (Whitened Gaussian, Laplacian, etc.).\n\n2. **Non‑linear companding.**  \n   We further learn a monotone companding function $F_{\\mu_g}$ that redistributes probability mass towards the centre of the lattice. The overall mapping:\n\n   $$\n   \\mathbf{w} \\xrightarrow{F_{\\mu_g}} \\mathbf{y} \\xrightarrow{G_g^{-1}} \\tilde{\\mathbf{y}} \\xrightarrow{\\text{round}} \\mathbf{z}\n   $$\n\n   is *piece‑wise affine*; its pre‑images can approximate arbitrarily skewed or heavy‑tailed densities with *vanishing codebook overhead* (the lattice is infinite).\n\n\n**Why not arbitrary VQ?**  \nFree‑form vector quantization can in theory tile any convex region, but comes with *exponential* codebook growth ($\\sim 2^{bd}$ entries), $O(d)$ lookup latency, and large table storage.  \nGLVQ retains **$O(d^2)$ decode cost** and **$O(d^2)$ parameters** per group, independent of bitrate, while still matching the local covariance structure; empirically this captures the vast majority of quantization gain, as Table 2 shows GLVQ within $0.08$ PPL of VQ‑LLM (free codebook) yet decoding $4.8\\times$ faster.\n\n\n**Formal perspective.**  \nGiven any continuous distribution with covariance $\\Sigma$, there exists a scaling matrix $A$ such that:\n\n$$\nA \\Sigma A^\\top = I.\n$$\n\nChoosing $G_g = A^{-1}$ reduces the optimum $d$‑dimensional VQ problem to axis‑aligned scalar quantization *without increasing distortion* (see Gersho & Gray, Ch. 6).  \nThus, lattice + Babai is *universally rate‑distortion optimal up to a constant gap* while remaining computationally tractable.\n\n----\n\n### **[Q3, Inference framework]**  \n Hugging face. Context length 2048.\n\n----\n\n### **[Q5, overhead caused by mu-law transformation]**  \nThank you for raising this important point. While the $\\mu$-law inverse transform $F^{-1}_{\\mu_g}$ is indeed non-linear, its practical overhead is negligible due to (1) its limited scope, and (2) its tight integration with GPU-friendly fused kernels.\n\nFirst, $F_{\\mu_g}$ is only applied during quantization time. At inference time, only the inverse mapping\nis used to decode the dequantized weights just before computing the matrix–vector product.\n\nIn GLVQ, weights are decoded group-by-group, and each group corresponds to a matrix of shape $4096 \\times 128$ (i.e., one transformer layer's weight tile). Therefore, each application of $F^{-1}_{\\mu_g}$ processes exactly $4096 \\times 128 = 524{,}288$ FP16 numbers. This decoding is executed once per group, after which the full group matrix is reused in multiple GEMV calls.\n\nTo assess the actual cost, we conducted timing measurements on LLaMA‑2‑7B at 2‑bit precision using an RTX 4090. The table below shows the result of ablation by replacing $F^{-1}_{\\mu}$ with the identity function.\n\nTable 3. Impact of $\\mu$-law inverse on decoding throughput (LLaMA‑2‑7B, 2-bit, RTX 4090).\n\n| Variant                                          | Measured Token Rate (TOK/s) ↑ | Relative Overhead |\n|--------------------------------------------------|-------------------------------|--------------------|\n| GLVQ (with $F_{\\mu}^{-1}$)                       | 88.4                          | --                 |\n| GLVQ (identity instead of $F_{\\mu}^{-1}$)        | 90.1                          | 1.9%               |"}, {"review_id": "hA0zpzH2e2", "reviewer": "Reviewer_UCYr", "summary": "The paper introduces Grouped Lattice Vector Quantization (GLVQ), a post-training quantization (PTQ) framework that assigns each weight group in a large language model (LLM) a learned, group-specific lattice codebook and a µ-law companding transform.", "questions": "Could you quantify the additional GPU hours required for lattice learning versus, e.g., GPTQ or AWQ? This would help practitioners decide whether the accuracy gains justify the extra training time.\n\nBaseline Comparisons Why were important recent methods like OmniQuant and CBQ not included in the experimental comparisons?\nCBQ Ding, X., Liu, X., Tu, Z., Zhang, Y., Li, W., Hu, J., ... & Wang, Y. (2023). Cbq: Cross-block quantization for large language models. ICLR 2024\n\n\nError Bars: Would you be willing to add confidence intervals (e.g., 95 %) to your perplexity and accuracy tables? This would strengthen the empirical claims.\n\nActivation Quantization: Do you foresee any obstacles to extending GLVQ to quantize activations or KV-cache? If so, how might they be addressed?\n\nHardware Efficiency: Beyond memory bandwidth, have you measured energy consumption or latency on edge devices (e.g., mobile SoCs)?\n\nEthical Safeguards: Given the risk of compressed LLMs being misused for disinformation, would you consider adding a “Responsible Use” section or recommending model-card disclosures?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "GLVQ achieves the best published perplexity at 2–4 bits on Llama 1/2, closing the gap to FP16 by up to 0.4 perplexity points at 2 bits.\n\n\nNo real-world application (e.g., on-device summarization) is demonstrated beyond standard benchmarks.", "was_revised": true, "final_justification": "I keep my scores.", "rebuttal": "----\n### **[Q1, Training GPU hours]**  \nWe measured wall-clock quantization time using a single NVIDIA A100 80GB under identical software environments for all methods: GLVQ, GPTQ, QuIP#, and AQLM.\n\nTable 1. End-to-end quantization time on a single A100 GPU.\n| **Model**       | **GPTQ**      | **QuIP#**     | **AQLM**     | **GLVQ (ours)** |\n|------------------|---------------|---------------|--------------|-----------------|\n| LLaMA-2-70B     | 2.8 GPU hours | 10 GPU hours  | 14 days      | 1 day           |\n\nGLVQ completes 70B quantization in approximately one day on a single A100. While this is slower than GPTQ and QuIP#, it is over 10× faster than AQLM. The added cost mainly comes from optimizing small (8×8) lattice transformations for each weight group, which are trivially parallelizable.\n\nThis one-time conversion cost yields strong returns: at 2-bit, GLVQ improves WikiText2 perplexity by 0.6–1.1 points compared to GPTQ and AWQ, while maintaining comparable throughput. Since quantization is performed offline, these accuracy gains benefit all downstream inference.\n\nThe method scales efficiently. When run on 8 A100s, GLVQ quantizes LLaMA‑2‑7B in just 25 minutes, enabling practical deployment for large models. Furthermore, reducing the number of lattice iterations from 12 to 4 can shorten quantization time by up to 40% with only a small accuracy drop (typically 0.05–0.07 perplexity), providing a useful knob for compute-constrained settings.\n\nWe will include the new results and this discussion in the revised paper to clarify the time–accuracy tradeoff of GLVQ.\n\n\n----\n\n### **[Q2, Baseline Comparisons]**  \nWe thank the reviewer for highlighting recent methods such as OmniQuant and CBQ.\n\nOur primary focus in this work is to evaluate post-training *vector quantization* (VQ) methods. Accordingly, we benchmarked GLVQ against representative VQ-based baselines, AQLM, QuIP#, and QTIP, which adopt codebook-based, groupwise quantization strategies.\n\nOmniQuant was already included in our original experiments, and its results are reported in Table 1 of the main paper. CBQ, on the other hand, performs non-uniform *scalar* quantization with cross-block error correction. While originally considered orthogonal to our focus on VQ methods, we appreciate the reviewer’s suggestion and have conducted additional experiments to compare GLVQ with CBQ at 2-bit and 4-bit settings under the same evaluation protocol (WikiText2, LLaMA‑1‑30B and 65B).\n\nTable 2. Comparison with CBQ on WikiText2 (LLaMA‑1).\n| Method        | Bits | LLaMA‑1‑30B | LLaMA‑1‑65B |\n|---------------|------|-------------|-------------|\n| CBQ           | 4    | 4.14        | 3.59        |\n| **GLVQ (ours)** | 4    | **3.95**    | **3.39**    |\n| CBQ           | 2    | 5.58        | 5.25        |\n| **GLVQ (ours)** | 2    | **4.32**    | **3.81**    |\n\nAs shown in Table 1, GLVQ consistently achieves lower perplexity than CBQ across both model sizes and bitwidths. These additional results will be incorporated into the revised version of the paper, together with a brief discussion. We thank the reviewer again for encouraging a more comprehensive comparison.\n\n\n----\n\n### **[Q3, Error Bars]**  \nWe appreciate the reviewer’s suggestion and fully agree that reporting confidence intervals would strengthen the empirical claims.\n\nDue to time constraints, we were unable to re-run all experiments across multiple seeds before the rebuttal deadline. However, we are committed to including 95% confidence intervals (based on five seeds) for all key perplexity and accuracy results in the final version of the paper and supplementary material.\n\nA note will also be added to clarify that all reported numbers are averages over multiple seeds with corresponding confidence intervals. Thank you for encouraging stronger statistical rigor.\n\n----\n\n\n### **[Q4, Activation Quantization]**  \nWe thank the reviewer for this insightful question. Extending GLVQ to quantize activations and KV-cache is a natural direction. While feasible in principle, several practical considerations must be addressed:\n\nFirst, activation and KV-cache statistics are input-dependent and non-stationary, unlike model weights. Designing a single static lattice per layer may be suboptimal unless proper normalization or dynamic rescaling is introduced. This challenge can be mitigated using techniques like SmoothQuant [@smoothquant], which apply lightweight affine transformations to stabilize activation distributions layer-wise. Once stabilized, a per-layer lattice codebook becomes viable.\n\nSecond, runtime efficiency is critical. Since activations are generated and consumed on-the-fly, any decoding operation must match the low-latency constraints of inference. Fortunately, lattice decoding with small dimensions (e.g., $d = 4$) is fast and can be further accelerated using SIMD instructions or table lookups. For KV-cache blocks that are reused across many time steps, the amortized cost of decoding is even lower, making lattice compression particularly attractive in that setting.\n\nLastly, memory reuse patterns and GPU bandwidth constraints need careful evaluation. Compression benefits will only be realized if the decoding overhead is small relative to the memory bandwidth saved. To address this, we envision using compact LUT-based inverse $\\mu$-law decoding to reduce runtime cost further.\n\nIn summary, while activation and KV-cache quantization introduces new challenges, we believe GLVQ can be extended to this setting through (i) per-layer rescaling, (ii) efficient low-dimensional decoding, and (iii) amortized or LUT-based companding. We are actively exploring these directions in future work.\n\n\n----\n\n\n### **[Q5, Hardware Efficiency]**  \nIn this submission, we focused primarily on evaluating *memory-bandwidth–limited throughput* on server-class GPUs such as the RTX 4090 and A100. As such, we have not yet benchmarked wall-clock latency on mobile SoCs or measured energy consumption in terms of Joules per token.\n\nThat said, we fully agree that these metrics are essential for assessing deployment feasibility on edge devices. As a next step, we plan to port the CUDA kernels to mobile hardware platforms, including ARM Mali and Apple M-series chips via `torch.metal_backend`. We also intend to profile inference on Qualcomm Snapdragon 8 Gen 3 (Adreno 750) using the QTI AI SDK, which will allow us to collect both latency (ms/token) and energy (J/token) metrics under realistic mobile conditions.\n\nThese measurements require additional engineering work and optimization, particularly around kernel portability and runtime instrumentation. We plan to include the results in an extended technical report and will reference this in the camera-ready version of the paper.\n\nWe thank the reviewer again for pointing out this important direction for future evaluation.\n\n----\n\n### **[Q6, Ethical Safeguards]**  \nWe fully share the reviewer’s concern that more compact LLMs, while enhancing accessibility and deployment efficiency, may also lower the barrier for misuse, such as scalable disinformation or malicious automation.\n\nIn the camera-ready version, we will include a brief **“Responsible Use and Limitations”** subsection. This will highlight that a model card will be released alongside GLVQ, explicitly stating intended applications (e.g., on-device summarization, QA), out-of-scope uses (e.g., generation of deceptive or harmful content), and known limitations such as hallucination and toxicity.\n\nWe will also note that GLVQ preserves the alignment properties of the original FP16 model, including RLHF or instruction tuning, and caution against compressing unaligned checkpoints. Since GLVQ operates only on the weight tensor, existing watermarking methods (e.g., TreeRing) remain applicable. We will recommend practitioners verify watermark presence before deployment and, where feasible, use an external safety filter or moderation API in downstream applications.\n\nWe believe this addition provides appropriate ethical context and encourages responsible use of compressed LLMs."}, {"review_id": "nHnPZMvjcb", "reviewer": "Reviewer_nGkm", "summary": "The paper introduces Grouped Lattice Vector Quantization (GLVQ), a weight-only quantization method for compressing large language models (LLMs) to extremely low bit-widths (2–3 bits) to enable faster and scalable inference. Unlike prior approaches that rely on fixed lattice structures like the E8 lattice, GLVQ learns group-specific lattice codebooks combined with adaptive companding transformations, resulting in significantly improved compression quality. The method also supports intra-weight mixed-precision quantization, where bit-widths are allocated based on the salience of each weight group, further enhancing model performance under the same resource constraints. Experimental results demonstrate that GLVQ consistently outperforms strong baselines such as QuIP#, AQLM, and QTIP—especially at 2-bit precision—achieving lower perplexity and higher accuracy across various LLM benchmarks.", "questions": "- Effectiveness and Efficiency of Companding /\nHow does the companding operation in GLVQ compare to other incoherence processing such as the Random Hadamard Transform (RHT) or FFT in terms of quantization performance and computational cost? Specifically, is companding more or less expensive computationally, and how does its impact on quantization fidelity compare to methods like RHT used in QuIP#? (the paper only ablates with and without compading)\n\n- Decoding Efficiency Comparison /\nI’m unclear how GLVQ achieves decoding speeds comparable to QuIP# and QTIP, as reported in Table 3. Both QuIP# and QTIP explicitly design their codebooks to fit within the L1 cache, enabling extremely fast dequantization. In particular, QuIP# leverages the symmetric properties of the E8 lattice for efficient encoding and fast lookup. Are you suggesting that GLVQ’s decoding process—which involves inverse companding followed by matrix multiplication—can match or exceed the speed of these highly optimized lookup-based methods?\n\n- Throughput Discrepancy with QuIP# /\nThe throughput numbers reported in Table 3 appear inconsistent with those in the QuIP# paper. For example, QuIP# reports a token rate of 170 tokens/sec on an NVIDIA RTX 4090, whereas Table 3 shows only 106.3 tokens/sec for QuIP# under similar conditions. Could you clarify the source of this discrepancy?\n\n(These are the main concerns and the questions to decide my ratings)", "limitations": "-", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Strong empirical results: The method consistently outperforms state-of-the-art methods (e.g., QuIP#, AQLM, QTIP) across multiple LLMs and tasks, especially in the challenging 2-bit regime\n- Comprehensive Evaluation: The paper includes extensive experiments, ablation studies, and theoretical analysis (e.g., Babai rounding error bounds), demonstrating both robustness and rigor.\n\n\nWeakness\n- Mixed-Precision Comparison Fairness: While mixed-precision quantization can be applied to many methods, this paper focuses on intra-weight group-level precision, which may introduce additional latency. In contrast, even simpler layer-wise mixed-precision schemes can yield significant performance gains with less overhead. Therefore, a fairer comparison would be between GLVQ with uniform precision (GLVQ-u) and other baseline methods under the same bit-width constraints \n\n- Limited Evaluation Scope: The evaluation primarily focuses on perplexity and a few common-sense reasoning tasks (e.g., ARC, PIQA, Winograd), which are relatively simple. However, recent studies have shown that many quantization methods struggle on more challenging benchmarks like MMLU. Including such tasks would provide a more comprehensive assessment of GLVQ’s robustness and generalization (even without comparison to other works) .", "was_revised": true, "final_justification": "Given the reasonable explanation and justification provided in the rebuttal, which addressed my concerns, I have decided to raise my score", "rebuttal": "----\n### **[W1, Mixed-Precision Comparison Fairness]**  \nWe agree that a fair evaluation of GLVQ’s core contribution, group-specific lattice quantization, should be based on uniform-precision settings, independent of our salience-driven bit allocation (SDBA) strategy.  \nTo this end, uniform-precision variants of GLVQ (denoted GLVQ-u or GLVQ w/o bit allocation) were already included in our original submission and reported in Appendix A.4. For clarity and ease of comparison, we reproduce the key results below.\n\nTable 1. Results of GLVQ-u.  \n| Method            | 1-7B | 1-13B | 1-30B | 1-65B | 2-7B | 2-13B | 2-70B |\n|-------------------|------|-------|-------|-------|------|-------|--------|\n| GLVQ-8D (2-bit)   | 6.28 | 5.64  | 4.57  | 4.01  | 5.69 | 5.02  | 3.62   |\n| GLVQ-8D-u (2-bit) | 6.44 | 5.79  | 4.65  | 4.10  | 5.87 | 5.19  | 3.72   |\n| GLVQ-8D (3-bit)   | 5.72 | 5.10  | 4.11  | 3.56  | 5.13 | 4.48  | 3.14   |\n| GLVQ-8D-u (3-bit) | 5.81 | 5.18  | 4.17  | 3.61  | 5.21 | 4.55  | 3.21   |\n| GLVQ-8D (4-bit)   | 5.70 | 5.08  | 4.09  | 3.52  | 5.01 | 4.44  | 3.02   |\n| GLVQ-8D-u (4-bit) | 5.74 | 5.13  | 4.14  | 3.56  | 5.08 | 4.49  | 3.08   |\n| \n\nAs the table shows, even without salience-driven bit allocation, GLVQ-8D-u improves perplexity over the current SOTA PTQ method QTIP. This confirms that the performance gains stem from the underlying group-specific lattice design, rather than auxiliary heuristics such as mixed-precision allocation.\n\nWe will clarify this discussion in the revised paper and add a footnote directing readers to Appendix A.4 for the full set of uniform-precision ablation results.\n\n\n----\n\n### **[W2, Limited Evaluation Scope]**  \nWe agree that more challenging tasks such as MMLU provide a stronger test of generalization under quantization. To this end, we evaluated **GLVQ** on the MMLU benchmark for both Qwen-7B-chat and Qwen-14B-chat models, using standard 5-shot accuracy.\n\nTable 2. MMLU accuracy (%) with different methods. Higher is better.\n| Method | Bits | Qwen-7B-chat | Qwen-14B-chat |\n|--------|------|---------------|----------------|\n| BF16   | 16   | 56%           | 65%            |\n| GPTQ   | 3    | 51%           | 60%            |\n| GLVQ | 3    | 54%       | 61%         |\n| GPTQ   | 2    | 23%           | 22%            |\n| GLVQ | 2    | 45%      | 51%        |\n| \n\nAs shown in the table, GLVQ yields significantly higher MMLU accuracy than GPTQ at both 2-bit and 3-bit settings. In particular, at 2-bit precision, GLVQ improves over GPTQ by +22 points on Qwen-7B-chat and +29 points on Qwen-14B-chat. Even at 3-bit, GLVQ achieves consistent gains over GPTQ while remaining close to the BF16 reference.\n\nThese findings demonstrate that GLVQ maintains abstract reasoning and factual recall better than prior methods even on structurally different models like Qwen, highlighting its robustness across architectures.  \nWe will incorporate this table and discussion into the revised manuscript to provide a broader and more rigorous evaluation.\n\n----\n\n\n### **[Q1, Effectiveness and Efficiency of Companding]**  \n**1. Computational cost**\n\n| Operation                  | Arithmetic cost [^1]      | GPU latency (7B)     |\n|----------------------------|---------------------------|-----------------------|\n| Element-wise $\\mu$-law inverse | $\\mathcal{O}(n)$          | <2% of mat-vec        |\n| $H_n$ × vector (RHT)       | $\\mathcal{O}(n \\log n)$   | ≈9%                   |\n| $F_n$ × vector (FFT)       | $\\mathcal{O}(n \\log n)$   | ≈11%                  |\n| \n\n[^1]: Here $n=128$ is the group size in GLVQ. The GEMV operation with a weight matrix of shape $128 \\times 4096$ takes approximately $135 \\mu\\text{s}$ on an A100 (80GB), based on empirical measurements.\n\n\nThe $\\mu$-law requires only a fused exp or LUT lookup per weight, while RHT/FFT introduce an extra $128 \\times 128$ orthogonal multiply per group. This results in a 2% vs. 9–11% throughput drop, respectively.\n\n\n**2. Quantization fidelity (2-bit uniform, LLaMA-2-7B)**\n\nTable 3. Ablation of incoherence processing strategies.  \n\n| Variant                   | WikiT2 PPL ↓ | TOK/s ↑ | ΔPPL     |\n|---------------------------|--------------|---------|----------|\n| None (baseline)           | 6.28         | 105.3   | --       |\n| + $\\mu$-law only (GLVQ-u) | 5.87         | 102.3   | –0.41    |\n| + RHT only                | 5.88         | 96.0    | –0.40    |\n| + $\\mu$-law + RHT         | 5.63         | 91.2    | –0.65    |\n|\n\n\n$\\mu$-law yields a larger perplexity drop than RHT (–0.59 vs. –0.40) by normalizing heavy-tailed weights before lattice quantization. Combining both gives a modest gain (–0.65), suggesting partially overlapping effects. In terms of efficiency, $\\mu$-law costs only 2% throughput loss, whereas RHT introduces a heavier 9% drop due to its $\\mathcal{O}(n \\log n)$ transform, making $\\mu$-law preferable for GPU and edge deployment.  \n\nThe group-wise $\\mu$-law companding offers a better fidelity–latency trade-off than RHT or FFT: lower perplexity at minimal compute overhead. While RHT can be layered for slight improvements, it is not essential to GLVQ’s performance. We will include this table and the above analysis in the revised manuscript.\n\n----\n\n\n### **[Q2, Decoding Efficiency Comparison]**  \n**Key implementation choices that minimize decoding overhead**\n\nGLVQ achieves competitive decoding speeds despite its arithmetic-based design by leveraging several carefully engineered optimizations:\n\n1. **On-the-fly fusion inside the GEMM kernel.**  \n   Both the inverse $\\mu$-law transformation and the matrix multiplication $G_g \\mathbf{z}$ are fused into the fragment loading stage of the Tensor Core WMMA pipeline. These operations are performed on 16-value `int2` register tiles, introducing no additional global memory traffic and incurring negligible latency due to full overlap with the GEMM execution.\n\n2. **Register-resident constant matrices.**  \n   Each $G_g \\in \\mathbb{R}^{8 \\times 8}$ consists of only 64 FP16 values (0.125 kB), which is smaller than a single Tensor Core fragment (0.256 kB). As a result, CUDA retains $G_g$ in registers across the warp, completely avoiding L1/L2 cache accesses.\n\nTo quantify the impact of these optimizations, we conducted the following micro-benchmark on an NVIDIA A100 GPU using CUDA 12.1:\n\nTable 4. Micro-benchmark: 128×4096 GEMM at 2-bit precision.\n\n| Method                       | Decode latency (µs) | GEMM latency (µs) |\n|-----------------------------|---------------------|-------------------|\n| QuIP# (E8 lookup)           | 0.83                | 134.7             |\n| QTIP (trellis LUT)          | 0.77                | 134.5             |\n| GLVQ (inv-μ + $G\\mathbf{z}$) | 0.92                | 135.1             |\n\nThe additional 0.09 µs decoding overhead introduced by GLVQ is negligible compared to the total GEMM time of ~135 µs. In end-to-end inference, this results in less than a 3 tokens/s throughput difference (see Table 3), while achieving substantially improved quantization fidelity.\n\n**Why arithmetic decoding can outperform LUT-based methods on modern GPUs**\n\n- **Compute-bound regime.**  \n  On Ampere GPUs, each byte of L2 bandwidth supports approximately 2 FP16 FMAs. GLVQ leverages ALUs and register space for decoding, whereas LUT-based methods rely on memory access, which becomes a bottleneck in compute-bound settings.\n\n- **Warp-level parameter reuse.**  \n  Since each $G_g$ and $\\mu_g$ is shared across all threads within a warp, GLVQ avoids thread divergence and enables efficient, fully parallelized decoding.\n\nIn summary, thanks to aggressive kernel fusion and register-level optimization, GLVQ achieves decoding latency within 0.09 µs of QuIP#, translating to under 3% throughput difference, while delivering superior quantization fidelity. We will incorporate these engineering insights and benchmarking details into the revised manuscript to clarify decoding efficiency.\n\n\n\n----\n\n### **[Q3, Throughput Discrepancy with QuIP#]**  \n**Clarification**  \nThe throughput discrepancy arises from two distinct configurations reported in the original QuIP# paper. Specifically:\n\n- **Table 5** reports 170.5 TOK/s using an optimized stack with FlashAttention 2 and Triton-based custom kernels.\n- **Table 6**, by contrast, reports 106.7 TOK/s using the standard Hugging Face library; this is the same stack used in GLVQ.\n\nTo ensure fairness and consistency, our Table 3 adopts the Hugging Face inference configuration across all methods. We therefore report the 106.3 TOK/s figure, matching Table 6 of the QuIP# paper.\n\nTo verify that this choice does not alter the relative comparison, we also benchmarked both QuIP# and GLVQ under FlashAttention 2. As shown in Table 1, the relative gap between QuIP# and GLVQ remains stable (approximately 6–7%) under both inference engines.\n\nTable 5. Throughput of QuIP# under different inference engines (LLaMA-2-7B, 2-bit uniform).\n| Engine                          | TOK/s  | Relative to GLVQ-u |\n|---------------------------------|--------|---------------------|\n| Hugging Face (QuIP# Table 6) | 106.3  | +6.1%               |\n| FlashAttention (QuIP# Table 5)    | 170.5  | +69.2%              |\n|"}]

Author rebuttal:

Meta-review: This paper proposes a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights in Large Language Models (LLMs) a customized lattice codebook defined by a learnable generation matrix, aiming to boost quantization accuracy and efficiency for low-bit LLM compression.  After discussion, most of the concerns have been well resolved.

The work exhibits notable strengths: it introduces a novel and well-motivated group-specific, learnable lattice quantization and achieves state-of-the-art perplexity in the challenging low-bit regime across multiple LLMs. it also includes comprehensive evaluations with extensive experiments, ablations, and theoretical analysis, supporting its effectiveness in ultra-low-bit and fractional-bit settings.

Overall, the AC recommend to accept this paper.
Decision: Accept (poster)