NestQuant: nested lattice quantization for matrix products and LLMs
Authors: Semyon Savkin
Eitan Porat
Or Ordentlich
Yury Polyanskiy
Venue: ICML 2025 poster (2025)
Keywords: large language models
quantization
Primary area: deep_learning->large_language_models
TL;DR: Nested lattice codes for LLM quantization

Abstract: Post-training quantization (PTQ) has emerged as a critical technique for efficient deployment of large language models (LLMs). This work proposes NestQuant, a novel PTQ scheme for  weights and activations that is based on self-similar nested lattices. Recent works have mathematically shown such quantizers to be information-theoretically optimal for low-precision matrix multiplication. We implement a practical low-complexity version of NestQuant based on Gosset lattice, making it a drop-in quantizer for any matrix multiplication step (e.g., in self-attention, MLP etc).  For example, NestQuant quantizes weights, KV-cache, and activations of Llama-3-8B to 4 bits, achieving perplexity of 6.6 on wikitext2. This represents more than 55\% reduction in perplexity gap with respect to unquantized model (perplexity of 6.14) compared to state-of-the-art Meta's SpinQuant (perplexity 7.3), OstQuant (7.3) and QuaRot (8.2). Comparisons on bigger models (up to 70B) and on various LLM evaluation benchmarks confirm uniform superiority of NestQuant.

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Semyon Savkin
Fitan Porat 2 Or Ordentlich 3 Yury Polyanskiy'
Abstract
Post-training quantization (PTQ) has emerged as critical technique for efficient deployment of large language models (LLMs). This work proposes NESTQUANT, a novel PTQ scheme for weights and activations that is based on self-similar nested lattices. Recent works have mathemati- cally shown such quantizers to be information- theoretically optimal for low-precision matrix multiplication. We implement a practical low- complexity version of NestQuant based on Gos- set lattice making it a drop-in quantizer for any matrix multiplication step (e.g., in self-attention, MLP etc). For example, NestQuant quantizes weights, KV-cache, and activations of Llama-3- 8B to 4 bits, achieving perplexity of 6.6 on wiki- text2. This represents more than 55% reduction in perplexity gap with respect to unquantized model (perplexity of 6. 14) compared to state-of-the-art Meta's SpinQuant (perplexity 7.3), OstQuant (7.3) and QuaRot (8.2). Comparisons on bigger models (up to 70B) and various LLM evaluation bench- marks confirm uniform superiority of NestQuant.
VeetQuan.. W K
Sprouunt wnight only
VestQuant. V KV
OSTOurtt weight only
.8) MestQuem: we git only
OSTOurtt w Ry +
0.5-
Rall precedon (FF16)
Quarst Rt +
0
SpinQuart W
O-Qunrt W A
7.3-
7.3
3.2
3.6
3.8
6.0
4.2
6is
Figure 1. Perplexity of quantized Llama-3-8B models for three regimes (weight-only, weights + KV cache, end-to-end) on wiki- text2 number of bits per entry.
ing pair of n x n matrices requires 2n FL.PPs and Rnn bytes to exchange between the core and memory (here and below R designates the number of bits required to store each entry of vector/matrix). So when matrices are large (such as during the pre-fill phase when the prompt is processed) the GPU is compute-bound, while when n is small (such as during generation) the GPU becomes memory- bound. One therefore needs to reduce R by quantizing both weights and the KV cache, thereby alleviating the memory bandwidth bottleneck during generation.
1. Introduction
There are three principal goals of post-training quantization (PTQ). First, reducing the number of bits per parameter allows for loading big models on cheap GPUs with limited memory, thus democratizing access to LLMs. This requires "weights-only" quantization algorithms of which the most popular are AWQ, GPTQ, and QulP (see references in Sec- tion 2.2).
The third goal of PTQ is to accelerate inference of giant LLMs that require hosting each layer on separate GPU (pipelining parallelism). For this goal one needs to quantize activations passed from one layer to the next to reduce the communication bottleneck.
The second goal of PTQ is to accelerate inference. In LLMs most of the compute spent multiplying matrices. Multiply-
Equual contribution 'Deparmment of Electrical Engincering and Computer Science, Massachusetts Institute of Technology, Cam- bridge, MA 'Independent 'Rachel and Selim Benin School of Com- puter Science and Engineering. Hebrew University of Jerusalem, Israel. Correspondence to: Semyon Savkin <semyon@ mit.edu>, Eitan Porat <ethan, porbat @ gmail.com>.
While quantization of weights to R 3, 4 and even R 2 bits has been achieved with minimal loss of quality, quan- tization of KV cache and activations has been much more challenging. Popular algorithms for full quantization are I.L.Mint80.. SmoothQuant and SpinQuant (see references in Section 2.2), the latter having state-of-the-art performance. This work proposes an alternative algorithm (NestQuant) for quantizing weights, KV-cache, and activations. The
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Model
Bits
Bits (no zstd)
ARC-C 1
ARC-E 1
Hellaawaa
PIQA
Winogrande
Zerosboot Avg
Wikitext2 ppl
Baseline (FP16)
16
0.54
0.78
0.79
0.81
0.74
0.73
6.1
Weights only
LLM-QAT
4,00
0.51
0.77
0.48
0.79
0.72
0.65
7.7
GPTQ
4,00
0.47
0.72
0.74
0.77
0.71
0.68
7.2
SpinQuant
4,00
0.54
0.77
0.78
0.80
0.72
0.72
6.5
NesiQuant q - 14, E: (ours)
3.99
4.06
0.53
0,78
0.79
0.80
0.73
0.72
6.3
Weights KV cuche
SpinQuant
4,00
0.51
0.77
0.77
0.78
0.69
0.70
6.6
NesiQuant q 14 (ours)
3.99
4,06
0.53
0.78
0.79
0.79
0.74
0.72
6.4
Weights, KV cache, activations
L.M-QQAT
4,00
0.27
0.41
0.38
0.60
0.53
0:44
52.5
Quarot
0.67
0.75
0.75
0.67
8.4
SpinQuant
0.51
0.75
0.75
0.77
0.66
0.68
7.3
NesiQuant y = 14, h (ours)
3.99
4.06
0.51
0,75
0.78
0.79
0.72
0.71
6.6
Table 4-bit quantization of Llama-3-8B. bits column NestQuant corresponds to actually measured average number of bits per entry (when vector of auxiliary scaling coefficients ( is compressed via zstd) and the second column shows quantization rate when no compression step is used.
algorithm motivated by recent theoretical work on approx- imate matrix multiplication and follows several classical ideas such as Conway and Sloane's Voronoi codes and their algorithms for finding the closest lattice vector in some canonical lattices in low/moderate dimensions.
The main source of improvement of NestQuant is demon- strated in Fig. 2 (although NestQuant uses an 8-dimensional Gosset lattice, not 2D hexagonal one). More details on this as well directions for improvement are discussed in Section 3.
Throughout this paper, NestQuant is described with the 8- dimensional Gosset lattice, but the framework is general and can be similarly implemented using any other base lattice, in any dimension, provided that a low-complexity closest lattice vector algorithm exists for this lattice.
Thus, we believe that NestQuant offers excellent alterna- tive to other algorithms. It quantizes weights, KV-cache and activations, and achieves significant improvement on both synthetic and real data.
1.2. Paper organization
1.1. Summary of results
We start with a detailed review of classical work on vector quantization and modern LLM quantization (Section 2). Then Section 3 we explain the motivation for each step of the algorithm. Section 4 contains the pseudocode of the algorithm and diagram of quantized LLM. Finally, Section 5 concludes with details about empirical performance. Further details and evaluations are relegated to the Appendices.
The NestQuant algorithm is described in Section 4. NestQuant a generic drop-in replacement for any matrix multiplication. Its performance for synthetic random Gaus- sian matrices comes pretty close to iiformation-theoreed limits (see Fig. 3) and significantly outperforms uniform quantization employed by SpinQuant. Switching from scalar (uniform) quantization to vector quantization requires some price to pay computationally (Section E.1), however, among vector quantizers NestQuant is rather economical as builds upon the Voronoi Codes framework of (Conway & Sloane, 1983). We mention that in the presence of activation quantization, it is important to quantize weights properly an innovation we call QA-LDLQ (Section 4.5).
2. Prior work
We briefly survey prior work, which we separate into work by information theorists and by the MI. community.
2.1. Information-theoreti quantization
Applying NestQuant to quantizing an actual L.MSs (Llama 2 and with parameter count ranging from IB to 70B) shows massive end-to-end improvement: Fig. shows significant reduction of perplexity compared to the previous methods; standard L.LM benchmarks; and Table 2 show NestQuant stays consistently on top for different model sizes. In fact, NestQuant in full quantization (weights, activations, and KV cache) outper- forms SOTA results with quantized weights and activations (but not KV cache).
Rate R quantization of an information source in IR" is the operation of encoding it to n/t bits, from which a de- coder can produce reconstruction X R that has small distortion with respect to X. The most popular distortion criterion is the quadratic loss, where the expected distortion is defined as D = π tention to this loss. Characterization of the optimal tradcoff between R and D a classic topic in information theory, e.g. (Polyanskiy & Wu, 2024, Part V).
Table confirm enhanced performance
_ X - X||2, and here we restrict at-
For Gaussian source X Ao0,, 4x) the rate-distortion
2

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
theorem states that any compressor at rate R must satisfy D 2 D(R) - 2-2R Furthermore, dimension n. in- creases there exist quantizers with distortion approaching D(R). Notably, such quantizers can be made universal, in the sense that they attain distortion ((R) not only for iid Gaussian X but for any (even adversarial) input as long as its Euclidean norm is (1 + vn.)
Remark 2.1. Using optimal quantizers, in terms of achiev- ing ( for quantizing X and Y, and then computing the inner product of the quantized vectors (maybe with some MMSE scaling) will not necessarily lead to the optimal per- formance (()). The reason is that the covariance matrix of each vector's quantization error affects the inner product distortion through its trace, and its Frobenius norm (Or- dentlich & Polyanskiy, 2024). For optimal lattice quantizers in any dimension, the Frobenius norm is guaranteed to be minimal (Zamir & Feder, 1996), and consequently, using lattice quantizers seems essential for achieving I(R).
One way for constructing these universal quantizers is based on lattices (Zamir, 2014) that admit much more structure than more classical random codes (and e-nets).
Arguably, the most notable lattice-based quantization scheme is the family of Voronoi codes (Conway& & Sloane, 1983), which we use in this work.
In this work we show that with appropriate tweaks Voronoi codes can indeed result in practical fast and efficient algo- rithms for LLM quantization. We emphasize that most of our work is on simply developing a drop-in replacement for quantized matrix product and as such is not specific to LLMs.
How does one convert a quantizer adapted Gaussian inputs to work (with the same guaranteed loss) non- Gaussian data? In a nutshell, the idea is simple: if U is chosen to be a random n X orthogonal matrix then the entries of U'X will be distributed as iid Gaussian (Stam, 1982). This idea of applying random rotations to smooth out the distribution of the quantizer's input may be viewed as special case of high-dimensional companding (Gersho, 1979), and has been been used in various applications, such as image compression (Hung & Meng, 1998), and as po- tential replacement for dithering (Hadad & Erez, 2016), to name a few.
The idea of applying random rotations to "Gaussianize" in- puts in the context of approximate inner product computa- tion is even more natural than in the standard quantization. Indeed, since one is only interested in the inner product, not vectors themselves, one does not need to store (even a seed used to generate the) random orthogonal matrix. This has been long exploited in locality-sensitive hashing (LSH) a algorithms, which can be viewed as an (extremely low rate) quantizers for inner product approximation (Charikar, 2002; Datar et al., 2004; Andoni & Indyk, 2008). Unsurprisingly, as we will see next, random rotations have also been found quite useful for quantizing LLMs.
In the context of LLMs, the goal in quantization is slightly different since quantization is used facilitate approximate matrix multiplication with reduced burden on the mem- ory bandwidth. For example, when quantizing two vectors X,Y E R" the goal is not to approximate them but to approximate their inner product. Recently, information- theoretic characterization of this task was completed in (Or- dentlich & Polyanskiy, 2024). Specifically, the authors show that if X,Y Y ~ NO, In) are independent then for any algorithm operating on the basis of rate-R R quantized representations of X and Y we must have
2.2. LLM quantization
One of the directions of prior research on LLM quantiza- tion is addressing the issue of activation outliers that hinder the quantization quality. These outliers are present in cer- tain dimensions of activations, weights and KV cache. In LLM.int8() of (Dettmers et al., 2022), these outlier dimen- sion are kept unquantized. In SmoothQuant (Xiao et al., 2023) authors balance the scale of outliers between weights and activations by modifying LayerNorm's 's diagonal matri- ces.
E - - 2 (R))
(1)
where TTY Y is the reconstructed inner product, and
I(R)= =
1 - (1 (1-(2.2-2 22.2-2 - 2-1R") ))
R
R
R <<< R
Going to random rotations, by rewriting matrix product AB = (AU)UU B) for an orthogonal matrix U, one gets matrices with much more Gaussian entries (few outliers) and can apply standard quantization algorithms. Some of the multiplications by U can be merged with the weights (i.e. do not require additional runtime FLOPs), while the rest are applied at runtime. For the latter, matrices U should have
2 2 2-22 2R -2-4R
R R
(2)
where R² Rs 0.906 a solution to a certain transcendental fixed-point equation.
The same paper also constructs universal quantizers based on nested lattices that asymptotically (as n oc) achieve this lower bound. Note that extension from vectors to matri- ces can be made trivially by observing that one can quantize each column separately and treat matrix product as a collec- tion of inner products.
'In LSH, one typically performs several random projections of the vector and quantizes them. This equivalent to performing random rotation and quantizing only a small number of entries of the rotated vector.
3

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
3. Outline of NestQuant approach
In this section we outline the main components of our ap- proach. A detailed description is brought in the following section.
When designing a quantizer, one needs to make some as- sumptions about the distribution of the source that will be fed to it. While weights (and sometimes activations) can be well-approximated by Gaussians, their magnitudes are wildly varied. Thus, one employs two ideas: normalization and random rotation.
Figure 2. Demonstrating advantage of NestQuant in 2D. Typical weights and activations are vectors inside the black circle. Uniform quantization wastes about 32% of allocated bitstrings for vectors outside of the circle, while nested hexagonal lattices only wastes 15% (explicitly enumerating points inside the circle to avoid the waste is too slow to do runtime). This allows NestQuant to use finer grid while quantizing to the same rate R. The gain becomes much more dramatic in higher dimensions.
In most of the literature, the normalization is done by taking input vector of large dimension (e.g. = 4096 for Llama-3), dividing by the Lxe norm to get entries to be in [-1,1] and then applying uniform quantization. This is suboptimal for two reasons: one is that uniform quantization induces error that distributed uniformly on the small cube, which is suboptimal from the MSE point of view. Second reason, much more serious, is known as the shaping gain and demonstrated on Fig. 2. When entries of the vector are Gaussian, it will typically lie inside the black circle. Thus those grid elements outside of it will almost never be used, wasting bitspace.
Normalization:
structure enable fast multiplication. For example, QuaRot (Ashkboos et al., 2024) uses randomized Hadamard matri- ces as coordinate transformations, which can be applied to a vector of size n in O(Ilog n) additions. SpinQuant (Liu et al., 2024) uses a rotation parametrization with four orthogonal matrices R1, R2, Ra, R4, where RS and R2 can be arbitrary orthonormal matrices, and R3 and R4 should have a fast multiplication algorithm. The authors use Cayley SGD (Li et al., 2020) to optimize Rj and R2 for minimiza- tion of the quantization error, while the matrices R3 and R4 are chosen to be random Hadamard.
Instead, we use normalization by the L--norm (see Algo- rithm 3) and then use points inside the Voronoi region of a Gosset lattice, which Fig. 2 (right) demonstrates wastes lot fewer bitstrings for rare vectors, thus allowing us to use finer grids.
Starting from LLM.int8() most of the schemes used uni- form quantization (i.e. where a floating point vector simply rounded to a nearest integer after an appropriate rescaling). To the best of our knowledge, so far non-scalar quantization has only been used for weight-only compression for LLMs in QulP# (T'seg et al., 2024a), which uses ESP codebook for 2-bit quantization, and applies Resdidual Vector Quan- tization (Juang & Gray, 1982) to get a 4-bit quantization scheme; and QTIP (Tseng et al., 2024b) which uses trelis codebook. Unfortunately these methods appear to be too expensive to apply in runtime, perhaps explaining why non- uniform quantization for activations and KV-cache was not attempted before this work.
When input to the quantizer significantly differs from the presumed model (of iid Gaussians), perfor- mance can become quite poor. As discussed previously, multiplying by a random orthoghonal matrix U provides an excellent fix. Specifically, U. vector becomes uniform on the n--phere of radius vTi, and small chunks of this vector have distribution very close to Gaussian iid. In particular, the total variation between any subset of d coordinates and (0, Ia) is (dd /n) (Stam, 1982), such that for d = of vn) what we quantize is effectively iid Gaussian.
Random rotation:
Lattices: A lattice A Rd is discrete subgroup of Rd Any lattice A C Rd has (non-umique) generating matrix G E Rdxd, such that A = GZd (that is, A is the integer span of the columns of G). We define the nearest neighbor quantizer QA : .Rd + A as
Finally, when quantizing weight matrices, one may notice that MSE distortion loss should be replaced by a weighted- MSE loss dependent on the statistics of the incoming acti- vations. We refer to this type of algorithms as LDLQ, fol- lowing authors of QuIP (Chee al., 2023), QulP# (Tseng et al., 2024a) and GPTQ (Frantar et al., 2022). In the pres- ence of activation quantization, however, the LDLQ needs to be modified (see Section 4.5 introducing QA-LDLQ). Equipping NestQuant with QA-LDIIQ incurs significant improvement; see results in Section 5.
Q argmin || AI|,
AEA
where ties are broken arbitrarily, but in a systematic manner. The Voronoi region VA is defined as the set of all points in Rd that are closer to than to any other lattice point
VA- (x E Rd
Qx(x)= = 0}
(4)
The covolume of a lattice is defined as covol(A) = det G = vVl().). We say that a lattice Ag C R nested
4

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
in the lattice A, C R4, if Ac C Ay. Note that for any inte- ger q 2 2 we have that qA C A, and that A A/gA = (Z/qZ)4. For an introduction to lattices and their role in quantization, see (Zamir, 2014).
would like to use the smallest possible 3 for which overload is rare. In order allow for smaller f3, we would like to choose S C IR" such that BS captures as much Gaussian mass as possible.
Complexity of lattice quantization: In order to explain our choice of nested lattice quantizer, we need to carefully balance several requirements. One of the key ones is com- plexity. It is known that finding (even approximating) nearest lattice point is a well-known cryptographic assump- tion (Dinur et al., 2003). Thus, we are not suggesting to directly operate on n--imennional lattices. Instead, we par- tition the vevector into sections, each of dimension d and apply lattice quantization to d-subvectors. Equivalently, our vector quantizers for IR are constructed as Cartesian prod- ucts of vector quantizers of small dimension d (we will take d = 8 for all experiments).
Denote by = (N Ta) the standard Gaussian measure. Since we need 2d = |A S ~ covol[A]: good shaping region S maximizes yuS)), which in turn minimizes the over- load probability that is approximated by 1 1- (()),,, under a volume constraint. Clearly, the optimal S under this crite- rion is rB where B = {i E IR | I (en(1)) } is Eu- clidean ball with radius (1) chosen such that vol(B) = 1, and r is chosen such that vol(r = rd satisfies the required volume constraint. Unfortunately, for >> the codebook C = A n rB loses much of the lattice structure, and does not admit an efficient enumeration, and consequently en- coding and decoding require using lookup table (LUT). QulP# used this approach with A Es (same as we do) and S = rB. However, this seems to only be possible for quamtizing weights and not activations as complexity makes runtime implementation too slow.?
vol(S)
Granular and overload quantization errors: There are two different sources of errors for lattice quantizers. The first is called granular quantization error, and is related to the second moment of the lattice Voronoi region. A common way to measure the granular error corresponding to lattice A C Id is via the normalized second moment (NSM) defined as
Using int8-multipliers: One often mentioned advantage of uniform quantization compared other approaches is the fact that it approximates any matrix as product of diagonal matrix (of norms) and integer matrix. Thus, during multiplication one can leverage faster int-cores rather than floating-point multiplication. Note that if there exists a scaling coefficient a > 0 such that aA C Zd, then one can still use int-multipliers even for lattice-quantized vectors.
1
G(A) vol((A) d DEVA
||=]]/de.
(5)
This quantity corresponds to the MSE when A is normalized to have unit covolume and is then used as a quantizer. It is well known that the optimal (smallest) NSM among all lattices in IRd approaches 2ee mir, 2014). Furthermore, for d = we get GZZ) = 12' Consequently, in terms of granular error, by using high- dimensional lattices instead of the simple scalar quantizer based on E we can already gain a factor of 2me ~ 1.42329 12 in performance (the Gosset lattice achieves 1.16 gain).
Voronoi codes/nested lattice codes: In Voronoi codes (Con- way & Sloane, 1983) the same lattice A is used for both quantization and shaping. In particular, the shaping region is taken as S = gVA.. where 9 = 2 is an integer. As elabo- rated below, if (z) admits an efficient implementation, one can efficiently perform encoding and decoding to the codebook A A (2R),) = A/2 A. Moreover, in stark contrast to ball-based shaping, the encoding and decoding complexity does not depend R.
1 from above as increases (Za-
Notice, however, that representing x as (r),, the nearest lattice point in A, requires infinitely many bits, since the lattice is infinite. Since we only have 2dR bitstrings to allocate, we need to select a subset of A that will be actually used. Selection of a body S C Rd so that |A nS] = 2dR is called shaping. If Q (xr) E S, then the quantization error (z) will be in VA and we will only suffer from a granular error. However, when (x( & S the quantization error is no longer in VA and may be have much greater magnitude than a typical granular error. We refer to those type of errors, where (x) & S. as overload errors.
In summary, good choice of lattice A should therefore have: 1) efficient lattice decoding algorithm; 2) small NSM; 3) large (v()x): 4) be a subset of standard integer lattice Zd
In this work, we use the Gosset lattice (Es) that satisfies all these properties. It has fast decoding algorithm (Al- gorithm 5), its NSM is R 0.0716821 R 1 1.2243 tuee & Allen, 2023), and its Gaussian mass ((FVE,) is very close to ((rBB) (the Gosset lattice has unit covolume, so vol(rVe,) = vol(rB)). The last point is illustrated in
_ (Agrell
Generally speaking, in order to achieve small quantization error, one must keep the probability of overload very small. This can be attained by scaling up the codebook to BC = 3A 0 BS with a large enough 3 > O such that overload becomes very rare. However, increasing B also increases the squared granular error by a factor of B². Thus, one
"e note that QulP# cleverly exploits symmetries of Es show that R 2 bit quantizer can be implemented using an LUT of size 2 2", but we believe this is still too slow, and furthermore does not naturally extend to different quantization rates.
5

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Figure 5, where the large loss for cubic shaping with re- spect lattice shaping is also evident. The gap between Voronoi/Ball shaping and cubic shaping becomes much more significant as the dimension increases. This follows since for large d we have that for X ~ 14 = [4t the 20 norm | concentrates around v dn.. Thus, for r 2v 2 In d we have that j(rCUBE) - O, whereas for any T (2 o(1) we have that μ(rB) + 1. Note that SpinQuant uses high-dimensional cubic shaping, and therefore its MSE distortion suffers OIIn d) multi- plicative gap with respect to the optimal distortion.
Quantzztion I Loss Companison (dimensioo n = 4096)
NedDuuent
-0-
Unilor (ipen)aan))
Lozae fraund ((
RMSE par emay
10-
40-
30-
vd
5%
10
Rote (sits)
Overload avoidance via union of Voronoi codes: Be- cause we rely on lattice quantizers of relatively small di- mension (d- = 8), even if (((Vxx) is very close to (r(B), overload events are unavoidable. This follows because in small dimension the norm of a iid Gaussian vector is not sufficiently concentrated. Thus, if one is restricted to C= 3(A n (2)) the parameter 39 must be taken quite large in order to keep the overload probability small. This in turn, incurs a significant penalty in the obtained distortion. As a remedy, rather than using a Voronoi code, we take as union of (a small number) of Voronoi codes in different scales. Namely, we take C B (2 Vx)), where 31 <... < By. The smallest values of B, are set such that overload is not too common but not extremely rare, such that for most realizations of a Gaussian vector X E IRd the dis- tortion is close to the fundamental limit D (R). Whenever X atypically large, there will be overload 3. (2F))) for small f, but not for large t, such that the quantization error will be in 3, for one of the larger values of {Br).
Figure 3. RMSE for quantized matrix multiplication for iid N (0. 1) matrices. NestQuant algo is optimized over & and multiple S's. Also shown information-hhoorttic theoretic lower bound from (1).
tal limit and significantly outperforms SpinQuant.
4. Detailed Method
4.1. Nested lattice codebook
In this section, we describe the construction for a Vector Quantization (VQ) codebook of size q" for quantizing a d-dimensional vector, where is an integer parameter. This construction is based on Voronoi codes (Conway & Sloane, 1983) and admits efficient encoding and decoding algo- rithms, whenever the base lattice has an efficient closest lat- tice vector algorithm. Another appealing feature of Voronoi codes is that the encoding/decoding complexity is indepen- dent of the quantization rate R = log2(9).
The details of choosing lc and values of 3g are described in Section F. Here we only note that the overall compression rate becomes R + glogg k. In some cases, we are using nvcomp (NVIDIA Corporation, 2025) for compressing a vector of n/8 chosen betas, in which case rate penalty is reduced below 1 logg k. We note that in our comparisons, including Table 1, we always use this effective rate for fair comparison with other algorithms.
Definition 4.1 (Voronoi code (Conway & Sloane, 1983)).
The Voronoi codebook with base lattice A C Rd and nesting ratio g E N, corresponding to rate R = loga (q) bits/entry, is defined as C = A V&A = A n (qVx)) C Rd. In particular X E A belongs to codebook iff À E VyA. and |C| = g²h
The Voronoi code consists of the minimum energy represen- tative of each coset in A/qA = (Z/gZ)². Consequently, we can represent each coset, and hence, each codeword in C, as an element of z (Conway & Sloane, 1983).
NestQuant, SpinQuant and theory: As mentioned above, the use of nested lattice codes is rooted in theory. In (Or- dentlich & Polyanskiy, 2024) it was shown that nested lat- tice quantizers of nigh-dimensions attain the optimal rate- distortion tradeoff for matrix multiplication. Since the lat- tices used for proving that result do not admit efficient lat- tice decoding, here we resort to n-dimensional lattices con- structed as the Cartesian product of d copies of the Gosset lattice, whose dimension is d = 8. To understand how much loss in efficiency this leads to, Fig. 3 compares NestQuant, SpinQuant (uniform quantization with cubic shaping) and cioommttion-theoreti lower bound (1), for synthetic data. Details of this experiment can be found in Section 5.1. We can that NestQuant is reasonably close to the fundamen-
Assuming we have access to an oracle Q (x), which maps x E Rd to its closest point in A, the quantization (encoding) and dequantization (decoding) of a Voronoi code are de- scribed in Algorithm I and Algorithm 2, respectively. Here, G E ddxd generator matrix of A. The encoder first maps He to its nearest lattice point p Q (x). Since it only has a budget of d.R. bits for describing p. it only describes the coset of A A/gA it belongs to. This is done by sending v mod q. where V E 2d is the integer vector for which = Gv, referred to as the coordinates of p. Upon receiving v mod g. the decoder knows that Q^(x) E P+ and has to choose one point in this coset as the reconstruc-
6

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Algorithm 3 NestQuant
Input: A vector of size db, 9, array of & scaling coefficients , .g. Q4 QA-1 n integers in {0, 10.1...,9 1} representation
quantized
- integers in {l....,At k}]
> scaling coefficient
indices
- ||
normalization coefficient
Figure The quantization scheme of multi-hoe head attention. H is Hadamard rotation described in 4.3. Q is the quantization function described 4.2
A - Avn
for j = to - do
err = x
for p = do
tion & E RR It chooses è as the minimum energy vector in + qA, corresponding to the unique point in p qÅ n V4A- We have that á = (x( iff @x E Vya = gVx. When (x)x) $ A the quantizer is in overload.
- Aldj 1..dj + d]
enc +- Encode 8
recon +- Decode(enc) Spp
if err > recon - 0/ then
err + |recon - v12
Algorithm Encode
Q dj + 1..dj + -- enc
Input: x E IRd, QA p- (z) e + G-
B, + P
end if end for
coordinates of p >> quantized representation of p
return mod g
end for Output: QA, B,
Algorithm 2 Decode
Algorithm 4 Dot product
Input: e E 4-' Q^ p p Ge
Input: QA,, B1, and QA2, B2, $2 - representations of two vectors of size n = db from Algorithm 3, array B uns < 0
> equivalent to answer modulo qA
return p - q QN(?)
for j
do
In our experiments for this paper, we used the Gosset (Eg) lattice as A with d = 8. This lattice is union of Dg and Da + 2. where D& contains elements of 2 with even sum of coordinates. There is a simple algorithm for finding the closest point in the Gosset lattice, first described in (Conway& & Sloane, 1982). We provide the pseudocode for this algorithm together with the estimation of its runtime in Appendix C.
Pi Decode (QAi, [dj + 1.. + d)
P2 Decode(QAa)) dj + 1.. + d)
ans ans + (p1 82)
end for return ans
of the network. Let n be the number of input features to the layer. Following (Tseng et al., 2024a; Liu et al., 2024; Lin et al., 2024):
4.2. Matrix quantization
When quantizing matrix, we normalize its rows, and quan- tize each block of entries using the codebook. The algo- rithm 3 describes the quantization procedure for each row of the matrix.
If n 2t, set H be Hadamard matrix obtained by Sylvester's construction
Otherwise, we decompose n = 2mm such that m is small and there exists a Hadamard matrix H1 of size m. We construct Hadamard matrix H₂ of size 2 using Sylvester's construction, and set U H1 H2.
We can take dot products of quantized vectors without com- plete dequantization using algorithm 4. We use in the generation stage on linear layers and for querying the KV cache.
Note that it's possible to multiply an r x n matrix by H in O(rn log n) in the first case and Ornnloog n + m)) in the second case, which is negligible to other computational costs and can be done online.
4.3. LLM quantization
Recall that we apply a rotation matrix H to every weight- activation pair of linear layer without changing the output
7

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
In NestQuant, we quantize all weights, activations, keys, and values using Algorithm 3. We merge the Hadamard rotation with the weights and quantize them. We also apply the Hadamard rotation and quantization to the activations before linear layers. We also apply rotation to keys and queries, because it will not change the attention scores, and we quantize keys and values before putting them in the KV cache. Figure illustrates the procedure for multi-head attention layers.
4.6. Algorithm summary
Here we describe the main steps of NestQuant.
1. Collect the statistics for LDLQ via calibration data. For each linear layer with in-dimension d, we compute a x d "Hessian" matrix H.
2.
We choose initial set of scaling coefficients 8, and for each weight we simulate LDLQ quantization with these coefficients, getting a set of 8-dimensional vec- tors to quantize.
When quantizing a weight, we modify the NestQuant al- gorithm by introducing corrections to unquantized weights when a certain vector piece is quantized via the QA-LDLQ mechanism, described in section 4.5. It is based on LDLQ algorithm, described in section 1.1 of (Tseng et al., 2024a).
3. We a dynamic programming algorithm described in Appendix F on the 8-vectors to find the optimal ß-values for each weight matrix.
4.4. Optimal scaling coefficients
4. We also run the dynamic programming algorithm for activations, keys, and values for each layer. To get the distribution of 8-vectors, we run the model on a small set of examples.
One of the important parts of the algorithm is finding the optimal set of Br. Given the distribution of d-dimensional vectors that are quantized via Voronoi codebook, it is possible find an optimal set of given size exactly using dynamic programming approach, which is described in Appendix F.
5. We quantize the weights using QA-LDLQ and precom- puted B.
6. During inference, we quantize any activation before it is passed to the linear layer, and any KV cache entry before it is saved.
4.5. LDLQ with quantized activations (QA-LDLIQ)
We note that in the presence of activation quantization, the LDLQ and GPTQ algorithms for weight quantization be- come significantly suboptimal 3, thus necessitating a correc- tion (QA-LDLQ) that we describe here.
Note that we do not undertake any expensive (but surely useful) fine-tuning, such as optimizing rotation matrices or post-quantization training, as in (Liu et al., 2024) and (Tseng et al., 2024a), since our goal is demonstrating the basic primitive, not obtaining the absolute SOTA numbers.
Let W be the weight (shape a x n), X be a random original (unquantized) activation vector (shape n x 1), and 2 be X's quantization error, which we model random zero- mean noise (shape n x 1) independent of X. Then, if U is the quantized weight, the output quantization error becomes B(U) == W UUXUUXX UX + Z), so that the loss to be minimized is [[| ([T) instead of B[]l(W- - 2 that LDLQ and GPTQ minimize.
5. Experiments
5.1. Simulated Data
We compared the mean Le loss per entry of SpinQuant to the uniform Loe -scaling quantizer (used in SpinQuant and other methods). The mean L2 loss per entry for the product of two matrices A E Bank B E T.mx is computed as |AA"" -A""_ ll2 We set n k m = 4096 and sampled am two matrices A, B with unit normal distribution Aij. Bij (0, We compare to the lower bound from (1).
Lemma 4.2. Suppose Z is independent from X, ElZ = 0, and let H = E[XX 1 and J = E. Then for any set Co C Raxn
U² argmin |[])
UECQ
-argmin (W- - UH - 1 (6) UECO
For NestQuant, we do grid search over (g. k). For given q and k. we find the best subset in 2.. 50) of scaling coefficients / of size k using the algorithm from Appendix F. Then we calculate the expected bits per entry computed as log2 + 1 is the probability that the i'th beta is used in quantization. In Figure 3, we plot the efficient frontier of bits per entry vs root mean La loss.
where W- = -H +J)
Thus, QA-LDLQ consists of a) computing W; b) running the standard LDLQ but with matrix W as input and Hessian set to H + J. More details, practical aspects of QA-LDLLQ, as well as proof of the lemma are found in Appendix B.
8 Σ p(3;) log2 p(3:) where p(B²)
IIn fact, using original LDLQ Llama-3-70B produces perplexity at W4A4 setting due to significant outliers in layer O.
8

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Bits (W-A-KV)
Method
Llama-2-7B
Llama-2-13B
Llama-2-70B
Llama-3-8B
Llama-3-70B
16-16-16
Floating point
5.47
4.88
3.32
6.14
2.86
4-16-16
QuaRot
5.60
5,00
3.41
QulP#
5.56
4.95
3,38
OstQuant
5.64
4.94
3.41
6.53
3.19
NestQuant
5,53
4.93
3.38
6.31
3.14
NestQuantM
5.55
4.95
6.35
4-16-4
NestQuant
5.57
4.96
3.39
6.37
3.19
NestQuantM
5.59
4.99
6.49
4-4-16
SpinQuant
5.9
5.2
3.8
7.1
OstQuant
5.60
5.14
3.57
7.24
3.97
DuQuant
6.08
5.33
3.76
8.06
4-4-4
QuaRot
6.10
5.40
3.79
8.16
6.66
SpinQuant
5.9
5.3
3.8
7.3
OstQuant
5.91
5.25
3.59
7.29
4.01
NestQuant
5.67
5.03
3.49
6.63
3.61
NestQuantM
5.73
5.07
6.82
Table 2.
The wikitext2 perplexity at context window 2048 various quantization methods of Llama models.
5.2. Llama results
Bits
Bits (no zstd)
W
+ KV
KV
14
3.99
4.06
6.308
6.379
6.6.3
We quantize the Llama-3-8B model (Llama Team, Al @ Meta, 2024) using different values of q. We choose the num- ber of scaling coefficients (k) to be equal to the Section H.I explains the rationale behind this choice. More details in Ap- compute the number
6.376
10
3.50
3.57
6,486
6.640
7,251
3.18
3.25
6.700
6.968
7.989
Table 3. Wikitext2 perplexity of NestQuant quantization of Llama- 3-8B different rates. The "bits" column is the bit rate per entry with zstd compression of scaling coefficients, and "bits (no zstd)" is the bit rate without compression. The "W", "W+KV", and "W+KV+A" describe the quantization regime (whether weights, KV cache, activations quantized). The perplexity of non- quantized model is 6. 6.1
the hyperparameter choice of the experiments pendix G. For each experiment, of bits per entry similar to Section 5.1, but for the setup of compressed 3 indices, we run the zstd compression algo- rithm instead of using the entropy of the distribution. As our evaluation metric, we use the perplexity on the validation split of wikitext2 with context length 2048.
We also perform the evaluation of NestQuant various zero-shot benchmarks: ARC-Easy and ARC-Challenge (Clark et al., 2018), Hellaswag (Zellers et al., 2019), (Bisk et al., 2020), and Winogrande (Sakaguchi et al., 2020). The results on 4-bit models with comparisons to other models are summarized in Table
NestQuant consistently achieves better perplexity metrics across different models for both weight-only regime and full quantization. In fact, for all models we have tested, except Llama-2-7B, NestQuant with W4A4KV4 (4-bit weights, activations, and KV-cache) quantization even outperforms previous works with W4A4KV16. On W4KV4A4 (4-bit weights, KV-cache, and activations) quantization of Llama 3-8B we achieve perplexity score of 6.6 compared to 7.3 in SpinQuant and OstQuant. Even without LDLQ, we achieve perplexity score of 6.8, which is still better. Fi- nally, NestQuant outperforms QuIP# for weight-only Llama- quantization.
5.3. Comparison to other methods
In comparisons to other methods, we focus on 4-bit setup, choosing g = and k = 4. We show the WikiText2 perplexity comparisons for multiple Llama models in ta- ble 2, and other benchmark comparisons (for Llama-3-8B) in table 1. The methods that we include in the table are SpinQuant (Liu et al., 2024), QuIP# (Tseng et al., 2024a), QuaRot (Ashkboos et al., 2024). DuQuant (Lin et al., 2024), and OstQuant (Hu et al., 2025). On the WikiText2 dataset, we computed the perplexity scores of the quantized models on context size 2048
In addition, evaluate simpler version of the algorithm that is easier to in hardware that is called NestQuantM. More detailes on it can be found in section D.
We present ablation studies for our design choices in in Appendix H.

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Impact statement
References
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Agrell, E. and Allen, B. On the best lattice quantizers. IEEE
Transactions on Information Theory, 2023.
Andoni, and Indyk, P. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM, 51(1):117-122) 2008
Acknowledgments
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B.,
We want to thank Nikita Lazarev and Shang Yang for im- portant discussions that helped us to implement an efficient CUDA kernel, well Guangxuan Xiao for discussions about the algorithm.
Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in ro- tated 1lms. In Advances in Neural Information Processing Systems 38 (NeurIPS 2024), 2024.
This work was supported in part by the National Science Foundation under Grant No CCF-21311 15 and the ISF under Grant 1641/21. We thank Foundry.ai ai and Lambda Labs for compute resources.
Bisk, Y., Zellers, R., Bras, R. I.., Gao, J., and Choi, Piqa: Reasoning about physical commonsense natural language. In Proceedings of the 34th AAAI Conference Artificial Intelligence (AAAI 2020), pp. 7432-7439, 2020.
Research was supported, in part, by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator under Cooperative Agree- ment Number FA8750-19-2-100
Charikar, M. Similarity estimation techniques from round- ing algorithms. In Proceedings of the 34th Annual ACM Symposium Theory of Computing (STOC 2002), pp. 380-388, 2002.
The views and conclu- sions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government authorized reproduce and distribute reprints for Gov- ernment purposes notwithstanding any copyright notation herein.
Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems 37 (NeurIPS 2023), 2023.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https:/ //axxiivvvoo./// abss/ 05457.
Conway, and Sloane, N. Fast quantizing and decoding algorithms for lattice quantizers and codes. IEEE Trans- actions Information Theory, 28(2):227-332, 1982. doi: 10. 10.1109/TIT. 1056484..
Conway, and Sloane, N. A fast encoding method for lattice codes and quantizers. IEEE Transactions Infor- mation Theory, 29(6):800-824 1983. doi: 10. 109//TT 1983.10
Datar, M., Immorlica, N., Indyk, P., and Mirrokni, V. Locality sensstivee hashing scheme based p-stable dis- tributions. In Proceedings of the 20th Annual Symposium on Computational Geometry (SCG 2004), pp. 253-262, 2004.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int80: 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pp. 29498-29511 2022.
10

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Dinur, I., Kindler, G., Raz, R., and Safra, S. Approx- imating CVP within almost-polynomial factors is np-hard. Combinatorica, 23(2):205-243, 2003. doi: 0177000493-00--001999
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR 2017), 2017.
NVIDIA Corporation. nvcomp: Gpu-accelerated com- pression library, 2025. URL htttps://devellee
Frantar, E., Singh, S. P., and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. In Advances in Neural Infor- mation Processing Systems 36 (NeurIPS 2022), 2022.
nvidia com/ comm//copp. Accessed 2025-01-30.
Ordentlich, O. and Polyanskiy, Y. Optimal quantization for matrix multiplication, 2024. URL https:t/paaaat al// http//as/2411.113733
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. In Advances in Neural Information Processing Systems 37 (NeurIPS 2023), 2023.
Polyanskiy, Y. and Wu, Y. Information Theory: From Cod- ing to Learning- Cambridge University Press, 2024.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.
Gersho, A. Asymptotically optimal block quantization. IEEE Transactions on Information Theory, 25(4):373- 380, 1979.
Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI 2020): pp. 7732-8740, 2020.
Hadad, R. and Erez, U. Dithered quantization via orthogonal transformations. IEEE Transactions on Signal Processing, 64(22):5887-5900, 2016.
Stam, A. J. Limit theorems for uniform distributions on spheres in high- euclidean spaces. Journal of Applied Probability, (11):22112288 1982.
Hu, X., Cheng, Y., Yang. D., Xu, 7.,, Yuan, Z., Yu,J. Xu, C.,, Jiang, Z., and Zhou, S. Ostquant: Refining large language model quantization with orthogonal and scaling transfor- mations for better distribution fitting. In International Conference Learning Representations (ICLR 2025), 2025.
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and Sa, C. D. Quip#: Even better Ilm quantization with hadamard inco- herence and lattice codebooks. In Proceedings of the 41 International Conference on Machine Learning (ICML 2024), volume 235 of Proceedings of Machine Learning Research, pp. 48630-48656, 2024a.
Hung, A. C. and Meng, H. Multidimensional rotations for robust quantization of image data. IEEE Transactions Image Processing, 701):1-12, 1998.
Tseng, A., Sun, Q., Hou, D., and Sa, C. Qtip: Quan- tization with trellises and incoherence processing. In Advances in Neural Information Processing Systems 38 (NeurIPS 2024), 2024b.
Juang, B. and Gray, A. H. Multiple stage vector quantiza- tion for speech coding. IEEE International Conference on Acoustics, Speech, and Signal Processing, 1982.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning (ICMI. 2023), volume 202 of Proceedings of Machine Learning Research, pp. 3732-3754, 2023.
Li, J., Li, F., and Todorovic, S. Efficient riemannian opti- mization on the stiefel manifold via the cayley transform. In International Conference on Learning Representations (ICLR 2020), 2020.
Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z., and Wei, Y. Duquant: Distributing outliers via dual transformation makes stronger quantized 1mms. In Advances in Neural Information Processing Systems 38 (NeurIPS 2024), 2024.
Zamir, R. Lattice Coding for Signals and Networks: A Struc- tured Coding Approach to Quantization, Modulation, and Multiuser Information Theory. Cambridge University Press, 2014.
Zamir. R. and Feder, M. On lattice quantization noise. IEEE Transactions on Information Theory, 42(-4)1152-1115, 1996.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary. D., Krishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant: Lim quantization with learned rotations, 2024. URL https: /ttpss///axvvv. wwwiv.wvvoooo org/ abs/ 2405. 6..
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics (ACL 2019), pp. 4791-4800 2019.
Llama Team, AI @ Meta. The llama herd of models, 2024. URL https: //tpxvv.vvv//////////a///aaaa/la org/ 2207/2407.0 21783.

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMS
A. Figures
1-
Figure 5. Complement Gaussian measure a 8-dimensional cube (corresponding to shaping using an for ball), a Voronoi region of the Gosset lattice Es (corresponding to shaping using Voronoi codes with base lattice Ess,, and a Euclidean ball (corresponding to shaping with a ball, which does not admit efficient implementation)
B. Details of QA-LDLQ
LDLQ details. We briefly mention the details of LDLQ (for more details we refer the reader to (Frantar et al., 2023) or (Chee
et al., 2023)). The main observation is that we want to minimize |[W - where U is the quantized weight and X the matrix of activations. We observe that IIW x12 = [I(W - LU)xxVDa | where LxDx and Lx is uni-lower-triangg triangular matrix and Dx is diagonal. LDLQ proceeds by optimally quantizing the last column of U (for example using the quantizer for NestQuant). It then recursively removes the feedback i, (W - U)(Lx E),, from the rest of the columns (here E; is the zero matrix with the ii index set to 1). This may be written compactly as U Q(W - (W- - ULx - I)) where Q quantizes each column separately.
H E|X ]]
Motivation. When quantizing Llama-3-70B with original LDLQ, we have noticed very poor (ppl ~ 10+) performance. We have observed that the reason for this is quantization of activations for small subset of linear layers (4 out of 560), and if they are left in full precision, the perplexity becomes more reasonable (less than 4). We have found them by choosing the layers with the largest amplification ratio- a concept we define next.
Consider a weight W (shape x n), where n is the number of in-features, and a is the number of out-features. We define B[IX] ||||||a]
amplification of random vector X (n x 1) by W as a(W, X)
Then, if X is the distribution of input a(W.Z)
activations, and Z is a random Gaussian vector, we define the amplification ratio for W as a(W.X). We note that if layer has large amplification ratio, its activations are harder to quantize, because the magnitude of quantization noise will be increased much more significantly than the magnitude of the activations themselves. One extreme example of this issue is the value projection of the attention of the first transformer block in Llama-3-70B, This layer has an amplification ratio of ~ 157, as computed from 10 wikitext2 sequences of length 2048. This makes naive 4-bit quantization of activations nearly impossible, as even small perturbations of the input of the layer are greatly amplified in the output.
We have developed QA-LDLQ to mitigate the issue of large amplification ratio. We model quantization noise as a random Gaussian vector with mean O and covariance matrix J = e² I, where E² depends on the quantization rate and the statistics of X. For large values of c² the modified weight matrix W = ( + f) is more robust to random perturbations of the input (thus, decreasing the amplification ratio), but on the other hand, large e² results in a larger bias, as expressed in the term C(W, H) below. Figure 6 demonstrates this tradeoff for the value projection layer mentioned earlier.
Proof of Lemma 4.2. Recalling the definition of W = WH + J)- that X and Z are statistically independent, and Z has zero mean, and that H, J are positive semi-definite symmetric matrices, we have
E )( = El[(W - U) - U2 (W - UU - -U)T + UJUT (W U)(H + J) HW,,
(7)
(8)
12

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Noica ampitication
Noise ampifiiction
00
60
40
2-
20
10-1
10-6
10-1
10-4
10-
10-2
10
1 for outputs
1-A² 2 outputs
Figure 6. We run QA-LDLQ for value projection layer of the first transformer block of Llama-3-70B. We try different values of e on logarithmic scale from 10-5 5 to For each e, we find modified weight W, and plot the amplification ratio W in y-axis, as well as how close the outputs the weight W to the outputs weight W. The value on x axis is defined as R² == SIWX-WXX (ar(WX) where X contains activation inputs from 10 sequences of length 2048 from wikitext2. The right plot is bottom right corner of the left plot, zoomed in. note that by paying small price the accuracy of weight, we can reduce amplification ratio dramatically.
where
H,
(9)
is independent of U, and therefore does not affect the minimization.
C. Gosset oracle
In this section, we discuss the algorithm for finding the nearest neighbour in Eig lattice and estimate its performance in FLOPs (Floating Point Operations). We note that Es Ds Ds + 2. where D3 contains vectors in Zg with even sum of coordinates. To compute VE, (z), we compute two candidate points: VD, (x) and z2 D (z), and choose the one that has smaller T distance to x.
To get (x)) we can round each coordinate to the nearest integer. the sum rounded coordinates is odd, we need to "flip" the rounding direction of the coordinate for which the flip would cost the least. Note that finding the closest point in VD++ works the same, but the rounding grid now contains half-integers, not integers.
In algorithm 5, we first round our vector down (getting d) and compute the mask (g) of whether it's optimal to round up for Dg. We note that the optimal rounding for Dg + 2 is d + 0.5, while the optimal rounding for Dg is d + g.
We want to understand whether rounding to Dg or Dy + is better. Let dist, be the absolute distance from the i-th entry J; E (d, d3 + 1] to the middle of this integer segment dl + 0.5 = 22,1. We note that the contribution of this point to the MSE for D& is (0.5 dist,)², ,, while for Dg + 2 is dist?. The difference is: 0.25 dist, + dist? dist? 0.25 dist,. If the sum of this value over negative (i.e. Σ dist, 2), it's optimal quantize Ds. otherwise Ds + z. In pseudocode, store Σ dist, A
We note that we should check the constraint that the sum of coordinates in D& is even, and if it is not, "flip" one of the rounding directions. The optimal coordinate to flip can be determined through dist, and the new value of flipped coordinate through g. We also need to update A given that the MSE difference changes.
The full pseudocode of the algorithm is in Algorithm
13

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Algorithm 5 Oracle for the Gosset lattice
Input: R
2: d t floor(a:)
3: #2 d + 0.5
4: g - (z x2)
5: - 2-9-1
6: - g
7: disl +- (x-xx)) -x2
8: A + Σ: dist,
9: if , F1, odd then
10:
pos argiinn dist
11:
31,pos poss $1,s
12:
A - + distpos 1
13: end if
14: if
Σ; 22, odd then
15:
pos = argmax dist
16:
22.pos + 22000s + p2,pos
A A +1 2 dist pos
18: end if
19: if A >>> then
20: return 2
21: else
22: return 22
23: end if
D. NestQuantM algorithm
Due to the high complexity of argmin and argmax operations algorithm for the fast hardware implementation, we propose a different, more simple version of NestQuant decoding. Instead of using argmax and argmin on lines 10 and 15, always assign pos to be equal (thus, indicating that we always "flip" the rounding direction of the first coordinate fix the parity). Note that this change only applied in the decoding stage, while during encoding the algorithm, which keeps the granular error the same. Let's denote our modified Gosset oracle as f: IR + R
the full version of
Lemma D.1. For vector R and vector Es, f(x + e) f(z) +
of coordinates. The modified Gosset oracle uses modified D& oracle g, and at imput constructs two candidate points (x) Ds and c2(x) D3 + 1 Then, the algorithm chooses the closest point among these candidates to æ. Note that e1 (x) g(x) and c2(x) = g (x 2) +
Proof. Recall that Dy Z contains all the integer vector with
We will prove that for u E Ds. g(x + u) = g(x) + u for any x E R Now, let's show the original lemma. Note that since we are choosing the closest candidate point, the condition on candidate sets {cr(# + 02 + v}} = (x), (2(2) + holds, then f(x + v) f(x) + u. Now, consider two cases:
1. Dg. Then:
(x + v) = g(x + v) = g(x) +u= (z) +
1
1
1
c2 u) = 9 2
+ = 9 x
+
+ c2(x) +
2
2. Dg + 1.4 Then, say that = + for u, E Ds.
14

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
ci + v) g(x + u) = x + u
2) x x + u
1
2/2
2) + + - c2())
2
1
(2 x 2 + + = g( + +w) + 2 = g(x) + w + 2 = g(z) + +o= -1)=
Thus, in both cases the condition candidate sets holds, and get f(x + + f(x) u.
Now we show that if u E Ds. g(x + u) = g(x) + u. Note that when evaluating g(x + u). we will get the same rounding directions as in (x)), since u is an integer vector. Since has even sum of coordinates, the parity will also be the same. Then, our decision to flip the rounding of the first coordinate will also match as well. Therefore, the vector between original and rounded coordinates will be the same:
g(x) -(x u)
g + u) =(xx)=
Let be the vector we obtain after rounding to Es, e be the lattice coordinates of v, and G be the generating matrix of the lattice. The compressed representation is e mod which corresponds to point v' = G(c mod q). The reconstructed point é is defined to be - -gf(v'/g by the decoding algorithm. Note that o' E g Let's assume that f(v//) = 0. Then:
1/ ) = v/ qs -''-0 = d - ( ( i= = u' - qf
1/
q
9
9
We have used Lemma D.1 in the third equality and our assumption in the fourth equality. Given this fact, we conclude that when using NestQuantM for decoding, the composition of encoding and decoding functions (for a fixed 3) is similar to the original NestQuant, except the shaping region has changed to the set of points u E Es such that f(v/q) = O. Since close to the original Gosset oracle, we expect this region to still capture Gaussian probability density well. In case of overload errors, we are still able to choose larger value of beta due to multi-beta strategy.
E. CUDA Kernel Implementation
The decoding algorithm 2 consists of a basis change p f- Gc and a subsequent computation of the coset of
p
Q
L
We leverage the asymmetry between encoding and decoding choosing
which fast to decode, In particular we use
0
0
2
0
0
0
0
0
0
2
0
0
0
1
0
0
0
0
0
2
0
G
=
1
4
2
2
2
2
2
2
0
0
2
0
0
0
0
0
0
G is not basis for Eg but for 2Eg, We use G because we want to work integer-lattice half-integers cannot be represented with integers. Moreover, we represent 8-vector 7 as two 32 bit integers each representing the two parts of the vector Zo, 31, Ø2, 33 and 24, I5, TG, I7.
For reference, the CUDA implementation of the implementation of + Gc.
15

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
device void f_st
2
const uint32_t enc, uint32_t cuto, uint32_t+ outl
3
uint32_t uint32_t t edd (enc & OXFOF0F050) >> 4;
0X0FOFOFOE;
6
7
3int32_t twe_even even << 1;
uint32_t twe_odd odd << 1;
uint32_t x0 even 0xFF;
uint32_t concatenator (1 << 24) (1 << 16) (1 << 8) 1,
12
uint32_t x0_concat x0 concatenatoor
uint32_E temp_ even vadd4 (two_even even OXFFFF3F00, x0 c0__oncat);
uint32 temp_ode odd vadd4 stwe_odd, x0 x0__oncat);
16
32_t s1 dp4a (two_even, (uint32_t) 0x01010100, (uint32_t) ;)
3int32_t s2 dp4a (two_odd, (uint 0x01010101, sl);
+out temp_evenj *outl temp_odd + s2;
The computation is done by decomposing into a sum of different matrices.
0
0
0
2
0
0
0
0
0
0
0
0
2
0
0
0
2
0
0
0
0
0
G=
+
4
2
2
2
2
2
000000000
2
0
0
0
0
0
0
0
O
O
2
0
0
0
0
0
0
O
0
0
0
2
0
0
0
O
O
2
0
0
0
0
0
2009
0
100
0
2
0
0
2
2
2
2
2
0
0
2
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
The next step is to compute and the coset of merge both operations into one function.
device void secode_nestquuat
uint32_t *enc, int *B_ local locddddeeeddd,, deccde, const int
4
5
uint32_t inp0, inpl; unsigned int dist [2];
const uint32_t MASK_FRACPART 0x1F1F1F1F;
const uint32. MASK INTPART = OXE0E0E0E0;
const int HALE _CONCAT 0x10101010;
const int TRD 0x40;
const int ONE 0x20; 13
const int HALF 0x10;
G_q_fast (enc [0], ainp0, simpl);
int32_t fracPart0 inpo MASK_FRACPART
16

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
int32_t fracPart1 inpl
MASK_FRACPART,
19
int32_t integerParto inp0 &
int32_t integerPartl = inpl & MASK_ INTPART;
int g0 (fracPartO & SALE_CONCAT] << 1;
24
int g1 (fracPart1 & HALF_CONCATE << 1;
int32_t change ( (g0 ONE) HALF); int32_t suml gem_sum (integerParto integerPartl);
28
int32_t two_parity:
(suml ONR) >> 4; vsub4 (fracPartO, (int32_t) (two_parity_ change) & 0xPP);
int32_t FracPartO_1
int32_t sum2 get sum(go, gi) suml; int32_t two paritta 2 (sum2 ONE) 4; vadd4 (fracPartO, (int32_t) t) (two Eawo_parity_ 2 change) & OxFF),
38
int32_t fracPartO
dist[0] vabsdiffs4 (fracPartO, HALF HALF_CONCAT); dist[1] vabsdiffs4 (fracPartl, HALF_CONCAT); [0], (1));
int Delta get_sum (D],
58
int32_t dist00 (dist [0] OxFF); Delta (two_ [two_parity_1 two_parity_2) 2) dist00;
Delta twe_parity_1 4;
42
if (Delta <- TWO)
i_local_decode [0] vsub4 (fracPartO_l, HALF CONCAT)
i_local_decode [1] vsub4 (fracPartl, HAGF_CONCAT);
else
47
B Alocal_decodel [0] vsub4 (fracPartO g0);
B local Llocal_decodel [1] vsub4 (fracPartl, gl);
Remember that for g 16, we would need to divide by 16. We used basis for 2 and not Fig every element is doubled. This is why in our code is represented by 16, 1 is represented by 32, and 2 is represented by 64. In order to avoid using the round function explicitly (which is not parallelized in hardware) we compute the integer and fractional part of the vector using masks (as can be seen in lines 17-21).
We compute g in two parts Do and 91 (see lines 23 and 24). This is done by checking that the fractional part is greater or equal (equivalently in our scaled basis, that the fractional part has the fourth LSBB one). We leverage the fact that
x- ([x] 0.5) = {x} 0.5
(lines 44 and 45) and
x- r-rndd((s)
|x| {x}
[[= + [x)
(lines 47 and 48). To compute the bitflip, compute 291 (corresponding to the first element in the vector, see line 26 for reference). We add (subtract) it from fractional part based on the parity check we compute in lines 28 and 32 respectively.
To decode betas in our kernel, we encode the betas as indices to a predefined dictionary (of size 4). Thus, each beta can be represented using bits.
1 int betal
>> shift) & 0x3;
int beta2 (beta_packed >> (shi ft + 2) 0x3; int decoded_ l beta_dict [betal];
int decoded beta2 beta dict [beta2];
Encoding and decoding kemelss share the same logical structure that the complex mapping defined for beta is shared (this allows fast contiguous reads of multiple betas at once).
17

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Table Runtime comparison of GEMV kernels 8192
8192 matrix using an NVIDIA A A100 GPU.
Method
Time (μt)
Baseline (16 bits)
97
NestQuantM (4.25 bits)
60
QulP# (2 bits)
38
QulP# (4 bits)
755
int4 uniform
31
E.1. Runtime comparison of GEMV
The QulP# computation involves invoking two calls to QulP# (2 bits), so we extrapolate the running time based QulP# (2 bits).
F. Dynamic programming for optimal B
Recall that instead using one instance of lattice codebook C, we use a union of codebooks C' scaled to different coefficients. Specifically, our final codebook C is parameterized by coefficients B1 S B2 S.... ... S Bk, and is equal to:
C C=BICUBCU...
Given a set of 8-vectors to quantize, we can find the set B that minimizes reconstruction error using a dynamic programming algorithm, which described in Appendix F.
When quantizing vector to the i-th scaled codebook, could either get small granular error when the vector is in V (0), or large overload error otherwise. If we use codebook with smaller B, we have larger chance of overload error, but the expected magnitude of granular error is smaller due to the volume of Voronoi region being smaller (Figure 7). We can have two strategies for encoding:
6.33
0.3
0.5
0.12
6.00
0.03
0.54
16
Ista
(a) Overload error probablity
(b) Granular RMSE for five sampled vectors
Figure 7. Granular and overload error for standard Gaussian vectors, 16
1. First-6: Use the smallest 3, which does not result in an overflow error.
2. Opt-3: Try all the values of 3, and choose the one that has the smallest reconstruction MSE.
Even though Opt-3 should provide smaller error, the definition of First-3 will be useful for us. We can note that the difference between error for Opt-3 and First-9 is not very siginificant (Table 5). Moreover, First-B can be used to determine the optimal set of 3; to use.
Let we have n samples U1, U2.... Un from the distribution vectors we are quantizing, and a large set of betas B, containing 31 < 82 < ... < Bm, from which we want take the optimal subset of size & which minimizes the loss under
18

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
K
2
4
6
8
10
OPT-3 FIRST-B
0.0878 0.0878
0.0795
0.0708
0.0669
0.0646
0.0798
0.0712
0.0676
0.0656
Table Mean RMSE for reconstructed iid standard Gaussian 8-vectors, q 16, betas uniform [0, 10].
First-8 strategy. For each vector vy and beta 3 we compute mseij - the MSE if we use Bg to quantize e; and overloads) whether overload error occurs in this scenario.
We solve the optimization problem with dynamic programming. Let's define dpi be the mimimum get if we have to quantize all the vectors which do not yield overload error for 3;, using B; and j smaller betas and First-8 strategy. If is large enough so that no vector has an overflow eπror on 3,, dpss has the answer to the problem. To compute the value of dpij. we can iterate over - the index of second largest beta in the set (the largest being 82). Then, the recalculation works the following way:
of MSE
dpis + min dpe--t
msEpi p.cond,
Σ
where cond, = overloadps A overloadd
By following the transtions this dynamic programming, we can reconstruct the optimal set of 3.
Algorithm Dynamic programming for finding the set of B
1: Input: vectors U, beta set B, m8Cij, overload.j
m, 2: dpi,j oc for in 0... m.m.m,, j in 0. ..k
3: from::j = null for i in O.. in O...k
4: dpo,o = 0
5: for = 1 to m do
6: for j = 1 to k do
for s = to do
8:
cond,- overlondps -overloadgi for p E 1...
9:
cost = =, cond, msEpi
10:
if dpij > dpaj-1 + cost then
11:
dpij + dpa.j-1 + cost
12:
fromi t
13:
end
end for
15: end for
16: end for
17: Let pos is chosen that Bpos has overflow enrors
18: result []
19: for = downto do
20: apsuul.app))
21: pos f- fromposs
end for
G. Llama experiment details
choose the train split of the Wikitext2 (Merity et al., 2017) dataset as a calibration dataset for computing H, and evaluate the model the validation split, computing the perplexity metric. For step 2 in the algorithm (Section 4.6), we
19

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
select 3 [3.5, 4.5, 6.0, 14.5, 25.0)/q, because is the / we get when optimizing them for weight quantization without consideration of L.D.Q. The overall universe of betas contains values from 1 to 40 with spacing ranging from 0.25 to 2. For running DP activations, keys, and values, we run the model batch of 6 full-length sequences, which is sufficient for this -ww-dimensional hyperparameter.
When choosing maximum beta for given distribution, we add a margin of 3.0 for weights and 4.0 to the maximum beta needed to have 0 overload errors on known data to account for potential overload errors unknown data. While small number of overload error does not affect perpixity significantly, we still aim to minimize their probability.
When computing perplexity for Wikitext2 with given context length, we average the perplexities for all the positions, which is standard for other works in quantization of LLMs.
H. Ablation studies
We found LDLQ to be useful in improving the quality of quantized model. In table 6, we compare the wikitex12 perplexity of models with and without L.I.Q.
Algorithm
W
W KV
W+KV
NestQuant
6.308
6.379
6.633
NestQuant (no LDLQ)
6,528
6.605
6.849
Table Effect of LDLQ on NestQuant (g = 14 and l: = 4) wikitext2 perplexity
While Hadamard matrices from Sylvester construction are commonly used in other works (QuIP#, Quarot), there are multiple ways to construct fast rotation for the case when dimension is not power (such as the down projection in MLP of Llama-3). We tested three possible options for rotation q 14, 1: 4, W KV A quantization.
Algorithm
W KV+A
Fourier S % H, S orthogonal, H Sylvester Hadamard
6.773
6,770
H, H, bardeoded Hadamard. Sylvester Hadamard
Table 7. Effect of rotation NestQuant 14 and k 4) wikitex12 perplexity
H.1. The choice of k
The value of k; i.e. the number of scaling coefficients is important hyperparameter of the algorithm. With increase of k, we decrease the quantization eиror by allowing each vector to be quantized to the lattice point with proper scaling, However, it increases the bitrate and makes the encoding slower, since we need try larger number of scaling coefficients.
We used A: = 3, 4, 5, 8 to quantize Llama-3-8B across different values of q, plotting the resulting perplexity against bitrate in Figure 8. We can see that using = 3 leads to a suboptimal performance of the quantization scheme, while the performances of l = 4, 5, 8 are comparable. our experiments, we use 4, because lower having results faster encoding.
Results for Llama3.2-1B
Here, we show the results of NestQuant on the newer 1B parameter model LI L.amm3.2--1B. We do experiments in the same setups as for the Llama-3-8B model, computing the wikitext2 perplexity.
Bits
Bits (no zstd)
+ KV
W W+KV+ +
14
3.99
4.06
10.061
10.529
11.197
10
3.50
3,57
10.377
11.552
14.191
3.25
Table Wikitext2 perplexity NestQuant quantization of Llama-3.2-18 3.2 The format of table is the same Table 3. The perplexity of non-quantized model is 9.749
20

---

NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
kmn
Pull precision (FP16)
9.0
wik
8.0
7,5
6.5
2.8
3.0
3.2
3.4
3.6
3.8
4.0
4.2
bits
Figure The perplexity-bitrace scaling of NestQuant with different values of all components of the model (weights, cache, activations) are quantized
J. Results for 3-bit model quantization
We publish the results for 3-bit quantization weights and activations on small models (Llama-3-8B and Llama-2-7B). use q 7 and k which results 2.98 bits per entry.
Bits (W-A-KV)
Methud
Lluma-2-7B
Llama-3-815
16-16-16
Floating point
6.14
4-4-16
NestQuant
5.64
6.56
3-3-16
NestQuant
21


Reviews:
[{"review_id": "Cz1gZ7Qbpj", "reviewer": "Reviewer_a2PJ", "summary": "The presented work proposes to use in an improved quantization technique for LLMs that wastes 17% less quantization \"space\". Written well and very balanced (theory + practical verification).\n\n## update after rebuttal\n\nInitially my own (internal) score was between 3 and 4 and I rounded up to 4, because I found the presentation compelling enough and the results good. The very healthy discussion here reaffirmed me that the submission should be accepted (for me it is now a solid 4) and hope the authors add all the additions discussed to the final version. Thank you to the authors for also actively supporting the rebuttal!", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Indeed we believe that this work may be interesting also to researchers outside the ML community. In particular, NestQuant consists of a combination of various ideas, including random rotations, nested lattice quantization for the induced Gaussian-like vectors, the overload avoidance mechanism with multiple beta values together with Algorithm 6 designed for choosing those values.\n\nThese ideas span information theory, communications, and even pure math (the lattices used for MatMul quantization should have properties that were not previously studied - in particular, their covariance matrix should have a small Frobenius norm). We certainly hope this work will initiate a broader interest in PTQ for LLMs within the mentioned research communities.\n\nIn addition to the evaluations provided in the paper, we also perform additional evaluations of NestQuant on models from Llama 2 series (7B, 13B, 70B), and Llama-3-70B (though we didn't have enough time to finish simulation of W+KV+A quantization for this model). The table below shows the Wikitext2 (seq. length 2048) perplexities of NestQuant and other PTQ methods, with NestQuant outperforming state of the art for most models and setups.\n\n| Bits (W-A-KV) | Method         | Llama-2-7B | Llama-2-13B | Llama-2-70B | Llama-3-8B | Llama-3-70B |\n|---------------|----------------|------------|-------------|-------------|-----------|-------------|\n| 16-16-16     | Floating point | 5.47       | 4.88        | 3.32        | 6.14      | 2.86        |\n| 4-16-16      | QuaRot         | 5.60       | 5.00        | 3.41        | -         | -           |\n|               | QuIP#          | 5.56       | 4.95        | **3.38**        | -         | -           |\n|               | OstQuant       | 5.64       | 4.94        | 3.41        | 6.53      | 3.19        |\n|               | NestQuant      | **5.53**       | **4.93**        | **3.38**        | **6.31**      | **3.14**        |\n|               | NestQuantM     | 5.55       | 4.95        | -           | 6.35      | -           |\n| 4-16-4              | NestQuant      | 5.57       | 4.96        | 3.39        | 6.37      | 3.19        |\n|        | NestQuantM     | 5.59       | 4.99        | -           | 6.49      | -           |\n| 4-4-16       | SpinQuant      | 5.9        | 5.2         | 3.8         | 7.1       | -           |\n|               | OstQuant       | 5.60       | 5.14        | 3.57        | 7.24      | 3.97        |\n|               | DuQuant        | 6.08       | 5.33        | 3.76        | -         | -           |\n| 4-4-4        | QuaRot         | 6.10       | 5.40        | 3.79        | 8.16      | 6.66        |\n|               | SpinQuant      | 5.9        | 5.3         | 3.8         | 7.3       | -           |\n|               | OstQuant       | 5.91       | 5.25        | 3.59        | 7.29      | 4.01        |\n|               | NestQuant      | **5.67**       | **5.03**        | **3.49**        | **6.63**      | TBD         |\n|               | NestQuantM     | 5.73       | 5.07        | -           | 6.82      | -           |\n\nIn this table, there is a method \"NestQuantM\", which corresponds to NestQuant with a simpler, but more computationally efficient E8 lattice oracle used in the decoding. We evaluated NestQuantM for Llama-2-7B, Llama-2-13B, and Llama-3-8B\n\nWe have attached a figure: https://imgur.com/a/bNW1D7K for reference.", "claims_and_evidence": "The claim of improvements by using an improved technique is proven by experimental results (up to 20% relative).<", "methods_and_evaluation_criteria": "All standard evaluation criteria have been applied, also perplexity which is known to be most sensitive.", "theoretical_claims": "Validated to be correct. I certainly appreciate the appendices much!", "experimental_designs_or_analyses": "Good.", "relation_to_broader_scientific_literature": "As the technique has been inspired by other work (Voronoi regions), I expect influence of a broader audience than just ML.", "essential_references_not_discussed": "Everything good.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8VE04XVmYO", "reviewer": "Reviewer_r651", "summary": "The authors propose NestQuant, a novel lattice-based quantization method designed to improve weight-activation quantization for LLM.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and providing valuable feedback! Here, we provide the answers to the questions.\n\n### **Evaluation on other models**\n\nWe have conducted additional experiments of NestQuant in compressing other LLMs. We chose models in Llama-2 series (7B, 13B, and 70B) and Llama3-70B, as most popular in the prior quantization literature (thus facilitating comparison). We considered three regimes: weight only, W+KV, and W+KV+A. We quantize the models to < 4 bits by choosing ($q=14$) for a fair comparison with other methods.\n\nPlease refer to the table in response to reviewer a2PJ for Wikitext2 perplexity results with context length 2048. NestQuant consistently achieves better perplexity metrics across different models for both weight-only regime and full quantization. In fact, for all models we have tested, except Llama-2-7B, NestQuant with W4A4KV4 quantization even outperforms previous works with W4A4KV16.\n\nWe hope that the new evaluation results would provide more evidence for robustness of the performance advantage of NestQuant.\n\n### **Comparison with DuQuant and OstQuant**\n\nIndeed we should have included these important references, and we thank the reviewer for this comment. We now compare our numerical results to the results reported in those references under the same setup. In all setups we’ve tested, NestQuant improves over the ppl reported in those papers. On a deeper level, both DuQuant and OstQuant can be viewed as “pre-processing” techniques that create weights and activations that are “easier” to quantize, while the quantization method used there is essentially round-to-nearest (RTN). The main innovation in NestQuant is the ability to use quantizers that are more effective than RTN, at the expense of additional cycles in encoding/decoding complexity. These quantizers are based on “good” nested lattices and on the overload avoidance mechanism. Thus, the approaches in DuQuant/OstQuant and that of NestQuant actually complement each other. Combining the two approaches is an excellent direction for future research that is expected to further improve the SOTA.\n\n### **Latency measurements**\n\nWe acknowledge that the significant improvement that NestQuant achieves in perplexity comes at a cost of slower quantization/dequantization speed. We provided FLOP count in our appendix, which show that the computational burden of NestQuant is reasonable. Creating a fully-optimized CUDA kernel for NestQuant is indeed an important task for future work. \n\nAs a temporary demonstration that NestQuant dequantization can indeed be fast on GPUs, we have implemented a CUDA kernel for a suboptimal version of NestQuant, which we refer to as NestQuantM. The difference between NestQuant and NestQuantM is that in the latter, the E8 optimal nearest neighbor decoder is replaced by a sub-optimal decoder. NestQuantM is close in ppl performance to NestQuant and still outperforms SOTA (see the ppl results table). Our CUDA Kernel does GEMV (matrix-vector) multiplication, assuming the matrix is quantized with NestQuantM and the vector is unquantized.\n\nBelow, we provide the runtime of NestQuantM GEMV kernel, compared to other baselines. The measurements are made for a $8192 \\times 8192$ matrix on A100 GPU.\n\n| Method | Time (us) |\n| -------- | ---------- |\n| Baseline (16 bits) | 97 |\n| NestQuantM (4.25 bits) | 50 |\n| QuIP# (2 bits) | 38 |\n| QuIP# (4 bits) | ~75 |\n| int4 uniform | 31 |\n\nNote that we estimate the runtime of QuIP# (4 bits) as double of the runtime of QuIP# (2 bits), since the size of the matrix that is loaded from DRAM doubles and the E8P dequantization procedure needs to be ran twice due to RVQ. Also, while we use 4.25 bit version with q=16 for benchmarking, the ppl for NestQuantM is still computed with q=14.\n\nThis kernel makes NestQuantM a practical quantization scheme for weight and KV cache quantization. We note that uniform quantization algorithms achieve a much higher perplexity values. Other vector quantization methods, such as QuIP#, QTIP, or AQLM, are only usable for weight quantization, since they require a significantly more complex encoding (e.g. QTIP requires solving dynamic programming problem for finding optimal trellis path). Additionally, we note that NestQuant can operate at any rate $R=\\log_2(q)$ for integer $q$ with essentially no change in the code. \n\nWe also wanted to stress again that the goal of this work was to improve the frontier of the tradeoff between the quality of the model and the amount of bits transferred into/from the compute cores. At this stage, we did not yet focus on improving actual generation speed. In terms of utility, however, we want to mention that transferring one byte from DRAM (HBM) requires 3000x the amount of energy of int8 multiplication (Horowitz, doi: 10.1109/ISSCC.2014.6757323). Thus, even accounting for the extra algorithmic overhead we believe that NestQuant already gives improved energy consumption for a given perplexity compared to other schemes.", "claims_and_evidence": "The authors provide evidence, such as RMSE error comparisons with SpinQuant, to demonstrate the effectiveness of NestQuant relative to uniform-based methods.", "methods_and_evaluation_criteria": "The evaluation benchmarks align well with those typically used for PTQ methods.", "theoretical_claims": "The theoretical section is clear.", "experimental_designs_or_analyses": "The primary experiments are conducted on Llama3-8B, with some extension to Llama3.2-1B. However, I am curious about the effectiveness of NestQuant on larger models, such as Llama3-70B, and other series like Qwen2.5-7B.\n\nAdditionally, the current experiments do not include speedup measurements, which are crucial, especially considering the use of the new lattice-based quantization. A detailed discussion of these measurements is recommended.", "relation_to_broader_scientific_literature": "There is a lack of comparison with other rotation-based PTQ methods, particularly the following:\n\n1. DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs, NeurIPS 2024.\n2. OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting, ICLR 2025.\n\nGiven the published date, I believe a comparison with DuQuant is essential, while the inclusion of OstQuant would also be valuable.", "essential_references_not_discussed": "The paper discusses weight-activation quantization and utilizes rotation transformation, but does not compare or discuss the following related rotation-based PTQ methods:\n\n- DuQuant (NeurIPS 2024)\n- OstQuant (ICLR 2025)\n\nConsidering their relevance, I recommend including a comparison with DuQuant, and OstQuant can be an additional option for comparison.", "other_strengths_and_weaknesses": "The lattice-based quantization approach is a promising and interesting concept.", "other_comments_or_suggestions": "More emphasis could be placed on evaluating the method's performance with larger models and its impact on inference speed and memory usage.", "questions_for_authors": "1. Could you include a comparison with DuQuant and OstQuant in the paper?\n2. Please consider adding more experiments on larger models, particularly Llama3-70B and Qwen2.5-7B.\n3. Could you provide a detailed analysis of the inference speedup and memory usage, especially given the new lattice-based quantization method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mz0wImzpc6", "reviewer": "Reviewer_c7kq", "summary": "The paper contributes a novel practical vector quantization method and applies it to quantize LLMs (post training). It is based on information theory with the key elements being:\n- Hadamard transforms to bring the distribution of vectors (weights/activations) closer to normality\n- Vector quantization of groups of 8 elements using the Gosset lattice. More precisely, a multi-scale Gasset lattice, where the set of scales is optimally estimated on a small calibration sample.\nThe method has been shown to achieve better LLM performance when quantizing weights and keys / values to 4 bits, compared to current SOTA, and the analysis suggests its practicality allowing also a speed-up besides memory savings.\n\n## update after rebuttal\n\nI had several concerns on the clairity, technical details and the how the reasoning is specific to preserving matrix products. The concerns have been resolved by the rebuttal, they can be addressed with a minor revision. From the experimental side, from my point of view, the paper presents enough evidence for improving SOTA quantization quality of large models and reasonable evidence of potentially low computation costs. It should certainly be accepted.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for such detailed reading and thoughtful comments! Please refer to other responses for new experiments.\n\nLattice defs clarity: We will provide references and a short definition of lattices and nested lattices early\n\n110: The exact statement is only given later in Sec 3-Random Rotation.\n\nEq. 1: $\\widehat{X^T Y}$ is our generic notation for the inner-product $X^TY$ from the quantization bits.\n\n### Q1\n\nThe fact that we can design quantizers for Gaussian matrices, and those work universally for any two matrices $X$, $Y$ is a special feature of the MatMul problem. The reason is that we can apply a rotation $U$ on both $X$ and $Y$, and quantize $UX$ and $UY$ (instead of $X$ and $Y$), without affecting the matrix product. The rotation makes $UX$ and $UY$ essentially Gaussian, so we only need to design quantizers for this distribution.\n\nNot any good vector quantizer (VQ) will work well for MatMul. In VQ only the trace of the quantization error covariance matrix matters. In MatMul quantization, its Frobenius norm is also important (O&P24). For “good” lattice quantizers in any dimension, this Frobenius norm is known to be small, which motivates the use of nested lattice quantizers.\n\nO&P24 show that for $R>0.906$ optimal nested *lattice* quantization of each column vector is indeed optimal for MatMul quantization. For smaller rates, further dimensionality reduction is needed. At this point we focused on just implementing the fast VQ, in the future other ideas from O&P24 will be added. For example, at R=1.2 bit/entry we found that in E8 NestQuant, dropping 10% of vector entries and at R=1 bit dropping 25% of vector entries works best.\n\nWe also note that off-the-shelf lattice VQs are either highly suboptimal or too slow for our purposes, and our novel multi-scale trick solves this.\n\n316: x is only assumed to be in $\\Lambda$, and this will indeed be made clear. We will add a reference to C&S’83 for bijection. See also our answer to Q2.\n\n### Q2\n\nIt's true that from $v$ mod $q$ the decoder can only know that the described point belongs to a particular coset of “coarse” lattice $q\\Lambda$. However, only one point (our decoder output!) in this coset belongs to the Voronoi cell of $q\\Lambda$, thus resolving ambiguity. This is described in C&S’83 and we will add a reference.\n\n### Q3\n\nAs explained in Q2, Alg1 describes a coset of the fine lattice, and Alg2 chooses the unique member in the coset that belongs to the cell of $q\\Lambda$. This simple enc/dec procedure is possible only due to the quotient group structure $\\Lambda/q\\Lambda$ of NestQuant. Shaping regions that are not cells of $q\\Lambda$ do not lend themselves to these enc/dec algorithms.\nIn our experiments we used $\\Lambda=E_8$, as in QuIP#. Note however, that NestQuant enables efficient encoding/decoding of the $2^{dR}$ points from E8, and allows to use any R that is log2(int), which we see as a major advantage. QuIP# uses a different set of $2^{dR}$ points and encodes them via slow residual quantization and LUT. For 2,3,4-bits and W-only quantization (which is the goal in QuIP# and QTIP) this is totally fine, but for KV+A that are quantized in runtime, those solutions are too slow.\n\nThe NestQuant framework is not restricted to E8. It can be implemented with any base lattice $\\Lambda$. In O&P24 it is shown that for “good” high-dim lattices NestQuant is information-theoretically optimal for MatMul quantization. Practical considerations prohibit high-dim lattices, and instead we use low-dim lattices. In such dimensions, overload errors are unavoidable and deteriorate performance. Our novel overload avoidance mechanism (multi-scale) is crucial for attaining SOTA ppl.\n\n### Q4\n\nWe don’t quantize queries for a fair comparison with other works (SpinQuant, QuaRot), which also keep queries in 16 bits when KV cache is quantized. In autoregressive generation the queries are used once, while keys and values are used by all subsequent tokens. We agree that applying head-wise Hadamard transformation on values would make them easier to quantize and are thankful for this suggestion.\n\nIn the final revision, we will clarify Figure 4. The Q-box around $HW_K$ group indeed means that H is applied to the weight and this matrix product is quantized: The $W_oH$ part of the diagram should be corrected to $HW_o$. In this diagram, H matrices on key-query path are applied per-head (i.e. in small dimension), while all the other H are full embedding dimension sized.\n\n### Q5\n\nWe agree that quantizing activations below int8 should be used for the scenario when they need to be sent over the network for very large models under pipelined setup. The answer to the question of whether Hadamard transform can be done on the fly depends on particular hardware and cache structure but we do mention that SpinQuant and others are already using this idea in practice. We believe that one activation vector should fit in the L1 cache of most GPU SMs, hence making fast Hadamard transform especially fast.", "claims_and_evidence": "I have not noticed any over-claiming or inconsistency in the claims and the experiments.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper devotes significant space to review prior work and background. It is not quite clear to me which parts of section 3 present original ideas. Mainly it is probably adaptation of results from information theory. I have some concerns about clarity, but they are also connected with the theoretical claims made.\n\nFirst, I find the paper lacks clarity in defining the following:\n1) Lattice. It is specified rather late in 308 and indirectly. \n2) nested lattice (139, 176) / nested lattice quantization. Should be important, given the title, right? Is nested lattice a lattice?\n3) Gosset lattice. It is specified in 324, but I do not see how is it a subset of standard integer lattice as claimed in 265-267. It is also not obvious that $D  \\cup D + \\frac{1}{2}$ is a lattice.\n\n110: I am confused by the claim that $UX$ is i.i.d. Gaussian, i.e. $UX \\sim N(0,I)$, because this necessarily implies that $X\\sim N(0, I)$ as well. \n\nEq. (1):  I believe the authors mean the hats on X and Y separately.\n\n**Q1** What part of the method makes it specific to approximating the product of matrices rather that optimizing the expected distortion of X and Y separately? It seems to me that all the arguments and constructions in Sections 3/4 (in particular, granular and overload errors) are motivated by the Normality of inputs and the expected distortion. The method for finding optimal scales also seems to be optimizing the expected distortion.\n\n Related to the above, How does Eq. (1-2) compare with a rate that one gets for $E(X' Y - \\hat X' \\hat Y)^2$ when quantizing $X$ and $Y$ optimally to minimize the distortions $E (X - \\hat X)^2$, $E (Y - \\hat Y)^2$? Doesn't it give $2*2^{-2 R}$, which is kind of very similar? I assume we are in the range $R > 1$...\n \n316: It is not clear whether $x$ is assumed in $\\Lambda$ or in $C$ at this point. The claim that $Q$ is a bijection between C and $\\mathbb{Z}_q^d$ for any generator perhaps needs a justification or reference. \n\n**Q2** I do not see how the algorithm implements the shaping according to the Voronoi region of $q\\Lambda$. The Encode procedure in Alg. 1 already assumes $x$ in the Voronoi cell of $q \\Lambda$ but in Alg. 3 this is not ensured. The encoder using $v {\\rm mod} q$  can of course encode any lattice point (then the input constraint of Alg.1 needs to be lifted). The question is whether the decoder will decode it with zero error iff $x \\in V_{q \\Lambda}(0)$? I guess yes, but would be nice to clarify in the paper.", "experimental_designs_or_analyses": "I find the experiments to be sound, comparing the perplexity at the same bits per entry for all methods in Table 1 and in Fig 1 for several rates. Llama results for different rates in Tables  2,3 should provide a broader basis for potential applications and further methods. Validation in Fig 5 takes into account both the perplexity and the change in the code length due to different k. Additional experiments on a synthetic problem (Fig 3),  and ablations in Tables 5,6 are also very useful. Perhaps it would be instructive to see what happens if orthogonal transforms are omitted.", "relation_to_broader_scientific_literature": "I am no expert, but what I see in the paper, it mentions many works in various contexts (e.g. locality-sensitive hashing).  The paper advocates quantization for approximating inner products and cites recent theoretical results in 124. As discussed above, it is not entirely clear whether this is actually used towards the design of the scheme. I do not see the paper to be contributing substantially in methods, so I would not expect it to impact in the broader scope.", "essential_references_not_discussed": "**Q3** I think I would like to see more clearly what constructs in the paper are claimed innovative / original. In particular, a more detailed discussion of what is different in the methods from Tseng et al. 2024. They seem to be using Gosset lattice E8 and Hadamard transforms as well. Why exactly their method is deemed impractical by the paper?\nAlso, what is the attribution of the encoding-decoding algorithms 1,2 for the lattice restricted to a cell of $q \\Lambda$?", "other_strengths_and_weaknesses": "**Q4** In Fig. 4. Why there is no quantization of queries (top path after orange Hadamard). Is it intentional, like in this is in-chip and not a computation bottleneck? Then I wonder why quantizing the values (bottom path) is without a Hadamard before it? Doesn't it degrade the quantization accuracy? In fact, from this diagram, I don't know why the whole bottom path till the pooling with attention weights cannot be just identity, because I think a linear transform, which is the same for all tokens, can be as well applied after the pooling. \nThe notation of the plot is confusing. I assume a group like $H W_k$ means $H$ applied to weights and then quantized (because of the Q supperscript). Also, there are many different colors in the Figure. I they have a purpose, it would be appropriate to explain in the caption. For instance, it would be nice if paired (same) Hadamard transforms were of the same color, but this seems not to be the case for the first on the left and those applied to $W_q$, $W_k$, $W_v$.\n\n**Q5** Can quantizing activations with a Hadamard transform actually achieve a memory saving, i.e. can the Hadamard transform be applied on the fly, without writing the activations to memory before quantizing them? If not, then there is a write and read of them involved, and they need to be decoded before used in the linear layer. It seems to me there is no advantage in memory use or speed compared to leaving activations as int8.", "other_comments_or_suggestions": "# Minor Issues \n\n035R: Would be useful to elaborate on what is KV cache, and why it is a bottleneck, regarding the memory.\n\n085R: It is not clear what is random for the expectation.\n\n173: How finding the nearest lattice point is an assumption?\n\n220: I find it hard to follow. Choosing $S \\in R^n$ and independently choosing $\\beta$ to scale $S$ seems redundant. \n\n230: What is the difference between ${\\rm covol}(\\Lambda)$ and ${\\rm vol}(\\mathcal{V}_\\Lambda)$ in (3)?\n\n236: $vol(B)$?\n\n245: One confusion I have is that lattice quantization is still uniform over each of its basis directions. The claim was rather obscure. It would help to detail the example in Fig 2: show the Generator, is this lattice a subset of integer grid if scaled? I guess not?\n\nI've noticed Gosset lattice has symmetries with respect to 4x4 Hadamard transform. Then perhaps the rotation can be simplified a bit building from $I_4$ blocks? I am not sure this makes sense, but though I would mention it anyway.", "questions_for_authors": "I marked the most important questions as Q1-5 above. In summary, I am concerned with that not everything is presented clearly enough and possibly somewhat misleading. I am open to rising the score if these questions are clarified / rebutted.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AvvphqdLPJ", "reviewer": "Reviewer_c956", "summary": "This work presents a matrix multiplication replacement for low precision quantized neural networks. A new vector quantization mapping is explored to maximize the efficiency of low bit number distribution. The authors propose to use a new encoding scheme to achieve near lower-bound compression.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comment and appreciate the provided feedback. Following the suggestions, we run the evaluations on a larger set of models and provide the perplexity results, as well as comparisons with other methods. The results are outlined in the table in the response to reviewer a2PJ. Our method demonstrates advantage in both weight-only quantization setup, and weight, KV cache, and activation quantization setup. In fact, the models where weights and KV cache were quantized with NestQuant are competitive even with weight-only quantization benchmarks.\n\nFor speedup measurements, we introduce a relaxed version of the algorithm, called NestQuantM. In this algorithm, the E8 oracle in dequantization prodcedure fixes the parity of coordinate sum by adjusting the first coordinate, as opposed to picking the best coordinate using argmax/argmin. Such a simplification preserves the correctness of the encoding-decoding procedure (with a slight distortion of the shaping region), but significantly improves the efficiency of the code on GPU. We tested NestQuantM on several models and did not observe significant deterioration in quantization quality, as shown in the ppl results table. We have implemented a CUDA kernel for NestQuantM. When running encode_e8 function, the kernel stores a scaled 8-dimensional vector as a collection of 8 int8 entries, packed into two 32-bit integers, and performs mass operations using regular integer operations and CUDA integer SIMD instructions. Please check the detailed description and benchmarking results in the response to reviewer r651.\n\nWe provide the comparisons with QuIP# in the response to reviewer a2Pj. We note that NestQuant outperforms or matches QuIP# performance on all models. In addition, QuIP# uses an inter-layer finetuning algorithm, while NestQuant assumes no access to the data except for Hessian calibration data. The focus of this work is on quantization of W+KV or W+KV+A . These setups are considerably different than W-only quantization (for which QuIP# was designed), because KV and A quantization is *done in runtime* and the encoding algorithm must therefore be fast. For W-only quantization, encoding is done offline, and may be slow.\n\nHaving said that, our experiments show that NestQuant attains SOTA ppl even for W-only quantization at R=4 bits. Going to the extreme low-rate regime, e.g., R=2 bits,  requires further ideas from O&P24 not discussed in the present manuscript (MMSE scaling + sparsification + dithering). We will implement them, but it takes more time, and the experiments may not converge in the rebuttal time-frame.\n\nWe also respond to the remaining comments:\n\n- We treat both QuaRot and SpinQuant (as well as other quantization schemes) as our benchmarks, and show improvement with respect to all previous quantization schemes. In Figure 3, we show results for synthetic data consisting of Gaussian iid matrices. Since the Gaussian iid distribution is isotropic, any rotation applied on it will not change its distribution, and SpinQuant and QuaRot are identical in this context, because (while they apply different rotations) they both use uniform quantizers.\n\n- We did mention in Section 2.2 that QuaRot already applied Hadamard transform, but the reviewer is right and we should have also mentioned this in Sec 4.3. We apologize for the omission. In a revision Sec. 4.3 will be sure to reference prior literature on rotations (SliceGPT, QuIP, QuaRot and SpinQuant)”\n\n- As mentioned above, QuIP# was designed as a W-only quantization method. Consequently, its dequantization algorithm was designed to run fast, but not its quantization algorithm. Our experiments show (small) improvement over QuIP# even in the W-only quantization regime, which is not the focus of our work. Conducting experiments for W+KV/W+KV+A quantization  with our Alg 1 + Alg 2 + overload avoidance mechanism replaced with the QuIP# E8P quantization/dequantization from Sections 4.2 & 4.3 in QuIP# paper will take a long time, because the QuIP# E8P quantization algorithm is not fast enough. Given the fact that NestQuant attains better ppl for W-only quantization, we expect the same trend also for W+KV and W+KV+A.", "claims_and_evidence": "The claims made in the submission are supported by extensive existing literature.", "methods_and_evaluation_criteria": "The authors evaluated their results on LLaMA3-8B, across several zero-shot reasoning tasks, accuracy, and a perplexity metric on the Wiki2 dataset. The major experiments are provided in Table 1, and more ablation studies are provided in Section 5. \n\nThere are many other experiments that need to be performed to demonstrate this method's effectiveness. \n\n1. Results on LLaMA2, including 7B/13B/70B where most existing literature has verified. \n\n2. Results on LLaMA3-70B, where QuaRot, SpinQuant also have provided the results. \n\nIn addition to models, the authors need to verify the actual latency on GPUs with dedicated kernels, Section C only provides simulated speedup, which is not enough. \n\nMoreover, for the weights only quantization, 2-bit quantization on all models mentioned above is needed.", "theoretical_claims": "This work does not introduce new theories.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "This work is related to the model compresion literature.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Weakness\n\n- SpinQuant is referred to as the uniform quantization baseline. However, SpinQuant does not use random Hadamard rotation, I believe the uniform quantization baseline is QuaRot. \n\n- Sec 4.3 overlaps QuaRot, and the authors do not mention their differences. \n\n- I am curious to see if the authors use E8 lattice in QUIP# and align all other setups like random rotation, how much performance improvement can we get?", "other_comments_or_suggestions": "The notation systems could be improved with a Table. Current notation involves multiple symbols yet lacks clear explanation.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel and effective post-training quantization method that achieves state-of-the-art results for low-precision quantization of large language models. The approach, based on nested lattices, demonstrates significant improvements in perplexity compared to previous methods, including SpinQuant and QuaRot.

Reviewers provided valuable and detailed feedback, especially during the rebuttal phase. While some concerns were well-addressed, some areas still require attention in the final camera-ready version:

- Comparison with DuQuant: The reviewers noted the importance of including comparisons against DuQuant, which the authors partially addressed. It is essential that these comparisons are clearly incorporated into the final version to provide a comprehensive view of the performance landscape.

- Latency verification with dedicated CUDA kernel: Reviewers emphasized verifying actual latency on GPUs using a dedicated CUDA kernel. This verification is crucial to ensure the practical applicability and demonstrate clear benefits in real-world inference scenarios. The authors provide an evidence via NestQuantM kernel and should be expanded and incorporated in the main text.

- Notation and clarity improvements: The methods section currently suffers from notation inconsistencies, with matrices and sets represented both by capital and calligraphic letters interchangeably. Additionally, several symbols introduced earlier in the manuscript are not consistently used in subsequent sections. This must be resolved for better readability and precision.

I strongly encourage the authors to incorporate the above points thoroughly in their final manuscript to enhance clarity, reproducibility, and impact. Overall, the submission represents a significant contribution to the field of LLM quantization and should be accepted.
Decision: Accept (poster)